It really depends on body type, muscle mass, bone structure, height, and how tight/lose someone wears their clothes.
I normally wear Levi's and here's how I fit into the pants:
14 ~200lbs
12 195-185
10 185-175
8 175-present
6? present
But my 8's are too big (I can pull them off without unbuttoning them) so it's probably more like a size 6 (28) in Levi's at the moment. With the last few pounds I've lost 2" off of my hips so I dropped a pants size very quickly.
I'm not that tall (5'6") but I have a large frame (primarily a larger upper body), do lots of weight lifting and cardio (I was a college athlete), I also carry a lot of weight in my bust, upper arms, and midsection. So it allows me to wear smaller pants. In tops I'm all over the place. I wear smalls at Old Navy and mediums in higher end clothes. I have a pretty small back so that makes a difference there too. It's just how I'm built.
I know it might seem unfair but, on the other hand, it sounds like you have a smaller frame, which means you can probably go down into the smaller sizes if you choose too. Most likely I'll settle at a higher weight than you would and will never look "slim" but more athletic instead.