Yes-it does get harder the smaller you get. The reason is because the smaller you are, the less calories your body uses during the same activities that a larger person is doing. It takes a lot more oomph to get a 220 pound body to walk a mile down the road than it does a 120 pound body.
My suggestion would be:
#1-PATIENCE.
#2-Up or advance your exercise. This doesn't always mean going from doing 30 minutes a day to an hour. It can mean switching from 3 pound hand weights to 5, or walking that same 2 miles on your treadmill at a slightly faster pace and finishing it in a faster time.