Psh, that's not just something I "noticed," that has been one of my main obstacles in dieting.
I'm a poor college kid, and I barely had room in my budget for a few boxes of Kraft Mac and Cheese, some Wonderbread and bologna. I've actually had to take on extra shifts to pay for my diet, and I don't eat extravagantly at all. I buy chicken when it's on sale, bag-o-lettuce and iceberg lettuce and mix them, maybe a cucumber, the cheapest fruits I can. I do manage to eat a balanced diet of 1100-1300cals of nutritional whole foods, but it has taken a lot to be able to afford it.