Rest assured, there HAS been a difference. You just maybe can't see it, for a couple reasons.
1. We see ourselves EVERY day. Changes happen VERY subtlely, over time. When you don't see a change day-to-day, it can seem like no changes are happening at all, when really, we just don't notice them.
2. Our eyes need a bit of time to catch up to our brains in terms of weight loss. Brains are experts at what is called "pattern recognition" - it recognizes "oh, I'm looking at myself" and fills in the details with previous images of itself. When you're losing weight, the brain actually perceives you as bigger than you are because of this...and it takes some time to go away.
Did you take any measurements at the start of your journey? If you're clothes are looser, you're losing inches in addition to lbs, but sometimes seeing the sheer number of inches can be motivating.
What about photos? Do you have before and current? It is often easier to see a difference in a picture than to see it in a mirror.
In terms of not going down a size, everyone is different in how long it takes them to go between sizes. Stores, even, are different in their sizing. If you tried on a pair of the same jeans you're wearing now (the looser ones), in one size smaller, they might fit, but a different brand at a different store might not. That is why I prefer to rely on inches and pounds, rather than on sizes - they just vary too much to be reliable!
It is virtually impossible to lose 21 lbs at your start weight and not have a visible difference, so take heart! You just need some time to start seeing it.
Lots of

to you.