When I first started, I was fluffy, ya know? Sedentary, really ate just whatever I wanted. When I first started walking and cutting calories, I lost alot fast but then it kind of leveled off. That could be the kind of thing that's happening to you.
Three weeks is a good start and is obviously working. That's still a half pound last week right? There's lots of reasons to slow down a bit or fluctuate in one day even. I think you'll be just fine.
If you find that nothing happens or you start to gain, then it might be time to look at your food and exercise intensity but I think you're doing fine.
Eat good food. Move yourself. Lift something.
You'll be surprised how much you don't miss grain.