I wish I could tell you that yes, the pounds should have just melted, but each of us loses at a different pace. And there are so many variables such as how many calories you consumed each day, are you carb sensitive, etc. Exercise can help with dieting, but from my own experience and what I've read from other posts on here, it won't make you lose weight by itself.
Now, personally, I'd be celebrating 20 pounds off. That's an average of 5 pounds a month. That's a good steady weight loss. By the end of a year, that's 60 pounds and it sure beats gaining 20 pounds!
So, congratulations on your weight loss and developing a walking program you're comfortable with. Good for you!