Relax! 50-100 calories isn't going to make much of a difference long-term. Just don't let it slip to 200-300 on a regular basis and you'll be just fine.
I agree with you. Just stay cool and keep going. Don't get anxious on the weight loss. Just put your best efforts and the results will be yours.