Dear Amanda,
Yep, I do go over my calories sometimes, and no, I don't let it bother me at all! Like JenMusic said, I think it's more important to pay attention to the bigger picture. If this is something that you're doing regularly (multiple times per week, every week), then yes, it might be a problem. But if this only happens occasionally, it's not going to have a huge impact on your weight loss.
To me, if there is a problem with what you describe, it's not that you went over your allotted calories for the day, it's that you ate a food that you didn't seem to want to eat and felt bad about it. For me, that would be the thing I'd want to work on - looking at why I chose to eat that food, why I didn't think about it first, and so on, rather than focusing on the calories. In my opinion, it's far more valuable to reflect on the emotional, psychological side of things rather than black and white things like calories eaten, pounds lost, and so on.
I guess another thing that seems problematic to me is that you're describing feeling "starving". Of course, weight loss via calorie restriction requires reducing calorie intake, but it doesn't mean you should feel uncomfortably hungry very frequently. It's normal (and healthy) to feel hungry before a meal, but if you keep finding yourself feeling really hungry, perhaps try to consider how you can alter what you're eating to be more satisfying to you.
In short, I don't have any foods that I "ban" or anything that's "off plan", and trust me, there have been a not inconsiderable number of days where I've gone over my calorie limit, but I've still lost 70lbs. Going over your calories for a day isn't the end of the world, but use this opportunity to reflect on things that
may affect your ability to lose weight more seriously - such as "automatic" eating and habits relating to food, emotional relationships with food, and so on.
Best of luck