I used to have this too. When I was working out a lot sometimes I would have two sandwiches for lunch
I have found that it's important to make sure you plan your food post workout carefully. It's tempting to want to eat super healthily so that you don't undo all your work but you should make sure you eat something substantial after exercise.
I normally go for fruit, yogurt and then a bit later some rice or potato dish. I don't know what kind of plan you are following but I hope this helps! Maybe you could try eating different types of food and keep a log of which ones don't make you have crazy cravings?
Also - please don't stop the workouts!!