You can absolutely do it!
It's not detrimental to your metabolism. The "breakfast is the most important meal of the day"/"6 mini meals > 3 meals a day"/"you gotta keep eating all day to stoke your metabolism" are all claims that have no scientific backing.
Intermittent Fasting is essentially what you're doing. Google it and find more info. There's some great, highly technical websites on it, and some more laymans articles in New York Times and other sites with more info.
Hunger is just what happens when our body EXPECTS food, not when we need it. Even for those folks who are hungry in the morning, that doesn't mean you HAVE to eat breakfast. Eating later and later and training your body to not expect food until noon is possible, and for some of us, extremely helpful. There are all sorts of studies that IF renders benefits for the heart, blood sugar control, and hunger management.
so keep doing what your body tells you it wants to do
good luck!