Nope, you're not insane! I remember feeling like that after breaking up with one of my ex's. First loves especially are hard to get over!
But eventually you will completely, and then thoughts of them will be more curiosity than heartbreak.
It sounds also like you didn't have enough time to get over the ex before dating a new guy, so maybe there are some things that have been left unprocessed.?