i definitely understand what you mean about having a muscular build. no one (i mean no one) would believe what i weigh (compared to my size). the few friends i have told were very surprised. my problem is not lack of muscle...just the padding i have all around said muscle
all that to say--i definitely have similar stuggles in this area.
anyway, figuring out calories is a trial and error process. i think you're on the right track with carefully measuring portions and trying to keep track of calores. it sounds like if you are choosing healthy options and eating small meals throughout the day--that's great. if you keep doing this, your body is going to learn when it is hungry and when it is full. allow your body to need different things from one day to another. if you are trying to stick to "X" number of calories per day, allow for days when you will need X-100 or X+100. i'm no expert, but i've found that the more strict i am...the more i am likely to snap and binge on something ridiculous...
good luck to you! like i said, i'm no expert, but feel free to message me anytime