You sound tiny! That is by no means the average size, believe me. Apparently the average size here is 14-16 (US 10-12.) So you have nothing to worry about. And whilst I don't see many very obese people in the UK, I don't see many tiny ones either.
This boyfriend sounds like he has ridiculous (and unrealistic) expectations of you.
Honestly hun, you are at skinny end of the spectrum over here too, just ignore him that was a very silly comment to make! I am 5'4" and i would consider myself very slim at even 120! I weighed that much when I was about 14 and hadn't properly developed yet and I was still very small then so you have absolutely nothing to worry about missy, so cheer up xx
You are perfectly normal by any standards. I am (I think) quite big boned, and at that weight I wasn't skinny, but was comfortably curvy; and, hate to point that out, but if your boyfriend is of tiny/slender build, it is totally unfair to send the ball in your park.