I know how you feel. I'm mostly okay with family, though I did have a hard time last thanksgiving seeing BIL who I hadn't seen in several years and his new wife, the marathon runner. But when we went home to Boston a couple of weeks ago I was very conscious of the possibility of running into someone who last saw me at 19 years old and 120 pounds. Gah! And an old boyfriend would have been the worst.
I do think it's a great idea to love and accept yourself at any size. Not sure how to get there, though.