At a party or large gathering, I usually just say I had a big lunch/dinner (if anyone even asks) and carry a glass of sparkling water with lemon. I usually try to be ready to improvise if need be. For instance, today we were invited to brunch at a friend's. They know me well and understood that I'd be making my own omelet and providing my own ingredients. Well, I showed up with everything ready, and everyone else decided they would rather go out to brunch. Nothing on the menu at the restaurant really fit what I needed, so I asked them to just give me grilled Romaine with a couple ounces of chicken with little or no oil in grilling and no oil/vinegar/dressing on the salad. The waitress understood I was on a restricted diet and it worked perfectly. So many people these days have restricted diets due to food allergies or other issues (or are just picky), so no one blinks an eye.
I grew up in a household where you had to eat every morsel on your plate. It took me years to realize I'm under no obligation to eat something I don't want to just to make someone else happy.