Your questions is: 'is it worth it?' The answer is going to depend on how YOU will react to the chocolate buffet -- what will the effect be on YOU and your weight loss? And only you can figure that out for yourself.

Can you go, enjoy yourself, and get right back on track? Or will it sabotage you in the long run?
The right answer would be NO for me. I'm been working at weight loss and maintenance for close to five years now and know how I'd react to a chocolate buffet. Not only would my weight shoot up in the short term - which would be very distressing to me, not to mention how darn long it would take to get it off again - but it would trigger
months of craving. I would be thinking about chocolate for months, seriously. And I hate having to deal with cravings!
Everyone's different, of course.

Unfortunately, moderation has never been part of my vocabulary, so I do far better by sticking with my usual healthy diet and avoiding fattening trigger foods that I have trouble controlling. On occasion, I'll have a dessert when I'm out to dinner with DH or at a special event - maybe once every three months or so. But a BUFFET of any kind (breakfast, chocolate, Chinese, any!) is a recipe for disaster for me and I run screaming the other way if ever faced with one.
It's been almost five years since I've had a French fry -- real soda -- a candy bar -- and a lot of other things. The last donut I had was October, 2004. Do I feel deprived? Heavens, no! I'm probably the happiest and healthiest person you'll ever meet. Here's the secret -- you can live your life perfectly happily without indulging in unhealthy food.
Anyway, enough rambling! You need to find YOUR way -- what works for you. Look at your weight loss history and think about how you react to your trigger foods and decide what's right for you. We're all cheering for you!