Only you know whether you can handle eating cake without it triggering a binge or knocking you off the wagon so to speak. If you don't think you can, then absolutely refuse and do not feel guilty about it. If you feel like you have the self control to treat yourself to a small slice and limit yourself, then do that and enjoy it.
As a compromise, why not look up healthier/low cal recipes or substitutions and make your own cake to meet your needs?