I'd like to point out that, considering that it's 3500 calories per pound of fat (ie, it takes consuming 3500 excess calories to gain a pound of fat, and conversely you have to create a 3500 calorie deficit to lose a pound of fat)...it would be VERY difficult for you to "gain" 3-4 lbs in one day. You'd have to have eaten at least 10,500 extra calories all in one day. It's *doable* (having seen that one special on whatever channel it was on, with people who consumed like 33,000 calories/day, I'll never say it's not possible) but you'd have to have had a pretty bad binge to do that.
So if you're only losing slowly, odds are it's just that your overall calorie deficit isn't as big as you think it is, or as big as you'd like it to be. Try tracking everything you eat, every day, for one week. Weigh, measure, etc. Enter it into Fitday, or write it down and calculate the calories by hand. At the end of the week, add up how many total calories you ate. Use one of those base-calorie-count calculators to figure out approximately how many calories you burn per day. Calculate how many calories (approximately) you're burning through exercise, add that to the basic per-day numbers. Compare the two numbers. The first number (your total calorie intake for the week) should be lower than the second (your total calories burned for the week), and how *much* it's lower by is what's going to tell you how much weight you're losing. It's the calorie deficit over time that matters, not one bad day per week messing with you. If you want to lose weight faster, mess with your calories-per-day until you find something that works. Just don't put *too* much weight on the weekends - yes, bad behavior on weekends can mess with what you've done during the week, but keep that perspective of "deficit over time" versus "every single day must be good" and it might help.
I wish I could help more about the family stuff on weekends, but actually...that's one of my worst demons right now. Love my family dearly, but I'm the only one seriously trying to lose weight (although 90% of my family could stand to try it themselves, but I'm not gonna preach at them about it), so it wreaks havoc with me when we go to Tahoe Joe's (really good steakhouse with huge steaks, incredibly yummy mashed potatoes, and very tasty "specialty" margaritas) or Johnny Carino's (Italian. Enough said.).
I'll let you know if I come up with any strategies. You do the same, ok?