What kind of time scale are you talking about - the next day, when you miss a day of exercise? or slacking off over a length of several weeks or months and tracking your weight loss over that time?
I usually see the scale go up
the day after a workout. I attribute this to the fluid retention that comes along with muscle healing and recovering. If I skip a workout, I may very well see the scale go down the next day, since I won't have that large fluid swing. In the long run, though, I'm confident I'll see better results if I keep at the workouts.