When your muscles are really sore, they hold on to more water. The soreness you feel is due to little tiny rips/tears in your muscles, and some extra water is needed to repair the muscles... this repairing process is how they grow stronger! So, gaining or maintaining a little weight when you start a new workout program, or change the intensity, is totally normal.
It's true that muscle is more dense than fat, so it will weigh more for the same volume.. and sometimes this means you get smaller without losing much weight, because you're adding some muscle weight while you lose the fat.
But, having more muscle makes it EASIER to lose the fat, because muscle also burns more calories during everyday activities than fat does! So definitely keep up with the exercise, build up some muscle, and it will make it easier in the long run. (Also, you'll have a sexy strong body waiting for you under the fat.
Not to mention how good exercise is for preventing diabetes and heart disease, maintaining strong bones, improving your mood, etc.