It really does count. If you are doing a lot of exercise, especially muscle building ones, you may want to start with the tape measure instead of the scale as your indicator...muscle weighs more than fat, and at the begining, you may find your weight doesn't seem to drop, while your measurements shrink.
Your metabolism will also count. If you diet too low, your body may go into starvation mode and hold on the weight. Some conditions make it hard to start losing weight.
But if you are eating sensibly, and keep a reasonable calorie deficit, you will lose weight.
You may want (and I recommend that you do this) write down everything you eat and keep a running total of your calories or points or whatever, so you can look back and reassure yourself of what you have done so you don't get the diet jitters.
But remember, the tape measure is your real friend. The scale can drive you crazy, but the measurements are the proof of the pudding!