If you're following a specific diet plan that says only count fat grams or calories from fat then do that. Otherwise,
count all calories for the day. A calorie is a calorie whether it comes from fat or carbohydrate and, if you eat too many calories, you won't lose weight no matter how low fat your diet is.
Find a low fat diet and read up on it...you may find specific diets that say only to count a specific thing. For example, with Atkins low carb, I only count carbs and can more or less ignore calories altogether (using common sense). I obviously won't lose weight if I eat nothing but bacon all day but I've been losing 2-4 pounds per week on a diet of veggies, meats, cheeses and other low carb goodies...when I actually counted calories I was up in the 2000 to 2500 range and still losing weight.