I think it's a consistency thing. If you start off counting them, keep doing it. If you start off not counting, keep doing that.
For example, I count almost everything, but I've never counted the sprays of Pam I use in cooking. I'm sure there is a calorie count, but I've never worried about it because it's something I use every day. As my weight loss has progressed, I've had to lower my calories a bit, but the Pam is still a "free" food because I use the same amount I've always used.
I think you're good not to count them, since you know that your use of them will be consistent over your weight loss.