I think the debate about whether what he said was appropriate could go on and on. Obviously different relationships have different boundaries, and what could be appropriate to one couple could be seen as a betrayal by another.
The real question is how do YOU feel in your relationship with HIM? Are you two happy in general? You mentioned that you're the one working. How is that working for you both? Do you get enough attention from him, or are you left feeling overlooked and under-appreciated? Does he feel ok about being the one at home or is it a source of frustration for him? How much time and energy is spent on maintaining the marriage in general? Do you feel that he is happy with where things are right now between the two of you?
That might seem like a lot of questions, but I know that how I would feel about the situation would be colored by how I felt about the relationship and it's stability at the time and to be honest, while it may feel hurtful it doesn't strike me as the main issue.
Just remember that he's with you for a reason and loves you for a reason
ETA: If he's at home because he's taking care of the kids then that's one thing, but otherwise I've gotta agree with Jay. He needs a job.