Whoa, whoa, whoa ... wait, wait, wait ... it's only been 2 weeks -- that's not a sign of an affair or anything else. Heck, I know people that take a sex break longer than that, and for many different reasons. The fact that your DH cuddles with you shows you that he cares about you.
The first thing to do, is just sit down and talk to him about it. I know the answer he gave you sounds kinda silly, but I wouldn't read so much into it. You did have some time apart, so maybe he got used to that, but you could at least discuss how often you would like to come together, so2speak.
Plus, lots of marriages go through "dry spells" but in time, things get better again (like another poster stated). I mean, we're not talking 2 months or 2 years here, but since it bothers you, talk to him right way; don't put it off. First, try to find out what he's thinking and feeling.