I have a problem

I have known a very good friend of mine for some time now. A few years ago I had a very bad crush on him. Against my better judgement, I ultimately broke down and told him how I felt. He explained he didn't see me the same way.
4 years later, I consider him my "other brother", our relationship has turned into this completely sibling-like friendship (he gets along disturbingly well with my real brother). We razz each other, we support each other, we even grocery shop together. We dieted together when he started his P90X kick.
In the past 2 months, we've spent more time with one another for some reason. 6 people at work and at our regular haunts have asked me or assumed we were dating. Several didn't believe me and thought I was covering something up.
It's difficult for me, as although I know he only wants my friendship, I am still attracted to him. He has a bit of the Peter Griffin disease. He's a bigger guy who sets his goal for much smaller girls that are generally not interested in dating him.
I wish no one would've asked/approached me about our relationship, as now its making things more difficult to tamp down my attraction to him, seeing as how other people are noticing our relationship chemistry. If I start to think about it, I find it too easy to tell myself that he's been more attentive lately, that we have longer conversations about everything and nothing. We act like an old married couple.
He's told me lately about another girl he's been trying to express interest in, so I don't really get the inclination that he's interested in me. Its taken us a long time to get this far with our friendship, and I do not want to do/say anything to jeopardize it in the slightest. I still catch myself doing stupid stuff like exercises to develop Venus dimples (back dimples) because he told me once he was attracted by them.
I am open to any and all advice please. I am trying very much to concentrate on my goals, to stay focused, but I fear I'm developing another crush on him. I can't tell how much of my attraction is because we're so comfortable with each other