if you're not comfortable in a dress shirt, i suggest dress pants, a tank and a cardigan. if you feel that looks unpolished, i suggest a belt to nip you at the waist, and black shoes/ballet flats.
i work in a business casual environment and i rotate between dresspants/cardigans or dark denim/boots/blazers. i also meet with clients.
I think the choice you made was excellent. Grays and Blacks you can't go wrong as it gives a nice conservative professional look. I have sat on many interviews and "casual" doesn't mean quite as casual as some folks take it as. I still prefer an attractive business look until such time you are actually hired as part of the team and can dress as they do.
