I dated a guy with Asperger's for about a year (and we're still great friends). He'd say some stuff that was unintentionally hurtful, but NOTHING like that. Obviously it varies case to case - but like in the case of my friend, he's put a lot of time and energy into learning how to navigate social situations and the vast majority of the time does extremely well (often better than I do, and I don't have Asperger's!).

Sometimes people are just mean and thoughtless, and with Asperger's it might slip out when otherwise he'd not say it. Tell him upfront that was unacceptable and hurtful, and if he doesn't make a serious effort to prevent that kind of thing from happening again, move on.