Sadly, those stores that you mentioned, Saks and Bergdorf Goodman, cater to size 2 lollipops, maybe even size 0. If you are over a size 4 they think you are in the plus sizes.
Nordstroms has lots of nice clothes. If he didn't actually come out and say "you're fat" maybe he was trying to be helpful.