My vote is to keep what you love, and toss the rest.
When I started gaining weight back, I tossed most of my skinny clothes that I didn't love. The stuff I still have is a pair of size 11 jeans, some size 9 goth pants
(Hey, I was darn proud of those...these were bought after my doc told me I could lose all the weight I wanted, but I'd never get a single digit size over my hips
), and a green size 12 formal gown.
Everything else went the way of the dinosaur.
If I'd held on to everything, my closet would be overflowing. I am nothing if not a clothes horse.