It's generally harder to lose weight since sometimes a person will lose muscle along with fat when you lose weight - this sets you up for quicker regain/difficulty losing weight in the future.
Muscle is metabolically active - it takes calories every day to maintain muscle. Fat is metabolically inactive, it takes no calories to maintain fat.
Imagine weigh 200 lbs and you lose 20 lbs. That's great! You lose 15 lbs of fat and 5 lbs of muscle. You quit dieting and regain the weight. The scale says 200 again, but you replaced 5 lbs of muscle with 5 lbs of fat. Fat takes 30 calories a day to maintain, that would be 150 calories a day. If you used to maintain on 2500 calories a day, you would now maintain on 2350 calories a day. If you go back to eating 2500 calories a day, you will have a surplus of 150 calories a day that will lead to addtional weight gain.
I'm a firm believer in this theory because it matches my life's experience with dieting. I always lost weight, stopped "dieting," regained the weight I lost and gained MORE WEIGHT on top of it. Every time, going from 140 lbs to 200 lbs. Dieting made me fat.
SIX YEARS at maintenance weight!
My very long weight loss story
"I saw an angel in the marble and I chiseled until I set it free."