I don't know the answer to whether or not we lose weight in our faces (enough to affect the fit of your glasses).
I do know however that from time to time I find myself having to mess with my glasses more as they keep falling and generally don't fit right. I just go and have them adjusted every so often and it fixes the issue. They just get worn out and wiggle loose; I don't know how else to explain it.
Most places (even Walmart) will do this for free, they just can't be held responsible if the break.