Don't give up, hon. I've fallen off of my plan more times than I can count in the last year, but thanks to picking myself up again and NOT giving up, I've still managed to lose 30lbs in the last year and a half. Could I have lost twice that amount in the same time frame had I been on plan the entire time? Absolutely. But this has been a long process and I'm still slipping up. It's not impossible to recover from slip-ups.
Instead of going off plan and then saying "oh well", and continuing to do it, draw yourself up and start over again--it's okay to make mistakes. It doesn't make you a failure--you just have to keep striving for that goal.
Keep at it, girl. It's never too late to start again.