these are challenging even for very in shape athletes. we used to do them as part of track practice. starting out i would just do like you are doing. try to stay as long as you can. you can try to shift the weight slightly from one leg to the other. try to think about something else. the rest period depends alot on the person. you might try doing a few in the morning, and a few more later in the day. it's one of those anywhere exercises.
i'm guessing you are already doing regular squats? you might also try lunges and lateral lunges. also called side lunges. and speed skaters. it's a sort of hopping side to side with your feet spread apart. the traditional squat, lunges, and wall squats all work your muscles in similar ways. you are moving up and down. then the side/lateral lunge and speed skaters work your muscles going side to side. believe me this makes a difference! the first time i did these i got major doms, after doing squats/lunges regularly for months and months. so as with everything, try your best, but look out for those doms, which i actually look forward to usually, but certain muscles make it hard to sit down lol.
so anyway, don't sweat it
even hard core athlets moan about how hard wall squats are lol.