I have ignored/avoided noticing all excess hair besides under my nose/on my upper lip, so I don't know if I am much help.
I used to remove the hair from my fingers, hands, arms and toes whenever I shaved my legs/under arms. My partner thinks that's just silly, so I don't do it all the time now. The hair on my fingers and hands is what bothers me the most.
I also find If i leave it alone it's thinner and lighter and not really such a big deal. Not like the way your legs or underarms would grow in. Obviously I understand if you really feel the need to remove the hair on your back, but I just wonder if you're just going to drive yourself crazy with it all. Try not to stress about it.