I had one once..and I agree with you...it made me so nuts that I took it off MYSELF!!!!
I didn't know what it was at the time, and I got it at the base with pointy tweezers and took it off. Yes...it hurt, but it never came back, and I haven't had one since.
Sorry if that's TMI
I'd talk to your doctor...they're supposedly really easy to remove.