You could have what is called oral allergy syndrome. It varies greatly in severity. Mine is very severe- I carry an epi pen because even cross-contamination can lead to a severe reaction for me. However for my little brother, bananas just make his throat a little itchy.
Look it up online though- my doctor actually wrote and maintains the about.com page on it!