I feel the same way, genetically modified anything is pretty scary. We use soy products in moderation and processed soy (tofu dogs, etc.) extremely moderately (at my DS' birthday, he had some extremely picky eaters come...I bought a package of hotdogs to keep the beast quiet. DS ate a tofudog and nobody was the wiser). We do try to avoid anything with overly processed soy products - it's a great, cheap filler....like HFCS is to sweetener.
DS was also allergic to regular milk when he was younger and we gave him soy and goats milk. The soy was given to him much to MIL's chagrin as she said she "heard it would give them boobs". I found so many articles for and against the use of soy that it was difficult to make any sort of informed decision on the matter. It seems there is always a website out there to corroborate any sort of stance one might take on an issue.