There are indeed alot of wives tales about aspartame, check out snopes.com.
One of the most widely-distributed scare-warning letter regarding aspartame and its many permutations are mostly fiction, or twisted fact.
http://www.snopes.com/medical/toxins/aspartame.asp
(seven or eight articles debunking aspartame myths)
That does not mean that some people may not have adverse affects from it, as MelodyL admitted, only that the claims that it is generally unsafe and that there is a huge government conspiracty to cover it up is untrue.
If our country could not keep wiretapping and abuses of suspected terrorists a secret, I hardly think the hidden dangers of artificial sweeteners could be kept under wraps. But as with all conspiracy theories, there's no stopping them once they get started.
If aspartame, caffeine, milk, chocolate, potatoes, strawberries, peanuts or broccoli seems to give you a headache, memory loss, personality change, rash or other symptoms, then stop ingesting them. The leap that any of those things are poisonous for the general population is irrational, but rather than believe the same of aspartame, it is somehow easier for some people to believe that all of the study results have been tainted.
I read a reader's digest article about a man who suddenly underwent large personality changes and became suddenly abusive to his family. He went into rages, and had migraine headaches.... and finally using a symptom journal like I mentioned earlier, they discovered he had a severe and unusual reaction to dairy products. He had rarely eaten dairy in the past, but had begun using it in his cofee instead of non-dairy creamer.
Personally, I think the unproportional aspartame terror (unlike peanuts and other allergens) is largely a result not of the real symptoms that some people have, but the magnifying psychological effects of the "warning" on the labels of food containing aspartame. The warning is meant only for people with PKU. I wrote a graduate school paper on the PKU diet, so I understand why aspartame is a hazard to people with the disease, once the leading preventable cause of mental ******ation in the US. It is a disorder in which the person cannot digest certain proteins properly, and in children the byproducts build up in the brain causing ******ation. The person with PKU must follow an extremely restricted (and very unpleasant) diet until they reach an age where brain development is complete (some doctors recommend age 7, some argue into adolescence). A banana or a piece of chocolate is just as hazardous to these people, but because these natural foods contain no such warnings, there aren't any banana or chocolate scares out there.
Even stopping aspartame use may not tell you if your symptoms were caused by aspartame because of the strong possibility of coincidence or the placebo effect being involved, but the health journal really can help you determine which factors are affecting your symptoms, especially if you are able to stick to it over time, even when you are feeling better.