A LOT of weird things can cause anxiety issues. I saw a therapist for my anxiety problems a few years ago and he asked me to keep a record of when the episodes happened. Many of them were precipitated by some kind of bodily discomfort--a minor stomach ache, cramps, the beginnings of a cold.
According to him, people prone to panic and anxiety episodes often find that physical changes, even minor ones, precede them. Just knowing that helped ease my panic; knowing that fact meant I could apply a little reason to my incipient anxiety attack.
Another thing to remember (and this has helped me tremendously) is that once an anxiety episode starts up, you're stuck for a little while with the chemical cocktail that it causes your body to shoot into your bloodstream. The adrenaline doesn't just evaporate, so you'll experience some symptoms after the initial onset, even after you feel mentally calmer.
Thinking about panic attacks in terms of physiology really helped me, so I hope it's some help to you to consider this side effect of exercise pestersome, but normal.