Even taking caffeine pills when I worked swing and double shifts, I've never had caffeine jitters. At most, caffeine makes me feel a bit restless, like there's something I should be doing instead of whatever I am doing.
I've been on a couple weight loss meds very similar to phentermine, and
I did experience jitters, anxiety and racing heartbeat. I was in my late teens and early twenties, and when I told the doctor, he asked if the sensation was uncomfortable. I think I said, "not uncomfortable, just really weird." He told me that if it was just weird, not to worry about it, but if it hurt or was uncomfortable or frightening to stop taking the pills and call him.
I'm not sure that was the best advice, but things were different then (25 to 30 years ago), and "risky" weight loss (even for a teenager) was seen as prefereable to morbid obesity. Well, maybe not all that different as I've been hearing quite a lot about teen liposuction and wls.
At any rate, I did get used to the drugs to where I didn't notice the effects, but others around me did. I lost a job on a hospital mental health and alcohol treatment floor, and I suspect it had something to do with my being on the medication. My boss asked if I was on (illegal) drugs, so I had to explain what I was taking - and that I was under close medical supervision. She seemed ok with it, but after that we had personality clash after personality clash. The stuff she would write up reprimands for, was rather petty -taking "her" pencil and not returning it (a standard #2 pencil, one of two dozen in a pencil cup - not a personalized memento), misspelling a word on the chalkboard during a class (her grammar and spelling were atrocious, and there were dozens of errors on the class handouts - maybe it didn't help that after she griped about my spelling error, every time I found an error, I'd circle it and put it on her desk with a note - but hey when she lectured me, she did say that "every small error diminishes our credibility with the patients").
Here's a link to a page on phentermine's possible side effects (remember you might not experience any or may find them tolerable, so talk to your doctor about your concerns).