I think the fact that she abruptly made you stop the Xanax is the problem. I would discuss it with her.
SSRI's alone will not cause serotonin syndrome.
And, NON-SSRI's (Tricyclic ADS, MAOI's) have the potential (like SSRI's) to cause it ONLY when taken with something that's contraindicated.
Believe it or not, many meds (not just for Anxiety/Depression) have the "potential" if taken with the wrong med...and again...that would have to be over time and in large doses.
Please hang in there....don't give into the fear.
But give that doctor hell for not letting you wean from Xanax. I would definitely call her out on that chit.