Getting of Mirtazapine (Remeron) should be a good option for you, considering it is well known to cause weight gain, and since weight gain is a concern for you (believe it or not, for some people, it isn't) then there's no point in getting fat just to manage your anxiety. Likewise a few current AZ posters can testify how unexpectantly this med can just "poop out" on you. However the medication does alleviate anxiety, so if you aren't really experiencing panic attacks at the moment, you really should be giving more credit to Remeron than you think. You probably will experience increased anxiety once you get off it, unless you get on a different anti anxiety medication to manage your anxiety disorder. You could decide to take another med later, after getting off Remeron, but considering your panic attack history, you're really playing russian roulette to not have a medication to ground you.
Klonopin is a good med to alleviate anxiety, particularly GABBA levels, in order for you to calm down. If your psychiatrist is willing to write you a never ending prescription of Klonopin, you may have that med to manage your anxiety. Do note the withdraw levels off a benzodiazepine are not very pleasant, so you should expect to be on a benzodiazepine for a long time.
However, I remember reading your previous posts about issues with ADHD, so there are issues beyond your brain's lack of glucocorticoid receptors, which could be supplied by a benzodiazepine like Klonopin (Clonazepam). Also if you're taking Guanfacine, you're taking a blood pressure medication to work as a psychiatric medication. I guess if it works for now, you can choose to stay on it, however I don't think you'll get optimum results with just Guanfacine and Klonopin.
If you have an anxiety disorder, and you've experienced panic attacks from hell (and I have too), then you want to be on the best medication type, so you don't experience them again. I think your best is to get on a antidepressant, or a SSRI. These medications are the best treatments for chronic anxiety disorders. In addition, antidepressants affect the two hippocampi, which are important for learning, as well as repair damage around your prefrontal cortex. The reason why this is important for you is that an antidepressant, at the correct dosage, could do a better job in managing the problems with focus and your ability to remember learning material, that you maybe having in school. Klonopin and Guanfacine will not assist in this.
If you're resistant to SSRIs because of their sexual side effects, raise those concerns with your psychiatrist, and the two of you can work around this obstacle. Ian can also direct you to a post he created on Anxiety Zone about working around sexual dysfunction with antidepressants. Also you may want to google herbs that affect the libido, which some antidepressant users claim to have upped their sexual drive. If you want to pursue them, you can ask your psychiatrist about his recommendations.