How was your blood pressure? Ok, we have many non-medical opinions telling you what the ER doc said is nonsense. It's not nonsense just highly unlikely. A CT of the ABD/Pelvis would only be performed in an ER setting if it was an emergency. So, if your BP wasn't in hypertensive crisis range, and your vitals were stable it could be worked up as an outpatient. I would read nothing in the lack of CT. And while what the doc said to you seems like instilling panic, he/she is doing his/her job by being thorough with educating you on follow-up. Docs don't like to miss things calling it anxiety.
With all of that said, pheo is very rare. With an established panic disorder all your symptoms fit nicely with that diagnosis. Now, if your BP was 210/100 I would reconsider. So, the question is should you see endocrine? I believe so for two reasons. One is your ER doc told you to do so, and two because you will believe you have this tumor until you put it to rest. With that said, most adrenal tumors are "benign" meaning they won't spread easily. They tend to be very treatable. I say this not because I think you have one but because I want you to know it's not a death sentence.
In a way pheochromocytoma can be a blessing if the adrenal tumor is benign. Why? Because it creates anxiety. You resect the tumor the anxiety goes away. So, get a second opinion from an endocrinologist and if he is concerned let him work you up. I suspect it will come back negative but it warrants a look.