Your allergist sounds right to me. I believe those foods are related to pollen.
I went through that with berries. I convinced myself that my throat was tingly and constricting every time I ate any kind of berry. It also moved onto to other foods. It was just seasonal allergies and maybe some anxiety/acid reflux mixed into the equation.
I got over it because my son developed food allergies. I witnessed him having a true allergic reaction to food and realized I was not having that at all. An allergic reaction to food involves things like uncontrollable coughing, swelling, vomiting and much worse. Trust me, if you have a food allergy, you know it. Now, I just worry about my son all the time instead of my imaginary food allergy.
If you are really worried, you can take a skin prick test to put the fear behind you.