Yeah I agree. I suffer from a chronic, untreatable, incurable, physical illness. I can't just 'let go' of that. I need to treat my suffering with care. Moreover, having just broken up form my girlfriend, there is a time for mourning and pain. I should not, and would not want to, attempt to force my attention away from my upset and into direct present moment, sensory experience. My experience is the suffering, I need to allow that and treat myself with kindness.
I was recently listening to a dharma talk by Gil Fronsdal, a guy who is an ordained Zen and Theravada monk and has practiced for years in Japan and Burma. He noted that the Buddha suffered from chronic back pain (from the asceticism of his youth, no doubt) and would often have to forgo teaching due to the pain. This actually really shocked me. He could not simply let go of his suffering. He could be with it, and he could treat himself with kindness. The whole point of his enlightenment was that he ended his antagonistic relationship with suffering. He learned to walk a middle path between asceticism and indulgence, between aversion and craving. Just because you let go of something, it doesn't mean it has to go anywhere.
Anyway, I just wanted to share this because it was a pitfall that I fell into hook, line and sinker when I started my practice, and it caused a huge amount of tension and anguish.I also recognise that in the context of anxiety people want to use mindfulness to circumvent their suffering. It is impossible to circumvent suffering, however it is possible to use mindfulness as a guide to navigate suffering.
That's not saying that you can't be mindful of the beauty of present moment experience whilst this is happening, it's just that you must allow for it to happen and be kind to it.
In addition to Jack Kornfield's A Path With Heart, I also recommend Sharon Salzberg's Lovingkindness: The Revolutionary Art of Happiness.