The difference is the cause. Anxiety has a source; something identifiable that is contributing to or causing the problems. It may not be something you should be worried about, but it definitely is something.
Panic on the other hand, is the same as anxiety, but with either no real source, or is itself the source. Once you start having anxiety about anxiety, you're having panic attacks.