I think I have something similar to what you have. I've learned to accept that it comes and goes. When it happens it seems scary at first but then I realize that the brain is an organ whose processes are affected by chemical releases and that at that moment my body released ALOT of those hormones. My doctors don't seem concerned after a few blood tests, so I assume my anxiety plays a part in this.
Does your deja vu accompany any other symptoms?