This sounds very similar to something I'm going through right now. I was out drinking with someone I've been on and off with for awhile, he had to leave so I went home too and he said he 'might' come over later (it was like 1 am at this point). I went home, fell asleep on the couch and woke up the next morning in my bed in sweats (I barely remember taking a shower and changing). Of course I immediately freaked out that I'd blacked out, he'd come over and we'd had unprotected sex. Even though there was no evidence of this whatsoever. There were no texts on my phone (I would have had to buzz him into the building and he doesn't know which apt. I live in). After obsessing for two days and googling how soon a pregnancy test will be accurate, I finally asked him if he came over. And of course he said "no, what the hell are you talking about?" So now he thinks I'm a freak but this should have made me feel better right? It helped a little, but I still can't shake this weird nagging feeling like something's not right. What if he's lying? I even went so far as to consider asking the building manager to review the security tapes to make sure he was never in the building! It's so frustrating! And the worst part is I know deep down it's crazy, but it feels so real.
Anyway, sorry that doesn't really help you at all. But your story really reminded me of myself so I had to share.