That's a really good point Hopel, that he's being reminded...
How about this. He made an error of judgement in believing it was you. He recognises that he got this wrong and maybe even feels bad about it BUT he HATES being wrong and, even more that, he REALLY HATES admitting that he was wrong. Doing so makes him feel small, squirmy, exposed and like his balls are shrivelling up in public.
You are a constant reminder of his feeling of humiliation. You are his humiliation. So, he either banishes you from his life, so he never has to face that feeling again, or he has to locate the problem with you.
Even though he knows he can't really deflect the blame onto you, he can project his feelings of guilt onto you. You could have done it, because you're the kind of person who might do that, so you might as well have done.
So it wasn't his mistake, or a mistake at all. So he has nothing to feel bad about and can carry on his life as the tip-top, upstanding, outstanding, decisive, incisive, sharp, witty, grade-A, alpha-male superstar he believes himself to be.
The deal is, that if you want to stay in his life, you have to accept this story too. He's right, you're inherently wrong (and will be wrong again, whenever else he cocks something up, including his behaviour towards you).