I was once ghosted by a friend, not a boyfriend. The immediate cause of it was an argument after I accidentally hurt her feelings with a flippant comment that was probably about as extreme/un-extreme as "it's not all about you." It was something I'd say to my own children in a jokey way and they would know I wasn't entirely serious, but I hadn't seen this friend for a while and had forgotten that she could be quick to take offence for unexpected reasons.
I believed for about 10 years that it was at least partly my fault - that she might have overreacted, but I had been insensitive and could have handled it better.
I've recently been in touch with someone else who knows her well, and I have discovered that she has a pattern of behaving like this with lots of people. She has been described as narcissistic. She remains friends with people as long as they make her feel good about herself by validating everything she does, but can't tolerate the slightest criticism. From what I've now heard, it's very likely that she would have been saying unpleasant things about me behind my back after she rejected my friendship, as she has done that to others.
You said you felt "like I’ve been manipulated and nobody else can see it." This is exactly how a person like this operates. By letting you know that you've terribly upset them, they make you feel uneasy about accepting that they were part of the problem too. It's only when you talk to others who have experienced the same thing that you really understand what's going on. Once could be a misunderstanding, lots of times is a person who really doesn't know how to sustain good friendships.
I don't know for sure that this man is similar, but you certainly have nothing to lose by staying away from him, and probably everything to gain. If he ever tries to patch things up with you, just let him know that you don't feel you are compatible. You don't have to give detailed reasons, and he won't believe them anyway because people like that are never in the wrong.