followed by telling me to say sorry.
Which you did, even though you weren't.
Eventually the reason for his recent behaviour came to light. The lock was changed as it jammed. I did not give him another key. He has needed to be let in.
So he has been punishing you because you have obstructed his cocklodging.
although he spends every night and weekend here, he goes from his business to his parents for tea and then to me late in the evening, and is here weekends.
So he sponges off his parents as well.
Each time he butted in, talked over me, nearly walked away...silly man
This isn't a silly man. This is a man with a massive sense of entitlement, who is manipulating you into providing him with free food and lodging, and the emotional thrill of having someone beg for his time and beg harder the worse he treats them. Making you feel bad makes him feel good.
If you must give it another go (and why, I have no idea), instead of allowing him to constantly test the boundaries, I would test them yourself. That is, you put your house off limits. The relationship will be conducted through a series of dates, half of which you will arrange and pay for, and half of which he will arrange and pay for. You will not contact him to prompt him to arrange 'his' dates, that will be on him.
I suspect you will see a diminishing level of interest in the relationship once he has to put in any work. And you need to appreciate you are worth more than that.