Personally I think MN really hates anyone having a new relationship. Especially if they have children.
So far the ONLY thing that has happened that's a bit bizarre is that someone has sent you a message saying 'f'. ... and given you a reason for it. For this ONE thing, you have people seeing a whole carnival bunting of red flags 🤔..
He sounds like a lovely guy OP. As far as I can understand, it hasn't happened again and he personally hasn't done anything to upset you. He appears kind, open (given you a key to his flat) and understanding that your kids are your priority as far as seeing each other goes.
Personally, having friends OLD , this guy sounds a bloody site better than most of the game players on dating sites. I would keep it exactly as you are. See him without the kids. Keep family life and him separate until doubts are gone... and if they aren't , then move on, if they are move forward with introducing to children when YOU are ready.
It's all very well the MN mafia going on about 'no introductions for years' but the reality is that you will eventually need to know, not only how he is with you and no competition for attention from dcs, THAT for me would be the thing that tells if he is a keeper. Not when you both have only each other to focus on. Family life with dc and a step-father figure is the real test.