I mean this kindly OP, but your partner sounds like an absolute arsehole. Seriously, his behaviour throughout what you've written here is horrendous, and it sounds like he has successfully convinced you it is all your fault.
Yes, you absolutely should be able to have a say in ANY change to contact. You are part of the household. You saying "of course I can't do that" as though that is morally correct is saddening and disturbing.
I cannot tell you how irritating it was, after reading how perfectly reasonable and accommodating you had been, and how much effort you have had to put in with his children and the ongoing toll they have taken on you, to get to this part:
He said I am horrible and how will he explain me shunning them. He thinks I should be happy watching the 10th performace/show, their TV shows etc and on a friday night I'm just not. I need space. So am I a terrible person to just want one night kid free?
This is gaslighting OP. You are NOT horrible to want space from his children, and he is attempting to bully you into thinking you are, even threatening to tell the children something untrue and hurtful in order to keep you in line.
His behaviour is appalling. First of all, I would have said no to them coming over those nights. It doesn't work for your schedule, they have a contact arrangement and should be with their mother that night. Failing that, I would back off all care of his children. I would tell him in no uncertain terms that though you like them, they are very hard work and not only should you not have to do that extra work on additional days you didn't agree to, you shouldn't have had to do it on the set days in the first place. He should have been taking the lead on all bedtimes etc. Thirdly, and only as a last resort, would I go back to your other flat.
NONE of those three options OP is unreasonable OP. None of them should be met with complaints or arguments from him, let alone all of them! If he can't handle these changes and it ends the relationship then so be it, it really sounds like that would be for the best. He doesn't respect you or consider you, and he gaslights you. Think about the title of your thread. "Is there something wrong with me?". That's how this man has made you feel, and no, there absolutely isn't something wrong with you.