Right.
So, presumably this is his day to look after his child? His one day in 7?
And he either didn't think about it at all (selfish/thoughtless/inconsiderate) or he did and decided to leave it to you (sexist, misogynistic, selfish/thoughtless/inconsiderate).
When you didn't accommodate his fuck up with open arms, he has chosen to gas light you to make you think you're in the wrong, rather than take any ownership himself (lack of self awareness/always right) and, this is abhorrent, call you names which are completely and utterly unacceptable.
This is LTB territory op, so many horrible character traits being put on display in one snapshot of your life.
You are not remotely unreasonable to say no, absolutely not.
I suspect though he'll just go about his day and leave the women to sort out his shit. (Possibly the poor actual mother who would desperately need a break from doing basically all the parenting of a difficult child).
It's too late for you now op, sorry, but for others - a man who barely sees his child is a massive red flag, do not marry/get pregnant with one, for you'll see very quickly why their previous relationship ended.