I think you have been unreasonable for allowing this man to get so comfortable disrespecting your son. Especially considering he is just 5 and sounds like a lovely little boy.
What would happen if you called him out on it? Would he back off? Would he escalate it? Take it out on you??
Obviously it takes time to get out of a mortgage, but if you stay, what did you expect to happen in the future with holidays? Not involve your son? What about Christmas? Family birthdays?
You said you don't get a lot of family help, but you said earlier that you moved to be closer to family for support. Not trying to contradict you as circumstances change in 5/6 years. Sometimes people don't help because they think you have it sussed and it doesn't occur to them to offer. Would people help if you asked?
Do your family know what it's like now? Or what your partner says and how he truly feels about your son?
Does your ex know? If yes, what has he said? If not, how do you think he'd react?
If your ex found a partner and had another child with her, and she didn't want your son tagging along while she plays happy families with just him and her baby, would he allow it? And what would you do? How would you feel?
I don't want to pile on, I'm just curious.
My eldest was 18 last month. I regret some decisions I made when he was younger and would give anything to go back for a re-do. You have the opportunity to change things. There's that Japanese proverb, "If you get on the wrong train, get off at the nearest station; the longer you stay, the more expensive the return trip will be". The price could be your son's mental health and your future relationship with him.
Edited to add, could you do a bit less for the bio dad? Sometimes people are useless because they get used to being over-reliant. Is it weaponised incompetence, or is he genuinely usless?