This man has failed to give his child any sort of boundaries growing up and is now reaping the "rewards" - a perpetual overgrown child who will be financially and emotionally dependent on him for as long as he continues to provide.
Your BF takes you out for your birthday - call from DS saying he's had an emergency of some sort and mum cannot for whatever reason help - BF apologises to you and cuts date short, rushes home to genuine emergency, then rearranges date with you for another day. All that would be fine.
BF makes a date to see you, son says "Dad I want to watch TV with you tonight", BF doesn't even say "No I've got plans with SaltnShake tonight but why don't you and me watch X film tomorrow night with some beers and a pizza" - no he just dumps you, in favour of an adult who he sees literally every day who then doesn't want to spend time with him anyway? Yeah, no.
If the sex is good then maybe keep him as a booty call but TBH his toxic parenting would be an instant fanny dryer for me. I wouldn't even keep him as a friend.
I genuinely feel sorry for the lad. At 21 he's too young and sheltered to realise how his parents are damaging him. He probably has got depression - who wouldn't be depressed with no job, no life outside of his parents, no independence, seemingly no future?
Bin him off OP. You can do so much better than this. Wouldn't Xmas on your own or with your kids feel better than being sidelined by this chump?