My SIL is in a very similar situation with her ex.
Boundaries. Boundaries. Boundaries. You have to be very firm on this as otherwise he'll continue to trample all over them.
So, this weekend "sorry, i can't swap weekensd as I have plans for this weekend, including work shifts."
Re birthday: "If you want to come down to see Ds on his birthday, I'm sure he'd love that. Do you want to take him out to McDonalds or something at lunch?"
If he complains he can't do this weekend AND next weekend, "well, DS would love it if you made this an early birthday weekend - he gets two lots of celebrations! Make this "your" birthday celebration and him and I will do something next week."
Re yourb irthday, "You're welcome to see him on his birthday but we have plans for the following day for mine."
Re staying at your house, "it's not practical for you to stay here when you're visiting DS. Let me know what time you want to pick him up and when you'll be bringing him back."
The thing about this sort of person is that they think their thinking is right, even though everyone else doesn't, so trying to explain it is pointless. As an example, exBIL was completely bemused when, 3 months after they broke up, we had SIL and their DC over to our house for our DS's birthday and he wasn't invited. (Not really bemused, more completely livid, frothing at the mouth, even briefly attempted to insist to SIL that he would come anyway and we'd have to let him him). As long as 2 years later, he had a complete go at her because she'd hosted a large family BBQ and not invited him and had planned a party with friends at her house and, again, had not invited him.