I think whilst your concern feels genuine znd no doubt is, it is also a potential escape for you from making the hard choice of telling him he needs to move out to his flat and give your space.
His children have two parents to dupport and prioritise them, and they have.
Your son has just you, and you admit that you haven't always done that.
You have rightly remedied his birthday disaster.
His children are for him and his Ex to sort.
Your son desperately needs you to be the parent and put him first.
Do not get into a discussion with him about this.
That would be so wrong and could cause your child to feel guilt for YOUR choices.
Don't burden him further with the responsibility of this relationship breakdown, that would be so very wrong.
He needs protecting.
Your language needs to be simple and unequivocally clear to your son.
The relationship was not working for YOU.
YOU decided alone to end things.
YOU alone are responsible for the end of the relationship.
Your son has no responsibility whatsoever.
Be careful OP.
Do not allow your son to be confused and to feel responsible.
There in is potential disaster.