What you describe here isn't unusual even for families where the parents are still together. All of this normally falls on the mum, however wrong that is. The fact your ex isn't very organised or good at taking on the mental load of parenthood doesn't mean he's not a fantastic dad, everything else about him suggests he is.
You are never going to get a very balanced view on mumsnet but I have to say I agree with the majority of people on here. Forget the social services bit, that's not going to happen, but your partner is seriously damaging your son because no matter what he does not say in front of him, his resentment will be obvious to your son. I find it unbearably sad that your son has gone from wanting to spend all his time with your partner to now not wanting to.
My ex has a daughter, it was very complicated circumstances and an extremely stressful situation, seeing her at the weekends brought a lot of stress, upset and inconvenience. However, I went out of my way to build a strong relationship with her, to make her feel loved and welcome every second and be clear that I/we would like to see more of her. And I did/do genuinely love her.
To not want your son there at weekends is frankly despicable. I could not live with a man that treated my child like that. Let's face it, kids don't get easier as they get older, this isn't going to get better!
I hear what you're saying about people in your ear telling you this is normal but sorry, that's utter garbage. You should do less listening to complete strangers telling you it's ok for your child to be unwanted in your house and more thinking about the feelings of a small child. I could not stay with a man that acted like that, it makes me feel sick, and whilst you understandably don't want your youngest to have a "broken family" that is way better than effectively sacrificing the feelings and happiness of your older child.