Can I offer you some advice which I took myself 20 years ago, after my last marriage had ended and when DD was very little?
Let him see his child as little as he wants.
Truly, I know this hurts, and goes against the grain.
But he will do that anyway- all the rows you have and all the agreements wont make him less selfish, wont make him put his child first, and wont make him listen to what is fair. It just wont.
All it would do to try to get him to increase access, is annoy and frustrate yourself.
So just say something like, yes, if thats what you want. I think it is poor parenting and totally selfish of you, but do what you want, and have him x days/hours.
Then move on with your life with no hope of better. You and DS will fill the gap he left, and as DS gets older Mr Selfish will either carry on as he is or will try to initiate more contact with his 'cool teenager'. Because he is a prat. And at that point, he will get short shrift from the child.
My daughter is now nearly 25, and sees her dad sometimes. He did try, unsuccessfully, to go 'clubbing' with her when she was 18 or so! She does love him and has him as part of her life, but she gives him the size of space in it, that he gave her as a child, IYSWIM. He has made that completely natural for her, she doesnt even see it herself, I dont think. So if she has a week off at Xmas, she arranges to spend 6 days with me and one with him, and is quite confused and surprised if he is huffy!