I think I'm within my rights to push back if he wants contact.
Actually you aren't. Sorry.
As your ex is named on your DS's birth certificate he has legal parental responsibility which is equal to yours. It is highly unlikely that a court wouldn't award him contact if he decided to pursue it. If I recall correctly your son is only very young, so a father wanting to get involved this early in his life is not likely to be seen as being "in and out of his life" the same way as if he was several years older and had frequently had contact then stopped.
By not facilitating contact between your DS and his father you are likely to look unreasonable in Court and it is more likely your ex would be awarded contact arrangements you wouldn't be happy with. I would also add that a Court is likely to be rather unhappy if your DS was calling someone else "dad" or had someone else presented to him as his dad. In fact they can (and sometimes do) order that this stop. They will also take a dim view of you not providing your new address to your ex as soon as you moved. As he has PR he is legally entitled to know where his son is living unless knowing this would put you or your DS at risk. If he asks for the address and your friend passses this onto you, you must provide it (unless you are at risk).
Also your DP cannot adopt your DS without everyone with legal PR (so both you and your ex) signing consent - and a judge is unlikely to take your ex's consent without him having had appropriate legal advice. In any case step-parent adoption is usually discouraged now with Courts instead preferring to award a step-parent PR agreement, which would mean you, your ex and your DP would all have PR.
Sorry if none of this is what you want to hear.
OP on your last post I said I felt that while I could empathise with your SIL's concerns but ultimately felt you were both adults and could make your own minds up. I'm afraid this post has made me understand your SIL's concerns even more and I'm sorry but I'm going to be quite blunt now.
I completely understand that what you are trying to do is make a happy family for yourself and your DS but at the moment this is a powder keg ready to blow up in your faces. Regardless of what you think of your ex, he is your DS's father - legally and biologically and his PR is equal to yours. You can't deny him contact just because you don't like him and you think it would be better for your son to grow up with just you and your DP.
I'm not denying that your DP may well turn out to be the responsible reliable male in his life and that it may well be your DP he has the closest relationship with. But when you allowed your ex to sign that birth certificate you gave him equal rights to you and you are not able to take those away.
Also bear in mind that if you got hit by a bus tomorrow your DS would be handed over to the only remaining person with PR - his father. Yes your DP could apply for the right to fight this through a Court but that would take time.
Sorry OP. I know you mean well but you can't just create a new family for your DS replacing the old one, when someone in the old one has PR.