I actually think you are in the right over this OP - and yes, if there is a chance to get this in front of a court then bring it on. Just be careful to put in writing (I would email him) what you are objecting to.
The issue you are facing is a territorial one by the new partner - and she is starting in on all that very early, so it is a thing you need to demonstrate you will not stand for. There was a thread a few days ago where the new wife had just had a baby and was wanting the ex to accept they were now "one family", which basically meant the Ex was supposed to accept her boys were now part of a new family she was not part of.
What always surprises me in these situations is it is less often the ex wife or partner who has issues with the new relationship, and very often the new partner who has issues with the old one. Effectively they want and try to "unhappen" it in so far as possible. Usually the dc have to be accepted up to a point, but the ex is usually the first limb to cut off. It's an insecurity thing, and while some women make fantastic SMs and can bring real support and depth to a family, it surprises me how many people who are challenged by the existence of a previous family will willingly walk into these situations. You would think they would put down their roots in fresh ground if it bothered them - or maybe they actually enjoy the strangling-off process, who knows! But that is what she is up to. There is no other good reason you cannot collect your son. It is about reinforcing to you you are unwanted, powerless, and she calls the shots in his world because this man is now her turf (lucky her by the sounds of it! 🙄)
I would suggest that until you are allowed to collect your son from the front door like a normal citizen, your DH comes and spends time with your son round at yours. So not witholding access at all. Failing that he does the back and forth, but I would actually feel uncomfortable having such a young child in a house with a woman who clearly has such malign feelings.