You are in absolutely no way obliged to have ANYONE with you who would make you uncomfortable or cause you stress.
It MAY however be a good idea to make this clear to him in advance, otherwise you might find yourself in labour and arguing with him, then simply giving in because you're in too much pain.
It will largely depend on what he is like however, and no-one here will know that better than you.
I would suggest saying to him that you want to be focusing on the labour and giving birth, and don't want anyone with you. That means that he would be able to look after the other child whilst you are in hospital, so makes perfect sense.
If you try and approach it from a "this is the practical solution" angle, rather than the "fuck off I don't want you angle" it will probably be much easier for him to accept and maybe avoid arguments.
If he tries to push it saying that you could leave DS with his family, then you will need to put your foot down and say "no, I want you to stay with him, keep him distracted, etc. You will be more helpful with him than you would with me".
If he STILL won't listen, then make it clear you want to do this alone, you have the right to get through the labour without a big source of stress there with you, and ask him to respect that.
Then make it very very clear to the midwives.