YANBU at all. If you do not want him there you should not have him there. It's not good for you or the baby.
When people ask, act shocked that they've asked.
Taken aback look and "Of course not. Good grief. He's not seeing me naked ever again."
Other potential responses:
"He can't be bothered providing support while I'm pregnant, he certainly doesn't get spectator rights at the birth."
Or "The baby has rights, he and I have responsibilities to the baby. I will be relieved if he lives up to his, but I'm not holding my breath."
Or "Having him there would be stressful which is bad for the baby and me during the birth. I can't believe he'd want to be there over putting his child's best interests first."
Having him come and see the baby immediately afterwards once you're covered up is different and much more acceptable. But even then, only if he shows he can be civil and responsible about it, put his child's needs first and be nice to you. Him being flakey and difficult now does not provide much hope for that, though there may still some time for him to step up.