You're lucky he's not Moroccan. They spend very little time at home and are usually parked in a cafe playing backgammon, drinking endless coffee or tea and watching the world go by. After work and on days off. They may be hands-on in other ways, and the younger generation much more so, but there is still a culture of the man spends time with other men, and the woman sorts out the children.
My Dad ran his own business and was usually out the door well before 7 am and home late evening. My Mum and Grandmother took care of us and did everything in the home, including caring for him, as he was the sole earner. Most evenings, after he had eaten, he would disappear to the pub and be there until closing time. We hardly saw him. On Sunday he would be at the pub twice. All afternoon, home, sleep, eat, then back to the pub again.
A lot of older generation were like that. Not saying it was right, but it didn't not work for us as a family unit. My Mum seemed happy not to have to go out to work and to focus on the home with her mother and us. I guess they both made that decision. She did not seem bothered at all that he was rarely around.
Different times now, but I do still think if someone is the sole earner, then they absolutely need time to blow off steam. You do too, as a mum, you need that too. The bigger issue here seems to be that your son is too nervous to be left with a baby-sitter. If you had someone who he would be happy with, or a family member or friend he was comfortable with, you could meet with him and/or do your own thing.
My Mum had her own mother to take care of us and that meant my parents could go out together and on some evenings, my Mum did her own thing. That was the old-fashioned way, close-knit communities, it was easier in some ways. I think you need to get your son used to being taken care of by someone else. You have a baby on the way and your focus will also be on the new baby. How will your son be then? He will have to get used to less attention at some point.
I also hope your partner has paternity leave?