Gosh, yes, this! People should say this more often. When I was younger I didn't even think it was possible. I thought babies had to have their father's surname. Unless perhaps you didn't know who he was, if he was a one night stand and you couldn't track him down via mutual friends.
It is difficult because you feel unable to talk to family or friends because you don't want to taint how they see your partner
I had this thought when deciding whether to leave my ex. I was the boiled frog and didn't know he was abusive, it crept up on me so gradually. Then one day it escalated.
I thought the same as you, that I couldn't tell most people, because if I decided to stay they wouldn't understand and would be angry with him. Luckily (for me, not them!) I had a few friends who had been in abusive relationships, so they were the ones I told and asked advice from because I knew they'd understand and not judge me.
Years later I can see that thought of yours for what it is, OP. It's twisted thinking. If telling something about him to your friends and family would make them angry with him, why is that? It's because they're angry at the way he's treating you and they know you don't deserve it (nobody deserves it). The twisted thinking has us doubting ourselves. Not that we consciously think we deserve bad treatment necessarily, but that we question whether we're being reasonable and whether we're in the right. Others who aren't caught up in the relationship and aren't experiencing the twisted thinking can see so obviously that we're definitely right and totally reasonable. That's why they'd be angry.
So I flip it round and now I have it as a marker. If someone does something questionable, I think would I be happy to tell my friends and family about this or would it make them angry with him/the situation? If it would make them angry, I walk away, because it's a marker that he is treating me badly.
What kind of weirdo wants other men shagging their pregnant wife?
The kind who is happy to effectively pimp her out, his payment being that he gets to shag the other man's partner.
Swinging is not something that should ever be done reluctantly or to appease someone else OP.
Your feelings are what to focus on here. In a relationship you should be feeling loved, safe, secure, contentment. Instead he's got you feeling anxious, stressed, confused, questioning yourself.