Yeah, I wouldn't say it's just you @needemptyhouse as I have known a few people (over the years,) who won't do a poo when their DH/DP is in the house. Some people have been with them for 10-15 years or more. I don't know how anyone can keep this up long-term/indefinitely.
However, even though I don't think it's just you, I do think it's odd, and as has been said, quite debilitating, and not very good for your health. If you need a poo, it's very unhealthy, and could potentially cause you long-term damage to constantly hold your poo in.
Moreover, as a previous poster said, I don't understand how anyone can 'hold it in' as most of the time, when I need a poo, I have to go within 5-10 minutes. On the occasions I do hold it in, it goes 'back up' and doesn't come back down til the next day, and basically (TMI sorry) I end up with a bit of a 'backlog.' In addition, I have an achey, narky tummy! All day, and sometimes well into the next day too!
As a few posters have said, maybe put music on or run a tap or put a fan on.
Hope you can get past this, because it's really not healthy, at all...