I don't see that it's a big issue really. The risks are no greater IMO than the babysitter being inn the same house, and to me, it's just like a bigger house. The babysitter surely would pop in periodically to check everything's ok.
Yes there theoretically could be a burglar - in which case, the adult is far more at risk than a child.
Yes the house could catch fire, but if it hasn't set off the alarm its entirely possible someone in the house wouldn't notice/be able to get upstairs in time.
Its a risk, but so is the babysitter having a heart attack (whenever they ate), or breaking their leg on the stairs.
Yes, the child may have to wait about a minute longer for soothing, but no more so than if the babysitter had gone to the loo. Presumably the op showers, and it's possible that baby wakes up, or takes a few moments to stir in the night when awoken, or has to disentangle herself from sex if baby cries. Crying for a minute or two (which is all it would be, if that), is not going to harm a child.
I think some of the concerns here have been rather melodramatic personally.
I think the issue is that it feels wrong, but that doesn't mean it's more risky.