Your DH should have listened and tried to understand, not dismissed your concern and told you to be more like him.
I've used the same sort of agent for an empty house I'm trying to sell. I go over there for viewings, many with lone men. Because it's empty, no-one would ever arrive home to find me and I make the arrangement direct, so no-one else knows I have a viewing, unless I tell them.
Mostly I am totally fine with this, have often taken baby dd with me and do tell DP, though he's out all day. But, I do have moments of imagining 'what if', especially when walking up the narrow stairs to the attic bedroom, from which there is no other exit.
I think:
The viewer had no idea who would be showing them around when they requested a viewing.
There are no 'staff pictures' or anything showing what I look like, they'd have no idea about age or anything.
They do see my name or speak to me to arrange, so yes, they could just keep requesting viewings until they get a woman.
They would probably assume I've told someone where I was going, when, meeting who. Of course they could use a false name.
I spend time downstairs first. Easy to get out from. Establish a feeling about them there. Let them walk in front going upstairs. If not comfortable, don't go up to the attic with them - that's never actually happened. I don't go into the cellar with anyone - narrow stairs, no other exit, not much to see.
I did once ask DP to call me during a viewing because I thought a man sounded creepy on the phone. He wasn't, was very nice.
But what I really think is that, if anyone was determined to attack a woman, they could easily attack me at any point in the viewing. No-one would hear or react. Also I don't have magic danger sensors. I can tell an oddball but not an apparently normal psychopath.
So yes, there's a risk. It's more than a walking down the street risk, or having a known tradesman in a family home that others might return to. Someone predatory could exploit the woman showing an empty house scenario quite easily. I just take the risk that there are a very tiny number of such people.