In my home, the home that I pay for, the home that is mine, I expect to be in control of what happens. I decide who comes over. I decide who is allowed inside. I decide how long they can stay, if they can stay at all, and I decide when they leave. (Inasmuch as, "it's time for you to go now", as opposed to "no, you can't leave yet, I'm not finished with you" lol. I'm not into kidnapping!!)
It is not rude, selfish, or entitled of me to expect this in my home.
Now consider this.
Imagine that I decide I fancy paying a call on LiviaDrusillaAugusta. I rock up to her doorstep, and I pound on the door for fifteen minutes at a stretch without her answering. Then, despite this lack of response, I have myself a wander around her property, peering in windows in the hopes of spotting her. If/when I spot her, I leverage that sighting into a forced invitation inside for a cup of tea.
In this scenario, my behavior would not be reasonable in the slightest. And frankly, I would not be at all surprised if she called the police and reported a stalker/prowler/intruder.
LiviaDrusillaAugusta (and I'm sorry to use you as an example here!!) has the exact same right that I have, to decide what happens in her own home. Just like the OP should. That guy, her DH's work colleage, he too gets to decide what happens in HIS home. But he doesn't get to decide about anywhere else. He certainly does not have some inalienable right to rock on up to OP's house and hammer on the door until she lets him in.
The point is, if you want to let people into your home at all hours, if you like visitors, if you don't mind popper-inners, if your kettle is always on and your home is always presentable... well, more power to you. You can, and should, do exactly what you want. In your home, that you pay for, and you have control over.
But don't you dare tell me, or anyone else, that we're somehow "rude" for exercising the exact same right over our homes that you are - and wanting something different.