We were brought up to take our shoes off at the door, and I do see it as cleaner and more comfortable. If you're relaxing on the sofa and want to curl your leg under you, or put your feet up on a stool or something I wouldn't want shoes that have been walking in who-knows-what all over my furniture. My kids constantly end up with chewing gum/squashed up sweets/animal poo on their shoes, and I'm sure it makes it's way onto adult shoes too.
I actually find it quite difficult to feel relaxed indoors in shoes, it feels wrong, like I'm sitting about with my coat on, but this is probably because it's what I'm used to.
We live in rainy Northern Ireland, so the vast majority of the time shoes will be wet and mucky, and your feet will be much cleaner and drier. Presumably people will be wearing fresh socks that day or have showered in the last 24 hours vs shoes that very rarely or never get washed.
I have never asked guests to take their shoes off, although a lot will when they see that that's what we prefer in our house. I can see why some people might consider it rude to be asked. It wouldn't really bother me if a guest felt more comfortable with their shoes on, so long as they weren't dirty.
I would always take shoes off when visiting someone (apart from my aunt's whose floors are filthy, I want as much distance between them and the soles of my feet as possible).
I carry a clean pair of socks in my handbag in case I want to try a pair of shoes on when I'm out shopping and not wearing socks, if my feet got really cold when I was at someone's house I would just pop those on.