Yes it is rude but sometimes I wonder whether they drive up, look at the house and know it's not suitable.
We had a specification when looking for our house that it had to have a downstairs loo because my MIL, could not climb stairs (she could not visit our old house for this reason as it only had a loo upstairs).
We were sent to look at one house (had not got picture) and arrived to find it had about six massive steep steps up to the front door. But it had a downstairs loo! We looked at it anyway out of politeness, but I did wonder if we had raised the guy's hopes because we knew there was no way we would be buying it.
Another one we were sent to view turned out to have no out front parking, you could only park at the back of the house and walk down a set of stepping stones through the back garden. Never mind MIL - I had a toddler who wasn't walking and was heavily pregnant. Again, we knew before we went in that we wouldn't buy it because the house we were in you couldn't park outside sometimes and it was an absolute pain in the ringpiece, and this was 10 times worse.
Both were lovely houses, but just not practical for us.
But as I say, it might have been kinder to drive away and not view - but it would have been rude. 