It would definitely be rude where I live to park your car outside anyone else's house. If we do it, we ask first and explain why it's necessary.
Obviously it is perfectly legal, but it does really piss people off and a polite knock to ask doesn't cost anything.
That reminds me of something that happened to my best friend when she was a child - decades ago, but I've never forgotten it. She was walking down a village street and an old lady physically blocked her way and stopped her, angrily demanding to know what was her reason and justification for walking there. Once she told her that she was going to her DGP's house - and identified who her DGP were and which house they lived in - the lady decreed that she was therefore allowed and let her continue on her way.
I'm sure that old lady believed my friend was being rude to walk down a public pavement that she potentially may not have had a direct residential link to. One of those two people was phenomenally rude (spoiler: not the younger one) - and I don't think most people would think she was at all reasonable; but it's not really any different in principle, be it walking down a public pavement or parking on a public road, once you self-appoint yourself the owner of somebody else's property that you somehow think should only be yours.
People can believe anything to be rude if they want to - somebody not saluting a magpie, buying two packs of sandwiches for their lunch, not dressing in an evening gown or morning suit when walking to the local shop for a pint of milk - but that's only an issue for them and doesn't mean that the person doing something that they arbitrarily consider rude needs to change their behaviour at all. Some people believe it to be 'unseemly' for women to go out to work - an opinion they're entitled to hold if they must, but not one to which you would expect to respond by giving up your job to appease their random egocentric views.
Just as the road outside our house doesn't belong to us, neither does our neighbour's house, which is actually attached to our house, so even closer to us than the road, which is separated by a pavement.
If it's weirdly accepted that you need to ask a random third party (whose only extremely tenuous 'connection' is that they happen to live nearby) for permission to use the property belonging to/controlled by another party, who has already given their express permission, that must make my neighbour's visitors the scum of the earth as not one of them has ever first knocked on our door to request our permission for them to enter the home of our neighbour who has invited them over.
Do you also knock on the door of the nearest house to ask permission before your kids use the local council-controlled park - or the house next door to the corner shop that you want to go into?
The only person being rude is the arrogant one assuming that they have exclusive rights to publicly-owned land. I'd love to see how these people would react if a member of the public tripped in a pothole on 'their' poorly-maintained bit of land - I wouldn't hold my breath waiting for them to admit liability and pay the compensation claim themselves.
There's nothing wrong with having generally-agreed parking behaviours in a residential street (although, in OP's case, why do OP and DH have less of a say in this agreement than any other resident?), but you cannot in any way dictate what others can do - residents, visitors or 'strangers' - and you cannot treat an informal agreement as legally-binding or enforce it by using intimidation.
It's also fine to politely ASK people if they would mind leaving a space for you, if you have a good reason why you think they would be willing to do so once they know - personal circumstances, delivery/removal lorry or hearse coming in the morning or whatever; or to request them not to take up two spaces with one vehicle - but the sheer arrogance of stomping over and bullying people as to how they can use public property just beggars belief.
It's odd how often the rudest people ever have a bee in their bonnet about other people - acting perfectly reasonably - being rude in their eyes.