We've lived in the house for 18 months (me, DH and 2 DC age 2 and 5). Neighbours across the drive are a couple in their late sixties. The husband has got this kind of "status" in the street. He waters everyone's plants, looks after their houses when they're on holiday, washes people's cars... etc. We've never asked him or encouraged him to do anything like this for us. I've talked to him a few times and he's come across as a racist, intolerant, set in his ways old man. So aside from a polite "hello" when I see him or his wife I don't take much to do with them. My DH is generally more chatty and has had a few more conversations with him. DH has had a few mild altercations with him when he's overstepped the mark in terms of comments about things that we do on our own property that are nothing to do with him, really trivial stuff like how we put rubbish in our bins for example. DH has always diffused them fairly quickly.
That's the background. Today, DC and I are at my parents for a festive family do. DH followed half an hour behind us. He told me that after we left neighbour approached him in the street and basically had a go at him telling him that I was not neighbourly/friendly towards him, I don't talk to him, we're nothing like the previous occupants of the house and apparently the situation has reached "boiling point". DH was a bit like 
but told him it wasn't appropriate for him to be saying things like this about me, that he doesn't know me and suggested he spoke to me directly about his concerns as he doesn't speak for me. Neighbour implied that as my husband DH should be apologising profusely for my "behaviour" and ensure that I toe the line in future
Now he's obviously a few sandwiches short of a picnic and quite frankly I want nothing more to do with him but the feminist in me is seething internally at the implication that DH should be in charge of me somehow. But part of me thinks there's no point in saying anything because he's not going to listen to anything I say and I would just end up getting more annoyed. DH thinks I should just ignore him.
WWYD? (Sorry this is so long, and a bit ridiculous!)