The tricky bit is that you and your OH have such different views.
M’y parents have a holiday home in France. It has been fantastic for us as a family. It has a pool, plenty of space etc and since DH and I both work in schools, we’ve spent two months there every summer for the last 10 years with our two boys. Plus we often nip down out of season just for a change of scene. However, I personally would not have chosen either the location or the house itself, and I am feeling increasingly frustrated at being buried in the boring arse end of nowhere every summer. It makes it very difficult to justify a different, expensive, holiday elsewhere though.
My parents have been perfectly happy to just go there. They aren’t interested in travelling elsewhere, they have no interest in being tourists. They are perfectly happy with “same shit, different view” tbh. They wear old clothes, read, potter in the garden, do some diy / maintenance, watch tv, drink too much wine etc just as they would at home - with the added benefit of a swimming pool and sunshine 12 weeks of the year (they are retired).
So I’d have to say, on balance, that YANBU. If you want to travel elsewhere, if you want variety, if you can afford different holidays, if you’ve only got a coupe of weeks a year: don’t do it.
Maybe when you are older / retired the idea of going to the same place for longer might appeal more? 2nd home rather than holiday home?