The problem is that even though no work is needed now, work will eventually be needed. For example, I live in a GII listed house and, after a while, the chimneys needed re-pointing. Just normal wear and tear over a number of years. I contacted the consultants who are involved when English Heritage need to fix something and got them to give me the age-appropriate recipe for the lime mortar. Just trying to do the right thing. 3 TIMES it was rejected and on the final submission, the conservation officers grudgingly agreed that this was ok but “will the sand be washed or unwashed?”. Aarrrrgh!
Shall I mention that when we needed a new roof (rain water was gushing through so we need to, you know, preserve the building) it took over a year and 10+ examples of tiles to get them to agree. And for the record, they agreed to the example we initially supplied.
It all rather depends on your conservation officers but in my experience, and the experience of various friends, conservation officers are aggressive, chippy, resentful nasty pieces of work whose principal MO is to work against you and not with you to protect our precious buildings.
I agree with your husband, I am afraid. I would never buy a listed building again. I am really sorry if this isnt the answer you were hoping for.