We're in a similar situation @Clarehandaust. We bought our current house (a three bed semi-detached Georgian cottage) in 2024, having lost out on a four bed link detached Georgian cottage just around the corner. I didn't like this one from the get go but DH persuaded me to go for it.
Both needed loads of work (ours hadn't been touched since around 1970, the other had been let out for decades), but the larger one had a nicer feel and is in a far better position, overlooking a pretty square.
I said to DH I'd only agree to buying this place on the understanding that when we'd finished the (mostly DIY) renovations, we'd look at moving to something more like the other cottage.
Ours is not quite finished, but now my preferred cottage has come back onto the market, having had a bit of work done (much of which hasn't improved it - inappropriate damp-proofing, original chimney removal, cheap bathrooms installed, awful shiny grey carpets fitted...plus they've left the manky old kitchen in place). If ours was finished I'd still buy the other one - but DH now wants to stay here 🙄
The cottages are all different and that's the only one I'd want to buy/sell up for in our area, but I'm still thinking we'll sell if something suitable outside the area comes up when we're ready. I just need to persuade DH. However, having done several major renovations, I don't want another huge project like this one was!
Where we are fortunate, is that our garden is totally private - it's not overlooked by any surrounding houses - which I agree is so important (we came from 0.5 acre in a cheaper location to a courtyard!) and our walls are very thick so on our attached side we hear nothing.
For me, any new house would need to be - a bit larger, have a bigger garden, in a prettier location (we're in the same conservation area as the cottage we missed out on, but less nice street), have more original features, preferably detached (as our previous few houses were).
However, in your case, I'm not sure that moving to somewhere where the only real advantage was a non-overlooked garden would be sufficient to persuade me.....