I think that no matter where you buy your first house, there are likely to be some problems, because first time buyers rarely have the sort of cash to buy a large, well maintained, detached house, with a large secluded garden, in a lovely area with great neighbours. In any case, even if money is no object, neighbours can still cause problems.
As a first time neighbour you will probably have to compromise on some things. You're really fortunate to be able to afford a detached property, and maybe the neighbour's overgrown garden has put other people off, so you might be able to buy it for less than similar houses. If the gardens are a reasonable size there are lots of ways to screen off the view of the neighbour's garden, but a nicer solution would be to get to know your neighbour, and if he's a nice chap but just can't manage the garden, offer help to keep it in order. In time, perhaps he will move away if he can't cope with the house/garden, but if not perhaps you could offer to buy part of his back garden - that could work well for you, if you enjoy gardening.
Overgrown gardens are good for wildlife, but well tended gardens can also attract wildlife, without the abundance of self seeding weeds and invasive species such as bindweed. There is room for compromise with large gardens, as you have room to leave a patch to go wild, perhaps right at the back of the garden.
Provided the garden isn't piled high with rubbish/rusty cars etc it wouldn't be a deal breaker for me, but I would steer clear if it was basically a rubbish dumping ground.