I think I visited our last surgery once in the year that we were registered, and that was for a smear test. If you have good health and visit rarely for minor issues then I think location and accessibility are all that really matters. Other factors are difficult to ascertain until you have some experience of using the practice. Asking your neighbours is a good way to get a vibe, but it is very personal. Obviously if you or any of your family have any chronic illnesses it is a different matter. NHS choices is good for telling you what surgeries you can access (gives distances and whether the practice is accepting new patients).
I am happy if my local surgery has several doctors, a nurse (more than one is good), and when you visit, friendly receptionists, a good atmosphere and a clean and relatively pleasant waiting room. There are a number of quality schemes GP practices can be a part of, so certificates on the wall are worth looking out for (the schemes run by the Royal College of General Practice are excellent, and only very good practices even attempt them). You could also try looking on the local PCT's web site and see if you can access any reports on local practice performance, how they did on the Quality and Outcomes Framework, what the results of their patient surveys was etc. This information may or may not be available, depending on local arrangements.
I also use my professional networks, just to check the practice isn't on any blacklists, but this is very confidential information!
For specialists it is difficult to access much information on individuals, so I would tend to ask about as to the reputation of the department, but as all our local hospitals are major teaching and research trusts, that's not really a major problem. again I would ask around to see if anyone I knew had views, but I can only do that because I work with a bunch of opinionated doctors and nurses