First of all, like everyone is saying, there's a common misconception that only the outside is listed. The inside and the surroundings are also listed, and need Listed Buildings Consent for changes.
My sister is a planning officer dealing with listed buildings, and her advice would be that you should never buy a listed building unless you are happy to live in it as it is. The issue is that you can't tell in advance what you can and can't change, because each individual house is different. It will depend on the history and architecture of the indivdual house. For example, I had our conservation officer round, and we chatted about removing the ceiling in one room. Her first response was 'no, you can't remove the ceiling, as this room would always have had a ceiling', to which I was able to say 'no, the ceiling was only put in 20 years ago before which it was an open vault' and waved the evidence in front of her, and she then said 'fine, remove the ceiling'. If I hadn't had the evidence, then I wouldn't have been able to remove the ceiling.
If you want to buy a listed building, I'd go on the SPAB Homeowners' course, which teaches you what to look for, how to maintain the house, and can save you a lot of money very quickly. If you have enough basic knowledge you can spot which tradesmen are talking bollocks and can avoid hiring them!
They are also absolute moneypits. I think you do realistically need either to be very handy with DIY yourself and willing to learn new skills and to spend your weekends on it, or you have to have a lot of money. We're budgeting on spending about £80,000 to get our house to a reasonable but not luxurious standard, with me doing quite a bit of simple labour myself.
Having said that, I have a listed house, my sister has a listed house, and we love them! But I never realised how much effort they took to look after, and how much of my time is spent on renovations and ongoing repairs.
If you have a survey, have a survey done by a historic buildings specialist - this is very important, as otherwise they may have no idea about what they're looking for.