I've been a landlord for 10 years now. Not through choice, I moved to look after a seriously ill family member, and it was midst of the recession and after putting my house on the market for months no one wanted to buy it, not even for a 'bucket price'. I found it daunting, but it's actually been OK.
Really, sod anyone who would judge you and doesn't know your circumstances. I was a single parent, young challenging kid, not able to work full-time, scraping together a living, no recourse to benefits, caring for sick relative, living in one room with my child in sick relatives house, crap Ex... Really quite a hard and poverty stricken time of my life - when I was a Landlord!! At one point I thought it might be easier just to default on the mortgage and get rid of the house - become bankrupt.
BUT I didn't, I got a good agent and my negative equity house and rented it out - I asked for families - and I made up my mind to make sure that the house was somewhere I'd be happy to live too.
I got screwed over by one or two people, and had to go into massive debt to do so. There are a few tenants who default on rent - that made me so angry. I was absolutely in tears one time, I had to find over a 1000 pounds to cover the rent once, put it on a credit card, worried that me and my child were going to cope that month. We are not all fat cats with plenty of money to spare.
But, as I still couldn't sell the house, I struggled on and now it's really fine. I always make sure any repairs are done straight away, I look after the house, but as it is through an agent I get a bit of a buffer - not loads of phone calls all the time and it is clear from the outset what is my responsibility and what is their responsibility. Some things you have to take on the chin, fences I have replaced that were probably damaged by the tenants, just not worth the hassle going through court. Cost in repairs and just do them, don't try to skim and cut corners. I get a gas and electricity check yearly, I got the firemen to check the property too before I let it out. I charge a rent that is a bit below what I could get for it, a nicer house than the usual soulless rental, and a reputation for being fair, and in return I can have a bit more choice of tenant.
Just do it, you won't regret it! And in return you can be a nice landlord.