My sister and her dh live in Swansea, on the Killay/Gower side, but still in the city IYKWIM. They bought their 3 bed victorian terrace about a year ago and paid £105,000 for it, cos it is in a sought-after are. Would have loved Mumbles but couldn't afford there. Prices vary wildly according to area of the city, so be careful with v cheap places!
They love the area- both big surfers/camper vanners and spend most summer weekends out on the Gower beaches. City itself is pretty rundown and there can be some anti-English sentiment esp. as many of the English are students so ususal town/gown problems. That said, I've always found people v friendly and helpful.
Might be worth spending a few days down there in a B&B to scout round and see what you make of area before committing to the move.HTH.