I think you need to think about the differences between a holiday let and a second home for you/your family to use. They're VERY different.
From a tax point of view, you can't claim costs for the proportion of time used by family or otherwise not available to let. That's a big mistake a lot of people make and HMRC are very hot on it. So you have to do your sums on the basis that you have to pro-rata all the costs, i.e. mortgage interest, utilities, contents, furniture, fittings, insurance, cleaning fees etc between the time you/your family use it (not allowable) and the time it's let out or available for letting.
Unless you live nearby and have plenty of time to manage it yourself, plus dealing with marketing/bookings admin, etc., you'd need to engage an agent to look after it for you. An agent isn't going to be too keen if it's not available to be let out during the busy peaks as they want their commission and management fees, so unless you're only intending to occupy it yourself (or family) in the off peak months, you may struggle to get an agent on board anyway. Some have pretty fixed contracts as to limits on how many weeks of the year you make it unavailable to let.
Who's going to clean it? You really should get professionals to maintain a decent standard and get them to do proper cleans after you/your family have stayed in it.
Holiday lets are very competitive - without getting an agent on board (i.e. Sykes, Cottages.com etc) you may struggle to get it rented out enough. Yes, you may not want it rented too much, but if you're not on one of the big booking sites, you may struggle to rent it out at all without a lot of hard work marketing it, creating your own website etc. You also need to work on getting very high ratings from past users, that means making it a pretty high standard of F&F, equipment, contents, cutlery & crockery and kitchen utensils, dishwasher, washing machine, sky tv, wifi, excellent cleanliness, etc - a few poor reviews and you're screwed really.