Hi
The most important thing of all is to make sure you look at all of the associated costs of buying to let vs a holiday home and then maybe decide on whether the difference is worth the lifestyle benefits (sounds like you might have done this anyway).
Speak to a recommended mortgage broker really soon, they will give you free advice. Holiday home mortgages can often be based on your income whereas buy to let is viewed more like a businesss so your income isn't so important as it's the rental income, not your income that covers the mortgage repayments.
Obviously using a holiday home yourself in the school holidays really cuts into your profits and managing agents on holiday lets usually charge around 25% (keep an eye out for extras for change over days too) whereas a managing agent for a buy to let will charge around 10%.
It also depends enormously on where you live in the country. A holiday home will be very profitable in the South West but it isn't a great area for buy to let profits but the North East can be good for buy to let but less profitable for holiday lets. If you have an agent looking after it though neither a buy to let nor a holiday home needs to be local to you. As a general rule the nicer the area, the less profit you will make from rental income each month. If you know where to look it is relatively easy to find buy to let properties which will give you a 5-10% return on your investment each month after all costs. Capital growth is then a bonus. If you buy in a very low income area you can get more but it depends on the type of tenants you're willing to take on.
For a summary of all the costs of a buy to let there is a free spreadsheet to download here www.thebroadgategroup.com/buy-to-let-advice/
Most of the costs will be similar for a holiday home but you will need to increase the management costs as explained above. It would be wise to speak to some holiday let agents in the area you are thinking of buying in to see what they charge. You may also need to pay more insurance on a holiday let as it will be unoccupied for extended periods.
Sounds like you're torn between your head and your heart. If it were me i'd get a very profitable buy to let and use the proceeds to pay for great holidays!
Good luck, it's a big investment so make sure you work out all the figures before taking the plunge!