Thank you very much for your replies.
Werewolf, the freeholder has known all along. I have emails from their agent to say that they are happy with it, so long as they are kept informed as them knowing about it means the buildings insurance remains valid. The freeholder's solicitor, however, is now saying that holiday lets invalidate the buildings insurance.
The lease says: "Not to use the Flat nor permit the same to be used for any purpose… other than as a private dwelling house in the occupation of one family only nor for any purpose from which a nuisance or annoyance can arise to the lessor or the owners lessees or occupiers of other flats in the building…nor for any illegal or immoral purpose." I am told this is pretty standard wording, but that it is now often interpreted to mean 'no holiday lets', although this isn't explicit. This wasn't flagged up when I bought it as an ongoing holiday let concern, though. That said, there were relatively few holiday lets around when I bought this one, so there wasn't any particular need for a lease to exclude them specifically (all new-builds around here do specifically exclude them now).
Littlemiss, I'm sorry to hear that you have had such a bad experience. Please don't think all holiday let owners or tenants are so awful. I vet all my tenants prior to arrival, and meet them when they arrive. I once turned away a group of middle-aged women who were drunk when I met them; they were apoplectic and threatened me with the police, Trip Advisor etc - but it does say in my contract with all guests that they will be denied access to the property in such circumstances. Most guests are in fact elderly couples (mostly Australian/American - we're on the tourist trail), and I can't imagine they are up raving at 5 AM. FWIW, I have had similarly bad experiences with neighbours - though unfortunately they were permanent residents, not tenants of any description. I ended up moving out.
I agree that saying it's my only source of income isn't a defence for flouting the law. There are other factors which I think might make a difference (such as the one outlined in my response to Werewolf regarding the agents). One of the people who has provided 'evidence' (none of which is apparently available for me to see - I suspect that this is because it doesn't exist) to the freeholder does himself run a holiday let in this particular development. However, he has a varied lease which explicitly permits this (the lease was varied 12 years ago, prior to the flats coming on the market for the first time). He is very keen to keep other holiday lettings out! There are three of us running holiday lets in total, out of about 25 properties. Of the remaining properties, about half are owner occupied, and the rest are let out on ASTs.
I don't have any kind of mortgage on my flat, so am not breaching any conditions on that score. I have bookings well into next summer, but have meanwhile blocked out all the remaining availability until 2021, pending advice.