Agree with you, OP. If you can afford a holiday then you can afford travel insurance. IMO, it's completely irresponsible to go abroad without it.
If you're only going away once in the year, You cab get cover from £5 (or £10 for a family)... Hardly breaking the bank when you've already forked out for the holiday.
Annual policies win if you jet off at least 2x a year - cover in Europe is around £9 (or £17 for the family), and £19 worldwide (or £37 for family)
Travel insurance doesn't just cover you while you're away, it'll also cover you for cancellation or anything else that might go wrong before you leave. So if you don't have it and you need to cancel, you've no protection. You should be taking out travel insurance at the same time as making your holiday booking.
Like PP have said, You still need insurance even if you have your EHIC card. Travel insurance does MUCH more than cover medical emergencies. You're also covered for theft, lost luggage and repatriation, which an EHIC doesn't do. But having an EHIC means you don't have to pay the excess on your policy if you show your EHIC when getting treatment.
However, Your claim could fail if you haven't declared all medical conditions (for you, your family or a travelling companion) before you travel. No matter how minor you think it is, tell them EVERYTHING. This applies to you, anyone you're travelling with or a family member at home (who you'd fly back for if they got ill).
If you lose your mobile/e-reader/laptop, you're probably not covered. Many travel insurance policies won't automatically include gadget cover. If you need it, select it as an add-on extension to your travel insurance, check your home insurance cover or consider a specialist gadget policy. However, In some cases, even if you've had one drink and you lose something or have an accident, they may say NO if you claim. You need to check the policy.
Also, You're never covered in dangerous destinations. If you visit a country the Foreign Office warns against travel to, you won't be covered for any claims as you're deemed to have ignored Govt advice.