Times I have travelled without insurance: 0 (so far as I recall)
Times I have claimed on my travel insurance: 0
Would I ever travel without insurance? No. And I'm not at all wealthy, although am healthy so insurance is not that expensive for me (I have an annual policy so I know I'm always covered). If insurance were very expensive, I'd have to consider if I could afford the trip.
I think that one of the reasons some people might not bother with travel insurance is that they think about the straightforward costs of e.g. travel delay, lost luggage etc and figure that they could suck up the cost, especially if it's 'offset' by saving several years of saving on not paying insurance premiums. What people don't tend to think of is the impact on others of you having/not having insurance, something which was brought home to me due to my close friend's experience:
When my friend's dad died (completely unexpectedly) while on holiday in the EU, her parents' travel insurance paid for her flight out to be with her mum - and they were there for 3 weeks while he was in a coma before he died - all the medical and repatriation costs for her dad, and flights home for them all (and would have covered accommodation for them both but they were staying in their own apartment). Although her parents are fairly well off and her mum could probably have afforded to pay these bills, it would have been distressing. What my friend valued far, far more was that the insurers took charge of everything. They spoke to the hospital and had their own doctor ring the family to ensure they understood what was going on with her dad and after he died they registered death and ensured copies of the certificate were obtained in English too, organised an undertaker to make arrangements to collect her dad from the hospital and transport to airport and then home to UK, including all the paperwork, then transport to a funeral home of their choice in UK. The family didn't have to do a thing except arrange the funeral, the same way they would have done if he'd died at home.
Having heard all this - and seen how much the family were struggling with the bereavement even without the added stress of making those arrangements in a foreign country - really brought home that travelling without insurance could have a far wider impact than just on myself and I think it would be pretty selfish.