You're not being unreasonable. You know your children and the kind of holidays you enjoy and should do whatever you want. But I'm not sure why "abroad" is necessarily more stressful than the UK, unless you are particular worried about the travelling part.
I know that I'd have been pretty miserable and frustrated if I'd waited until my children were over 5 before we left the country. But then, my idea of a holiday has never been relaxing by the pool/sea, doing nothing but sunbathing or reading my book, then going out late into the evening. I can see that is quite hard to replicate with young children and could get stressful as a result. I've always seen holidays as a chance to see different places, and I did that with young children in tow too.
By the time they were 5/6, our two had been on holidays to Provence, Belgium, Vienna, Umbria, Switzerland, Paris, Norway/Sweden, Croatia and Japan! Not including trips to the Netherlands to see family. I may have forgotten some others. In the primary school years they got to many other places including South Africa, California, Morocco.
Our most miserable holiday with young children was when I attempted a family friendly beach holiday in the UK. It rained half the time and they didn't much like the beach anyway! I wished I hadn't bothered. In contrast, travelling around Japan with a 5 and 6 year old had it's occasional tricky moments, but I didn't care because I hadn't planned it around being child friendly anyway, and I was loving being somewhere new and exciting. They still love sushi now.