I expect the OP has inherited a strong 'work ethic' as well as the money, and that makes such decisions harder.
I'm in a situation where I don't enjoy my 3-days a week job, we don't need the money (at all) and I can think of plenty of hobbies/volunteer work I could do to keep myself busy. But my inner work ethic (which I never realised was so vocal) keeps telling me that if I can work, I should work, and I feel guilty and 'spoilt' to be considering giving up what is a very well-paid, stress-free, WFH, flexible, part-time hours job that most people would give their eye teeth to have.
So OP, I understand why you are feeling the way you do, but two observations:
(1) you have young children - you don't need an 'excuse' not to work
(2) you're currently working full-time, which again, is more than most people with kids and no mortgage do.
So you're setting the bar very high in terms of your work ethic already - you're not in-line with how the majority of (wealthy) people with children live, even without the £2 million inheritance.
If I do give up my career, I'd like to work (very part time) in a farmshop/a cafe/a National Trust property, as I enjoy 'work' per se, but then I'd have the guilt of taking a minimum wage job away from someone who needs it more than I do.