OP id say, simply from what you've written, in your situation no, I would not give up work. Here's why.
Right now everything is great, right? DH has this dream income and while your not at the level you wanted to be in your job, you have enjoyed it. Now, DH is actually, subtly (and I'm not suggesting this is intentional) starting to indicate that he thinks his income outweighs the enjoyment you could have in future from your career.
He wants you to stop working (independently of his company) because then you can take all the school holidays, right? It sounds great, it really does. But you're now in a position where if you choose to choose the option that's intellectually stimulating for you, you're reducing family time and his holiday options. This decision is coming about not because you hate your work, but because of his desire to spend the money he earns in a different way.
Also, how are these long holidays going to look? Is he going to take a solid six weeks off over the summer holidays for example and not answer work calls and emails while you're on holiday? If the answer is yes, then that's different, but I suspect that like most people with that kind of income (and less too), that it's simply impossible to run a company and do that.
Finally I'd say that it depends on what your values are and what you want for your kids. I had to give up work for my husbands career. For me the lack of intellectual stimulation was awful so I'm retraining. One thing it made me realise too though was that I didn't want my kids growing up thinking the man was the provider and the mum didn't work. I know that's fine for some people but personally for me it wasn't and I didn't realise until a significant period of not working.
In a dream situation, where I had far more money, what I'd do is set up a charity/organisation to meet a special interest of mine to do with community work. I wouldn't stop working.
If you do choose to stop though, my advice would be to speak - independently - to financial specialist in high net worth marriages (and divorces) and find a way to set up a system where you're properly taken care of, and not under any risk should something bad happen in your relationship. I'm sure you have life insurance, and this is see along the same lines. A mechanism where money is transferred into your personal account, not "he lets me spend whatever I want and doesn't say anything". If he wants you to forfeit proper career advancement tie his holidays, then you should have significant, long-term financial security in your own name.