I think only you can know what feels right for you and your DCs and the fact that you're asking this questions might mean you're wanting to spend more time with them? I too am jealous!
I'm a single mum of twins aged almost 9, with no family/ inlaws/ ex and when I'm not working (which I only do whilst the children are at school ), I'm with my twins. I haven't been out (for pleasure) since they were born and the only times they've been away overnight - x3 ever in their lives to friends - it was so I could catch up on work and because of certain issues with those friends, it may not happen again.
However, despite my own envy and realisation that I probably need SOME me time, as I get none at all, I think that quality of presence with your children is paramount. If you spent more time with them but were in a resentful space, feeling irritable and tetchy, then they'd be better off with others who care about them too, whilst you recuperate.
Another big factor is that they're with their father or grandparents. I think children thrive from lots of loving relationships in their lives - another thing I envy that others have and my children don't - they've just got me. So your children may be benefiting hugely from contact with other family members.
The only hesitation I have - which is purely subjective and personal - is that your youngest is probably still at the stage when loads of close physical contact with you would help. I b/fed till my twins were 28 months and we co-slept till they were 5 - all part of the attachment pareting/ Continuum Concept philosophy. I'm not sure I'd do it exactly the same if i had my time again, as I can now see I went too far in the direction of suppressing all my own needs in favour of theirs.
However, I can see how they've benefited from that close physical contact with me and wonder if your youngets especially might be needing more from you?
But again, I can't and don't judge you because you have to decide what works best for you. If I had family or even close friends who really loved my sons and enjoyed being with them, I'd be much more likely to take some time out for me whilst they were with them.
Listen to your own feleings about this and ask yourself why you've questioned this and then do what feels right for you, would be my advice.
Meanwhile, I'm still working on getting any time at all for me and we're heading into a decade of none!!!!!!!!!!!!!! I guess it's all about getting the balance right between your own needs and those of your DCs and I haven't got that balance right for me yet.