I was a single mum for 9 years, and can count the number of times I went out in all that time on the fingers of 2 hands.
Despite working FT (my only source of adult "company"), I couldn't afford to go out, never mind pay for a babysitter, and babysitting circles don't really work if you're on your own unless you take your child along too (not usually very practical). I had no family nearby, and I doubt they'd have been much help even if they had been.
But I never took any risks with my son. Once is all it takes.
Yes ...... of course it can be bloody miserable, feeling trapped and forgotten about but when you have a child (or children) however shitty it feels, you don't come first - they do. There is nothing intrinsicly wrong in wanting some fun and a social life, but only if it can be safely achieved. If you can't do it responsibly and safely, then tough shit.
Your mum isn't being selfish. Maybe she could do more but you should regard this as a bonus, not as an entitlement since they are your children, not hers.
Unfortunately, if you want adult company, you have to use your imagination and accept that for quite some while, getting that adult company may not be in quite the way you'd choose. You don't say if you work or not but if not, there is reasonable scope for you to meet up with other mums during the day (yes, I know you'd have the kids with you if they weren't in nursery or school, but it's better than nothing). You also have a PC ...... again, it may not be quite the same thing but you can interact with all sorts of other adults in all sorts of ways - and in live time - via chat forums and maybe with a webcam. I wish I had a PC when I was a single mum .... most of the time I didn't.
Please don't take any more risks.