As lilyfire says, "I expect my children would have been sort of fine at school, but I don't think they'd have so much fun." I feel just the same about my older daughter. She has no special needs and I expect she'd cope with school as well as anybody would: she's averagely bright, very sociable, and eager to please. In fact she did try school for a short while but came out again after a term. There were things she liked about school but she said it took all of her time and she didn't want to spend five days a week there.
The way I see it is, why settle for "coping" at school when she could be actually thriving at home? The fact that your two are mardy and fed up for much of the time suggests that school isn't exactly what they need. Maybe school is OKish and even fun for them sometimes, but five days is just too much?
One thing I like about home education is that if there is something that isn't working well, you can usually experiment and find ways to fix it. If your child is finding maths dull, you can look for a different way to do it. If she doesn't want to read at four you can wait a few years. If she isn't enjoying the company of other girls her age, she can play with younger kids. If she's rattling around indoors acting hyper, you can take her outdoors. If other kids are picking on her you can stick close and help her sort it out, or take her somewhere else to find kids she'll get on with better. If the child is ready for more freedom than she has been getting, you can teach her to ride the bus and then let her go places on her own, regardless of whether other children of her age are ready to do so.
You wondered whether home educating would be selfish. On the contrary, I think that you are now hearing your instincts giving you a nudge in the right direction towards what your children need. It's the same reason why cuddling a baby feels nice and we do silly things to make the baby laugh. Go ahead, be selfish!