I don’t think it’s such a terrible idea tbh. I wouldn’t go on a very organised trip on a luxury reserve though (I actually find those exhausting rather than relaxing anyway).
I lived in SA for a few years pre-DC and have safaried with friends who had young DC with them. We stayed in a rented house with pool in Marloth Park which is a private estate on the Southern border of Kruger Park. You can watch elephants in Kruger bathing in the Crocodile River from the comfort of your balcony in Marloth if you are lucky enough to find a house along the boundary line.
Like many private housing/game estates that border the parks, antelope, zebra, warthogs and sometimes giraffe wander around freely and may come up to your garden fence to investigate.
The time we visited with friends with kids, we did a self drive in Kruger one day (started out later than the crack of dawn). We also did a guided one (early start) with a guide whose safari vehicle could seat maybe 12 people but we had booked a private tour so had it all to ourselves. It wasn’t that expensive to do that and I have done similar in other parks around SA.
You could have a similar experience in Pilanesberg which is closer to Joburg and is near Sun City if you want to mix it up with some very child centric activities.
I have rented similar properties in Hoedspruit (can’t remember the name of the estate) up the west side of Kruger and in St Lucia for visits to Hluhluwe/Imfolozi (spelling almost certainly incorrect). Marloth was my favourite and we visited many times and always had a good time.
You have to do a bit of searching to build your trip that way, but it’s vastly cheaper than the luxury organised safari and very comfortable if you stay in a nice house.
I wouldn’t do a big intensive safari with little kids (3 or 4 days is plenty) and would probably stick to SA and combine it with other holiday activities such as visiting Cape Town, a beach trip etc.