I suffer vertigo and standing on a moving bus would be hell for me, however I would absolutely give up my seat for a mother or father with a young child. Toddlers are unsafe on their feet as it is (hence why they're called toddlers) but put them on a moving bus where most adults require two arms to remain standing, and have a parent hold on with one hand, probably have bags to hang on to and hold a toddler with the other hand and this then becomes unsafe.
Imagine, the bus stops abruptly, and the 16 stone able bodied man who was standing behind, and who'd just taken his hand off the rail to scratch his nose, loses his balance and falls on top of the toddler. An able bodied adult might have been caused to fall themselves if someone falls into them but the damage would not be as great as that caused to a child. If the toddler was in a seat he'd be much safer.
I agree with the OP about his choice being the wrong one. The toddler needed the seat. My 7yo would have the ability to hold on whilst standing, but a toddling 3yo wouldn't. I would expect my children to give up their seat for any adult able bodied or not, the same as I was taught, but not at 3yo.
However I believe the OP has recognised how difficult it is to ask for a seat, which is exactly how the man would have felt. He had to ask for the seat. If OP then wanted a seat for her child she would've had to have asked herself. Which is clearly something that is very difficult.
So, whilst I understand exactly how you feel, OP, I'm sure you now realise exactly how he felt in having to ask for the seat.
But yes, asking a toddler to stand was probably not the best choice, but this man has probably encountered a lot of discriminatory comments in his time on public transport, and probably felt that a woman with a child would be least likely to verbally abuse him.
So I agree with both the OP, and the man.