YANBU. Why on earth would he refuse to give t up his seat AND abuse you, I don't get it?
Everyone knows they should give a seat to a person with small kids, or elderly person or pregnant woman. To have a go at you about it just for asking is just ridiculous, and I'd say over-aggressive to the point of being mentally ill.
If I'd been on the bus with you, I would have noticed and given you a seat instantly. It's not safe for little kids to stand on the bus, but I think a lot of people just don't give a monkeys tbh.
I think he sounds ill or deeply weird to be honest, that's not normal. Don't let it upset you, feel sorry for him, his life must be dreadful if he lives it at that angry pitch.