I think it does depend on the child but we waited until the children were older as I had one who could not stay still or sit still.
The pros, it is an amazing place, we have been several times and we absolutely love it, the parades, rides, fireworks, shows and full immersion into Disney is incredible.
The cons with a toddler, the flight is long. ie I believe the first time we went the flight was 10am so check in from 7am and obviously getting to the airport from wherever you are staying. We chose an airport hotel. Then the flight, we landed at 3pm local time but they are 5 hours behind so realistically 8pm. You then have to clear customs and immigration and then the 25-35 minute journey to the hotel depending on where you are staying. That time difference can mean a very early wake up the next few days.
The average Magic Kingdom crowd level is around 60,000 and no I didn't add a zero, sixty thousand people. There are a lot of rides that have no height restrictions but the queues even with the fast pass system will still see you standing around. It will depend how well your toddler will queue.
The heat and humidity, I have seen adults trying to encourage young children to drink to keep them hydrated.
The need for sleep and whether they would sleep in the stroller. They also play music everywhere so it can be loud. If they won't sleep then where is your hotel and how long will it take you to get to it? If you hire a car you have to get a boat or monorail back to the car park, then hop onto a golf buggy style train to take you to your car that is parked in that lot of 60,000 people. Or if you use Disney transport if you are staying on site then you have to queue for a bus. Those buses can get very busy with people standing for 15 minutes to get back.
Characters can seem ridiculously huge to toddlers who may be very fearful. Oh and parking your stroller amongst the other hundred or so if you are on a very young child ride ie Peter Pan.
As I said, we love it but this is the reality. We fly from Dublin to clear customs and immigration in Dublin so get off the plane collect the luggage and leave. But that flight leaves at 3pm UK time, plus you have to get to Dublin airport.