We recently took DS (19 months) and apart from the beaches which he absolutely loved, I'd recommend The Isle of Wight Steam Railway. We had a lovely day there going on the railway, the cafe is a little bit expensive, so I'd recommend taking a packed lunch, they have a nice playground and a small indoor playroom as well. DS was fascinated by the trains and we felt very relaxed there.
I personally would avoid Alum Bay unless you are planning to go on a boat trip to see the Needles a bit closer as the parking is £4 all day and the rides and shop are pretty shite IMHO!
Yarmouth is a nice small town if you're staying in the West Wight, there's lots of nice shops and a good place to buy gifts. You can walk to the end of the pier which is nice and watch people fishing. I haven't been to Fort Victoria for a long time, but heard its good now.
I'd recommend Colwell Bay and Totland Bay as good child friendly beaches in the West Wight, as well as Compton Bay although the steps are quite steep on the way down the cliff to the beach at Compton. The water is very clean and not commercialised like Sandown/Shanklin.
Freshwater Bay is nice for adults to enjoy the views, but not much good for children, it's not very safe to swim there unless you're a very good swimmer as the sea bed shelves suddenly and there's an undercurrent.
Ryde beach is good for children too, there's always plenty to watch with the hovercraft and boats etc., there's also a children's play area at Appleby and SeaView has a wildlife park which was recommended to me.