As others have said, it very much depends on what kind of family you are and what you enjoy. We spent time on Arran every summer for years and years when our children were young because we have family there (and stayed with them, which makes for a different holiday, of course).
We do like walking, and we're interested in history and we like nature and beautiful countryside. We also don't have a car. Midges have never been an issue for us - maybe we've just been lucky. Weather is usually at least as good as we would expect in London.
Getting there: we get the train from Euston to Glasgow Central, which takes 4.5 hours. Then train from Central to Ardrossan Harbour, which takes an hour. The train connects with the ferry (all being well) so straight onto the ferry. Another hour and you're in Brodick. We did once get a plane from Stansted to Prestwick, but it saved very little time, as you still have to get to Ardrossan and wait for the ferry.
If you drive from London and take the car to the island, you will need to book onto the ferry well in advance as it gets busy in the summer. No need to book if you cross as pedestrians.
We get about on Arran by bus. Buses meet the ferry. You can see all the routes and timetables online.
Places we used to go with the children:
Brodick has a crazy golf course, some very nice cafes and restaurants, Auchrannie Spa and various sports activities including a pool. (There's another pool at Blackwaterfoot in the Kinloch Hotel there, south end of the island.) I love walking on Brodick beach across to Cladach, which is the bottom of the Goatfell path, and takes you into Brodick Castle grounds. There's an excellent cafe called the Wineport at Cladach.
Isle of Arran Museum on the outskirts of Brodick - excellent wet weather option, but I really like it anyway, and there are a lot of child-focussed exhibits.
Brodick Castle - National Trust for Scotland. If you belong to the National Trust (Eng/Wales/NI) you'll get free entry by reciprocal arrangement (at least, this is how it used to work - best to check). You pay to get into the gardens and you pay more to look round the castle itself. The gardens are lovely and include an adventure playground.
The NTS also owns a huge chunk of land around and behind the castle, including the tallest mountain, Goatfell. Access to that is free. There is a clear, well-maintained path all the way to the top of Goatfell which is very well used all year round. It's quite a pull to the top of Goatfell but if your children are used to walking and have sensible footwear, they should be able to manage it. If I can do it, anybody can. Best saved for a clear day as the views from the top are wonderful.
Glenashdale Falls - Whiting Bay, one of the other villages on the east coast. Lovely walk, another good path, not a difficult walk at all. There are a couple of excellent cafes in Whiting Bay and a lovely beach.
King's Cave on the west coast is a great well signposted walk along the coast with an interesting cave on the way. There are standing stones nearby at Machrie.
Kildonan beach - glorious. South end of the island. Good place to see seals.
Lochranza - beautiful village at the north of the island. Deer roam about round the ruined castle (free entrance, as there's not much of it). The children are a bit young to be interested in the distillery, but I enjoyed the tour there a few years ago.
Glen Sannox, north end of the east coast - terrific beach. Pony trekking somewhere round there (we haven't tried that, but we've seen them).
Your children are probably too little for golf, but I'll just mention that it's a golfers' paradise (we are not golfers but you can't miss the golf courses as you go round the island).
Hope this helps. I love Arran.