At the end of the day, people have been taking their babies and children to Glastonbury - the biggest one - for years. I have never heard anyone say that they or their kids are permanently scarred for life over the experience. In fact, most of the teens/YAs I know that went as kids love to brag about the fact that they went to their first festival when they're tiny. I was sat waiting for Elton this year and realised I was surrounded by tiny babies (there was one who couldn't be older than 2m old) right up to 5/6 years old. The whole time I was there 95% of the time the kids (who weren't being dragged to inappropriate stuff!) looked happy and like they were having the time of their lives (and the littlest were perfectly content).
I got a lot of stick on another post for saying I was worried about taking what will be my 6mo on a long haul holiday next summer so I'm really surprised by the amount of people saying you're being unreasonable for taking your children to what is essentially a camping holiday with entertainment in this country.
Family camping will be fine - anyone camping in family expecting not to have children crying in the night has unrealistic expectations imo (and still better than the alternative which is pitching a tent in the midst of drunken screaming teens).
I don't have children yet (due in Dec) but have worked a lot of different festivals and dealt with lots of families at those festivals and the festival trolley suggestion is a great one but have a back up plan in case it's really muddy, otherwise you guys will be knackered dragging it through the mud. Keep an eye on your 2.5 yo with their wristband - they play with them and tighten them up pretty quick. Wristbanders will swap it for a new one if they do. I've also known them to put wristbands on ankles instead as they're less likely to play with them. Extra wristbands with a phone no on are worthwhile too, particularly if your 2.5yo is a wanderer.
And I'd just be prepared to come home if it's too much for either you or your kids. My friend took her 2 young boys camping and they both got D&V the first night and it was hell, so they packed up and left. Hoping that doesn't happen to you but try not to be too disappointed if something like that does happen.
If there is a town nearby, popping there on one morning for a bit of normality and stocking up on supplies is always a good idea (and saves spending a fortune on festival food all weekend!)
In the past 10 years of working festivals and of all the kids I've encountered in that time, I can only recall seeing one truly miserable child (and from the snippet of conversation I caught, it was because she hadn't been allowed more sweets). I know it's a snapshot in time and I'm not at all suggesting your kids won't get upset at some point over the weekend but as long as you're sensible and don't force them to stay up later than they can manage, or take them into moshpits or get yourselves into a state where you can't take care of them (which it sounds like you won't if you're not drinking!), I honestly think they'll have a great time and if they don't, lesson learnt, no lasting damage will be done and you know not to take them again.