We've taken our son to a local family festival. He was 16 months the first time, then we missed a year because of Covid and we've taken him again when he was 3 and 4. We're going again next month and he'll be 5 this time.
The first time we took him we tried to stick to his routine of nap, meal and bedtimes. There's plenty on in the day to make up for what we miss when he's in bed. But we try and camp near the arena and can always hear and/or get a glimpse of the main stage so we don't miss out completely. Last year we kept him up slightly later than normal but he didn't sleep in any later the next morning so he was a bit tired. He's a year older now though so may cope with a later night a bit better. We'll still miss the headline acts as they are always on later but that's our choice.
It is family friendly as they have a big inflatable 'village', free circus, outdoor/garden games like giant jenga, skittles, table tennis, bouncy castle, hula hoops, badminton etc. all in a separate children's corner of the festival. There's family camping areas too where they encourage silence after a certain time. So it is very family friendly. Yes there's lots of drunk people, but there's also lots of people that aren't drunk. It's walking distance from the town centre so you can escape the confines of the festival if you want.
Would I recommend it? You certainly get a different experience than if you were an adults only group, but it sounds like the OP knows that and is prepared. There can be a lot of walking so definitely invest in a little cart to pull the toddler round in. We've recently sold ours as he's a bit older now, but it came in really useful the last couple of years.
Go for it @Hiphopopotamonster. If it doesn't work out for you then try again in a couple of years. Our 5 year old loves camping and loves the festival. There's more families that go than childless couples/groups, but that depends on the festival and if it's promoted as family friendly.