Disclaimer: I live in Norfolk!
Norfolk really is lovely - there are pretty EH and NT places to visit, wonderful beaches, parks and woods, classic seaside towns, crabs, fish and chips, pretty villages, adventure places for kids and lots of things to do and places to see.
OTOH, I've been visiting the Isle of Wight for almost all my life - my grandparents lived there and now my mum does. And we have wonderful holidays there - big sandy beaches, rocky beaches for clearer water (get some cheap surf shoes from Sports Direct and the stones are not a problem!), attractions like Blackgang Chine and the Roman Villa at Brading, pretty villages, lovely seafood, and the big attractions of Osborne House and Carisbrook Castle.
Two big tips: If you plan to visit IoW, and want to go to both Osborne and Carisbrook, seriously consider getting English Heritage membership as they are absurdly expensive otherwise. One adult can take up to 6 kids, so you'd only need a membership for adults. And it pays for itself pretty much in 2 visits.
If you are coming to Norfolk, take a look at The Fairyland Trust, if your dates match up with their summer event on 24/25 May book and go! It's a magical day, kids love it, so do adults, we have been to loads of their things and it's lovely and nature-focused, not at all plastic!