Penrhyn Castle outside Bangor is great- It has some old steam trains the kids can climb on and pretend to drive, and a nice cafe, and huge grounds. I don't know if you'd want to go into the actual castle with such young children, but when I worked there you could pay a smaller fee for going into the grounds only.
There's a free slate museum in Llanberis, which in itself is a bit dull for young kids, but there's a nice playground, a lovely spooky lake where people go diving (not the big Padarn lake- this one is in a gully, and is an amazing azure colour because of the slate) and you can visit the old quarry hospital, which my DS LOVES because it's spooky and has old medical gear. 
:o
Pili Palas on Anglesey is a butterfly zoo type thing- It's great, has snakes and creepy crawlies, a soft play area and a playground.
If it's nice weather, you should go for a walk in Glynllifon. It's a beautiful old manor with lovely grounds, and there's a few things there that relate to stories from the Mabinogi- Your 4yo would probably like that! Perfect place for a picnic and a run around, and doesn't seem to attract the tourists it should.
I have now made myself homesick! :)