Can you wait a year? The castle is currently closed for redevelopment - see here
The Castle is in the top half of the park. Excellent new kids play area round the corner from it. You could probably spend a couple of hours in the castle when it re-opens and lunch in the park
Firstsite is just round the corner to see the Berryfields Mosaic - they also have free family art activities on saturdays - check the website to see what's on - free entrance and fab building. They also have a nice cafe. Lots of nice little family friendly cafes all over Colchester.
Over the road from the castle you also have the natural history museum which whilst not about Romans is quite nice to stop for half an hour or so - free to visit.