We (me, ds1 - 8, and ds2 - 7) spent a couple of hours there today, and had a fun time. We are members of English Heritage, so it was free, and the boys enjoyed doing the children's trail - which was also free.
We'd planned to do Tullie House as well, but we'd run out of energy. We spied Bitts Park swingpark from the battlements of the castle, but unfortunately when we got down to ground level discovered it was closed for refurbishments 
Anyway, we liked Carlisle a lot - lovely town centre, and cute little cathedral. So hope to be back again soon.