Lived in Suffolk and went to Norfolk, regularly. They are both stunning counties. Have you looked at Centre Parks? It was too close for us to ever use, but we had friends who swore by it.
Has no one mentioned Aldeburgh? Fab location, great restaurants, shops, ice-creams and brilliant fish and chips - as long as you are not a veggie.
Do you like bitter? Both Suffolk and Norfolk brew beautiful beers. Ceramics, deli and great seconds bookshop in Southwold. Brilliant second-hand book shop in Wells (I'd have to check that).
Personally, always liked Norwich - city girl, really. Then again, Bury St. Edmunds is a fab place. Actually, this is a win-win situation because you can stay in one and visit the other.