I grew up in Norfolk, all my family live there, I go often.
Make extended visits and stays before you make a decision. Stay in February (not half term) and August.
I love it for its beauty, skies and beaches.
The coastal areas are generally remote from infrastructure, towns etc.
Things to consider: public transport is sparse. Access to a hospital from the N Norfolk coast , for example, is an hour. And much more difficult by public transport. GP surgeries can be a long way away, dentists further.
Cinemas: Few and far between. Teens will need a lift. Theatre: Norwich or Kings Lynn.
Shops: many villages no longer have a shop, supermarkets are only in the bigger towns.
Phone / 4g signal still terrible in big swathes of the N Norfolk coast.
General insularity: coastal and rural deprivation is real. Poverty due to many seasonal workers, loss of employment opportunities etc exists. Lack of access to cultural opportunities and multi cultural life, much conservative thinking. My sister is a teacher in a North Norfolk secondary and is very aware of the restricted horizons of the kids. And yes racism. She sees and hears it all the time. I have mixed race kids and when we have been up there in ‘non tourist’ situations have seen it. Not necessarily full on EDL stuff, but sheer ignorance borne of insularity and lack of experience. But not liveable with, for my family.
Swaffham is not a place to live if you want to be near the sea. 
Visiting, spending every summer there in hol, is a very different experience to living in a sparsely populated area with a bitter Easterly wind from October to March.