I can only speak of Hertford, although I have visited BS and I liked the town centre - yes, a bit close to the airport, but handy for holidays and a fast rail link into town.
Re Hertford - Not sure I agree about the TOWIE vibe, although it has one of the few nightclubs in the area so people come in from 'over the border' on Saturday evenings!
On the other hand, during the day it has a positive vibe I think - lots of independent shops and a weekly market, excellent schools - it is the historic county town. There is a good mixture of housing and it is a friendly place. There is a theatre/cinema, but only one screen, so Stevenage is the nearest place for multi screens, bowling etc. There are lots of places to eat and drink.
The villages around such as the Hadhams, Hertingforbury etc are uber 'posh', but it depends what you are looking for really - probably the only way is to look around and try to get a feel for what is right for you.
There are two rail links - Hertford East and Hertford North which take about 40-50 mins into the city, so there is a fall back position if there are problems on one of the lines. It does feel frustratingly slow sometimes though, as it is much quicker into town from Stevenage which is further out.
And yes, it is all solidly Tory, if that is a problem.