OP, I've lived near Dorchester for nearly 25 years and know Bridport and Weymouth well.
There are NO motorway links nearby, ( an hour to Taunton, M5, or an hour to Ringwood, M27, M3...)
However, Poole is great for ferry to France, Plymouth ferry to Spain...
Bridport is by far the 'nicer' vibe. Quite arty, bit Boho, has, what I'd call "good self-esteem". Good eateries, some nice quirky pubs, lovely market each Saturday up South Street, some nice antique shops.
Weymouth hasn't got good self-esteem. But many love it, it's got the old Victorian-type Esplanade, a great beach and it's own charm really.
But it has a big country lines drug problem, ( actually, this exists in Bridprt too, to be fair).
There is more evident poverty, the high street is definitely grim. GP surgeries are over-subscribed there, and even more on Portland. There are good community clinics though and DCH ( Dorset County Hospital) is good ( in Dorchester).
Swanage, I know less well, but is a really pretty, happy kind of place, with Festival-type events and, again, a good sense of self-esteem. Also, excellent for the South West Coastal path, Corfe Castle, Durdle Dor etc...good eateries, (excellent fishmonger btw).
What you've heard about Portland will be echoed by many, I think and there is definitely some truth in it, but some swear real allegiance and love it.
Good luck, I think, if it were me, I'd be favouring Bridport, but have great fun visiting and researching.
You may find it a bit of a culture shock to move from London, but it sounds like you're ready for change.
Oh, one last thing, be prepared to do most of your shopping online, no John Lewis nearer than Bournemouth just forget about ANY big shops that you'd get in London easily. I was brought up in London but I get quite wide-eyed whenever I go back, at the sheer availability of everything.
Best of luck