I think it's helpful to understand what B's own family life was like as a child. It wasn't easy as his mother had severe depression and Parkinson's from 40. His parents divorced when he was quite young. His home life was in some ways chaotic.
Now, none of that excuses his sexual exploits, but it maybe shows how the upper classes have a different attitude to parenting, in some ways.
Likewise, Carrie is a 'love child' of an affair her mother had.
They each had a fairly unconventional upbringing.
It's not as if he will be living in a 3-bed semi on a housing estate once he leaves office. She has money, as does her family, and so does he.
It makes it far easier to be an older parent bringing up two young children compared to Mr and Mrs Average.
So the idea of 2 young DCs in your 50s probably doesn't bother him at all, as it doesn't Mick Jagger and other ageing rock stars.