They're very different in terms of personality.
I only have Golden Retriever experience but the question is something I get asked a lot so I've spoken to a lot of lab/flattie owners over the years about it. Tbh I think a Golden Retriever is more similar to a Flat Coat retriever than a Labrador Retriever.
Temperament: even though you got your Golden Retriever at 4, you probably already know that they stay puppies for almost all their lives. Even when they're 8/9/10 they can still have that 'puppy energy.'
I think labs (at least the ones I know) are a bit different. They tend to mature slightly faster and once they're 4/5 they're 'adults' in a way that Golden Retrievers never are.
Labs are a bit more 'independent' and 'robust' as well I think. I've never known a Golden not want to be around their family 24/7 which means they can be terrible for separation anxiety because of that if they're left long or not trained well.
Training: Personally I think Golden Retrievers are more challenging to train. Obviously they're typically food motivated, like labs, but they're also more 'desperate to please.' Which works in your favour if you know what you're doing, but if you don't can be a nightmare.
Golden's are far more sensitive to harsh words than people typically realise, and so when they do something wrong - assuming they're not having a manic moment where all they can focus on is being a massive nob - they tend to be quite 'upset' and that can present challenges later down the line with training as they do 'hold a grudge' (as my mother puts it). They also tend to be more easily distracted - again because of the slow mental growth they have.
Labs don't necessarily have that 'sensitivity' to the same levels that Golden's do and in terms of personality are more 'resiliant' - even if poorly bred (a whole separate thread could be done in the differences in temperament between a well bred and badly bred dog) - so if they're told off, they just bounce back.
Health: Health is the biggest difference. Obviously both have the usual issues with skin, elbows and hips. But Golden's are far more pre-disposed to cancer than labs (although UK lines less severely effected than US lines).
Puppy Years: Golden Retrievers are little bastards as puppies. I've had them for twenty years (I breed them) as an owner and they have driven me to tears. By week 7 of puppy raising I am usually ready to throw them out the door to their new families and having to keep them until 8 weeks is akin to torture (honestly, if it was used as a form of torture I would not be surprise). And it improves horribly slowly. It takes months for them to really understand the strength of their bite - easily 6/7 months in extreme cases - and after that, they'll still want to mouth and if they get over-excited they'll still mouth hard. My boy sometimes walks alongside me on walks with my wrist in his mouth. It's his thing. There's never any malice in the puppy teething, but they will get a 'look' that is vicious - because they think it's a form of play - and that can be intimidating to children. if I had a £ for every time someone said their Golden Retriever puppy was vicious and it was just playing, I'd be as rich as Musk.
All puppies are rough. But Golden's are up there with GSDs/Malis and Collies when it comes to teething. Labs are rotten as well, but not as bad as they're not typically as manic (see above).
So I guess it depends what you're willing to put up with and what you want. I adore Goldens. There will never be another breed like them for me. But they're rough and not everyone wants to be watched whilst they shower...