I haven't read entire thread, just the OP so others may already have advised the same, and apologies if I've missed or am repeating anything.
I think you're wise to be considering this. In my experience of raising a male lab from puppyhood, in their first few years, they're not ideally suited to being around young kids all of the time without years of onerous training. When I went to pick up my 2mo lab from breeder he was playing gently and delightedly with a 3yo child, and politely ignoring a young baby in high chair. First impressions can be deceptive! As a puppy he literally caused me blood sweat and tears. Despite hundreds of hours of training and countless classes even my adult neighbours knew not to stroke him lest he jump up on them with his 5.5 stone hulk. My relatives refused to dog sit him because he was too full on for them, even for 1hr (i have him 24-7).. Think Marley and Me dog sitter saying "that is not a dog, that is evil with a dogs face". The dog wasnt evil (certainly mine isnt, hes a big sop, just extrordinarily big and boisterous). Think the dog trainer scene where trainer is convinced that SHE can manage his behaviour better than the owners only to be pulled over and to ban him from class. Mine is EXACTLY like Marley. Not a bad bone in his body. Lives all people and dogs, all food and fun. Sooo much joy for life. Extremely loveable. But A LOT. I occassionally struggle to hold him back as a large adult. I would definitely not want to be a child who weighed less than him having to live with him.
So my advice would be to choose a dog that weighs LESS THAN half of smallest family member. The lighter the better.
I'd also emphasise that dogs truly are individuals just like people are. You really can't simply choose a breed and know the individual you get will be a good fit.
However you can significantly stack the odds in your favour by looking at breed traits inherent in dogs genetic breed and being realistic about these and not romanticising these in any way. Having a retriever greet you at door with a fluffy toy is cute, having them pick up and chew EVERYTHING as a puppy is not.
Listen to people with experience. The first breeder I went to refused to sell me a male lab as she thought he'd be too much for me. She recommended a female if I was determined too get a lab, but none were available in her litter or others (it was winter). I thought I knew better. and went on to buy a 2mo 10lb male lab who was available elsewhere , he was tiny and cute and Id read ALL the books, had a lab in wider family, and thought I knew what I was doing, so. what could go wrong? Mwar har har. Suffice to say, I didn't know better. I should have listened. He's adoreable but to say he makes his presence felt is an understatement.
So yes, I'd say go small for first time dog. Know the breed traits. But then be prepared to welcome a complete individual.
I don't have any experience of small dogs though so take that with a pinch of salt. But I do have experience of big male lab that could potentially help you avoid mistaking their idylic image as family dogs for closer to Marley an Me reality, at least some, when young and in my case.
All that said, I love my dog and wouldn't trade him for world. Retrievers are the only breed for me. But given that you're flexible, I'd strongly suggest smaller breed.
Dog walking is also easy to say you'll want, and always enjoy. Reality really does get in the way. Days where someone's had to go to A&E, you've had to work 12 hours; you're too fed up to want to get out of bed: it's raining horizontally all happen. Even on a good day, walking the same route for literally the 500th time gets old. So much easier with a small dog who only needs 30 min but can manage more than with a retriever who needs at least 3 times that every day or will make his own entertainment (in ways that will not entertain you!)
Good luck, whatever you decide. They really are peoples best friend.