@LabradorLadbradoo an under exercised and under stimulated dog is a prime candidate for becoming destructive (which you appear to already be seeing) and when not properly addressed that's when it escalates.
https://lovingcarepethospital.net/the-dangers-of-not-getting-your-pet-enough-exercise/
https://www.sniffspot.com/blog/dog-exercise/signs-your-dog-isnt-getting-enough-exercise
And it's not just about the risk of the dog, but the damage done to the dog when it's mental and physical needs aren't met - you're talking obesity, depression, anxiety, joint issues, early arthritis etc. It can also manifest in socialisation issues (reactivity, nervousness, snapping, lunging, barking).
We were assured over and over that labradors are friendly and great with kids.
Labs are lovely and great with kids. But that's when their needs are met. Any dog is a risk around children when their needs aren't met. It's nurture over nature, every single time, when it comes to dogs. A good breeder can set the foundations, but it's up to the new family after 8 weeks.
I say this as gently as I can, a working-line labrador adolescent (and particularly a male - male adolescent's are dickheads) with two primary aged children and both of which have SEN and two parents who work full time and have never had a dog before...is a risky move. The breeder and the vet were so incredibly irresponsible in pointing you to this dog.
I feel for you, as it sounds like you're really trying your best - you just didn't realise what it would be like. Dog ownership is rough, and quite a shock (because no one really talks about how horrible it is after puppyhood) and it feels never-ending at times. My girl is just over a year now, and I'm finding adolescence rough (and I go through this every 3-4 years...).
But, if you are committed to giving your dog what he needs, then it's entirely manageable and in time you likely will have a lovely dog. But you have to put in the time, effort and money. FWIW, it sounds like you genuinely want to do right by him and want to give him the life he deserves - which is better than most dogs have (a lot of people simply do not care if their dogs needs are met). I don't see any reason why you can't keep him and work on his behaviour.
You don't even necessarily need gun-dog classes, although that would be helpful. I've never taken my Goldens. I do their training in my back garden and then in fields near the house myself. I learnt through trial and error (in the day before you-tube videos - I sound so old...) and none of my dogs suffered from that. It's just about finding the time (and money) which I appreciate can be hard with young children. With dogs, engagement and persistence is key.
Sorry...that got VERY long!