update** thank you everyone for your help and support. We have been working on a lot of the suggestions and although things are not perfect, they are improving. We are working on the Quiet command, although with varied results so may be a much longer process, we have changed where we go for walks, very isolated, big fields where he can see who is coming so he doesn't feel trapped which is making a huge difference. I've also been rewarding recall with a whistle and some treats on every walk like we did in the puppy days, and he is getting much better at realising he needs to listen.
The barking in the house goes on, only at me and when DH is not in the house. We have all got better at not reacting at him when he does it, despite the incessant in your face barking, so that is definitely a help. It is beyond frustrating though and exhausting for everyone!
I have joined a kelpie facebook group which is also giving us plenty of advice as we don't know anyone else with Kelpies and as a breed they do things differently so it is fantastic to hear from others with them.
We are starting to teach him the stacking cups game to help with the mental stimulation. There have been a lot more games with him and the kids, learning tricks etc, so all that is helping to keep him calm. We still need to check on the pain aspect, he doesn't present as though he is in pain, but it still needs to be looked at as a possible cause so we will investigate further.
He has worked out how to take the muzzle off, so we are using it sporadically and sticking with using the lead but if the field is empty we can let him run which is a delight to see. Eifla, he is scared of big bouncy black dogs so would never go for your beloved dog, but I get your point we have to be vigilant at all times. Believe me, we are trying and are very aware of the responsibility.
All in all, we are putting a lot of time and effort into making sure he is happier and we are giving him what he needs so hopefully he will continue to improve.
Thank you, everyone, for giving me practical advice when I needed it. I live in a very middle class area where the reaction is he must be Bad and should be Put Down so this has been quite literally a life saver.