but mostly I just want to walk with him and do our thing
I think a great many behavioural problems would be helped by owners thinking a bit more like this, tbh.
There is a lot to be said for an owner spending more time getting and maintaining a dog's focus on them rather than allowing it to have an overly exciting time playing with other dogs.
If I had to think about how many times an 'overly' cautious owner bothered me or my dog it would be zero. If I thought about how many times a overly relaxed owner and dog with poor recall around other dogs bothered us - it must be close to a hundred. Often not big issues, often just disrupting our walk or Battendog's concentration when we are off doing something specific, or nicking treats out my pouch which also happens a lot if I don't guard them.
I also don't think it helps nervous owners to have those with more confident dogs sneer at them and think "no wonder they have nervous dogs". Nervousness and anxiety to situations is no more under conscious control than our heart rate it and no of us have any idea what that person's experiences are or why they might be nervous.
Behavioural theory is behavioural theory, whether it's a dog or a human. In both species, a stressful situation is only ever made worse by the thought that someone else present is angry, upset, stressed, grumpy etc. It is only ever made better by that person being upbeat, friendly, happy etc.
How on earth would a nervous owner ever be encouraged to allow dog-dog interaction if they constantly got the impression the other owner was judging them? A smile and move on can make all the difference to the other owner and, by extension, to their dog.