Above all go with your instinct and do what you want to do - I know we're all busy offering you advice here as well but ignore it if it feels wrong
Indeed. So feel free to ignore what follows 
I'm not big on dominance and hierarchical models when understanding what horses are up to. Studies by ethologists such as Lucy Rees on feral ponies give very different interpretations of equine behaviour. And a 5-year study of zebras indicated that wild equines don't dominate each other or live in hierarchies, at least not simplistic ones. We do see aggressive behaviour in domestic horses and competition for resources, but that's more specifically tied to the conditions under which they're kept.
Dominance just doesn't make much sense to me. Humans have a clear concept of dominance. We're aggressive territorial primates and hunter-gatherers. We have the physiology and anatomy of hunters. Horses are cooperative grazing animals with the physiology and anatomy of prey animals. Dominance just uses up a lot of energy to little end when you can save energy by cooperating with each other.
Training-wise, you can get results by "dominating" a horse. If you smack it when it shies it will quite quickly learn not to shy with you (though not always, its fear of whatever it's shying at will sometimes surpass its fear of you smacking it). However, I do think that as herd animals they prefer to be around confident people, which might then be interpreted as dominating them.
So, Puppymouse, I think when you took your horse round the village and he refused to move forward, you were tense. You didn't want to go, you were with someone you perhaps don't trust that much. That tension transmitted to him. Someone in his herd was nervous - and in horse-think that means there maybe a predator around, so he was also nervous and on the alert. I'm in no way blaming you for being tense - well all get like that sometimes, it's completely natural. However, when you're dealing with a herd animal evolved to flee when it picks up on small signals, they do pick up on tension and react accordingly.
The next day you were with the yard owner who you probably trust more. You were in a better frame of mind, you were relaxed and you were on diazepam (I think, apologies if I'm wrong, I prefer gin). So you were calm and relaxed. If he was looking to you for a signal, what he was getting from you was "happy, happy, chilled". So he went through the village without an issue.
But I agree with Frosty, go with your instincts. Take time to develop a relationship with your horse that you enjoy and that's safe for both of you.