As everyone else has said, a good level of training and manners on the ground are absolutely vital for everyone's safety including people around you so in one sense I understand your yard people's concern if you're inexperienced and have a rude/bargey horse, but how you get there is the thing and for that you need an understanding of horse behaviours and psychology. Horses are unique animals, they're not like dogs or humans - they're domesticated prey animals with a strong flight instinct who live in a near constant state of alertness and fear, which drives a lot of the behaviours that we attribute to 'naughtiness' - even for instance when my horse drags me across the yard to get to his food or puts his ears back if I approach him while eating, I may think it's just him being greedy which on some level it is, but also there are a lot of herd dynamics at play about who gets access to food and when, and it can be a real anxiety trigger for a lot of horses in a way we just don't understand. Plus while they are intelligent animals in a lot of ways, their understanding of human communication particularly language/verbal cues is quite poor, between horses they communicate pretty exclusively using body language, as vocal as they get is squeals, screams or nickers. Getting angry with a horse rarely works, nor does shouting or hitting or other forms of 'punishment' - the horse may well feel a lot of fear and aversion to whatever your punishment is, but they're unlikely to make the causal connection between the behaviour they were showing and the consequence, unless your timing and communication with them is absolutely spot on - and equally in the same way I see a lot of people who are doing their best to train using more 'gentle' positive reinforcement based methodologies like clicker training because again not all horses find it easy to understand the causal link between a positive behaviour and a reward (especially a food based reward) either - you can clicker train a horse but it requires an awful lot of skill, much more so than a dog (or a child 
) I'd say.
Timing and communication in a way the horse understands is absolutely vital, which yes usually entails some kind of physical response as that's what they understand best - which is why touch of some kind whether using sticks or spurs or bits or poll pressure or flags or ropes or whatever aid you use is a vital part of every horse training methodology across the world.
I think as a first time horse owner it's really, really helpful to enlist some professional help in working with your horse on the ground, I don't know why there's such a sense out there that this is somehow weak or a sign of failure. It doesn't have to be full on 'natural horsemanship' waving flags around and joining up unless that's your thing, it can be a more traditional teacher, but just some input into how to encourage the behaviours you do want and swiftly and effectively correct those you don't, in the very basic settings of your everyday handling routine and surroundings is so helpful. In the meantime rather than getting angry or overly physical as your yard people are so helpfully suggesting, I'd do two simple things, firstly just don't put yourself into a position where he's likely to be misbehaving with you in the first place, if you can avoid it. So if leading is an issue, minimize how much you need to lead him around - don't keep moving him from stable to tie-up to back to stable to wash down to the arena back to wash down etc etc unless necessary. If towing you to food is an issue try and never lead him to/from places where he's imminently about to get food, get into a routine where he's contained somewhere safe first, then food comes - my boy's manners go to pot wherever food is involved so for instance he is never, ever allowed to graze in hand or be fed treats in hand because I know it blows his mind, and I try not to even lead him past the grassy patches where I know other people hand-graze as it's just asking for trouble, if I need to go down there I will do it from on board as he understands much better than being ridden is not the time for eating. Or if it's about chasing towards or trying to get away from other horses, again avoid those situations altogether, bring him in first or last from the field or with one well behaved friend, whatever works. Second - when you do need to lead or do whatever is problematic - use your physical aids, not to whack or pull or hurt him in any way but for communication - presumably if he's broken to ride he understands bridle aids so lead him in a bridle and if you need him to stop or back up use your half halt just as you would from on board (you can also get pressure halters like a Dually or a BeNice but I wouldn't use these if you're relatively inexperienced until an instructor has shown you how they work). Practice going round a quiet area without distractions in hand, stopping, backing up, moving off again then stopping again with lots of praise when he gets it right, until he understands and responds to a quiet aid. You can carry a schooling whip in hand too if he naps or plants just to tap him hindquarters and remind him to move forward. Patience and consistency are the key, it's easy to fall into the trap of ignoring low-level poor behaviours as you can work around them and only starting to try and correct when it's a big/dangerous problem but of course again horses don't understand why a little bit of pulling or barging is sometimes OK but sometimes not, you need to be firm on the boundaries all the time and have the means to back this up where you need to.