OK
First read "The information and Advice Leaflets" section of this website here "Hey look at me, I'm a house rabbit would be a good place to start" Loads of info on neutering/toilet training too.
More specifically, have a look here about rabbit handling.
Rabbits do not like being picked up so you need to change the association (very gently and gradually over a long period of time) from being a hateful experience, to a positive one, usually with the use of food that he likes and possibly clicker training (yes, rabbits are a lot more intelligent than everyone thinks!).
There are also lots of videos on You Tube about this sort of thing. You want to nip the problem in the bud before it turns in to aggressive behaviour and biting etc.
In the meantime, if he needs urgent care from the vet, it sometimes help to gently but firmly wrap them in a towel, and/or cover their eyes.
Grooming every few days also helps with the bonding process between him and you and helps to familiarise him with being handled. (Also a good way to check out any odd lumps and bumps.)
One last point, as they are group animals, it's not really ideal keeping a single rabbit on its own. He would be better (and possibly more relaxed) with a spayed female (properly introduced as described on RWA website). But if he has loads of interaction with you and your family, then it is not as bad as being totally alone and shoved in a hutch at the end of the garden, as lots of rabbits are sadly.
HTH! Good luck with it!