Cats are territorial and, as he's a rescue, you do not know his past , anything that may have happened to him which will affect his behaviour especially with other cats. Usually it's a female who is the local 'Queen' so perhaps he has been in a situation where a Queen (or two) has dominated and possibly hurt him when he was younger. Which would explain his dislike of other females..
Keeping him in at night is a good idea, most cats tend to visit their neighbourhood gardens and territory so if a few are out at the same time, they will 'meet' each other.. and there will be fights, especially if they are older (kittens and young cats will often make up little 'bands of brothers & sisters')
If he was mine, I'd advise the neighbours to have a water scoocher on hand so, if he does appear in their garden and is being aggressive, water on him won't hurt but will discourage him and hopefully he'll learn... I had to do this with a large male who terrorized my own cats in our garden.. even after they submitted to him... it wasn't pleasant but he would have hurt my cat if I didn't intervene.. he eventually got the hint and stopped coming anywhere near us.. especially when he saw me!
If that doesn't work, make him a house cat. It's not that bad. Get a cat tree, scratching posts and some toys to play with him with. Cardboard boxes are good too - cats love them!!
As they get older, cats do slow down and spend more time at home.. my 'old boy' is still a hunter but he spends most of the time in the house and only goes out to visit the neighbours or for the loo!! My wee cat is indoors all the time - they are both happy!! Good Luck!!!