You made a happy agreement with him that he would no longer go on lads holidays.
His mates are badgering you either because he’s asked them to (or with his agreement) or because he adds a certain something to the group that they miss when he isn’t with them, eg maybe it’s because he's the life and soul of the party.
If he’s made an agreement with you that he isn’t going any more and then asked them to badger you, (or agreed to them badgering you) then the agreement he made means nothing, in which case you are back to square one. Stopping him going when he wants to go isn’t the solution to the problem of lack of trust in him.
Does he know they were badgering you to let him go? If his motives are good and he knows they were doing that, he would have put them straight immediately telling them …it’s not that she won’t allow me to go it’s that we have agreed, for our own reasons, that I’m not going so please drop it, it’s my decision, you are interfering and it’s none of your business.
Making an agreement with you not to go isn’t controlling at all as it’s done with understanding on both sides. And the agreement is kept with good will.
Stopping him doing something he wants to do IS controlling. But you aren't controlling because you talked about it and came to an agreement. The agreement stands. Neither of you has retracted it.That’s good.
So you don’t need to ask if you are being unreasonable to stop him as you aren’t stopping him, the agreement you have with him not to go stands. Talking about problems and coming to an agreement is not unreasonable. In fact, it’s very positively reasonable.
So the question is … does he know they were badgering you and what was his response. If he doesn’t know they were badgering you, you need to tell him and see what his response is.
In other words, is he reneging on his agreement with you or not?