We pulled our son out of one of the Oxford choirs after a year as a probationer, and it was the best decision ever, though painful at the time.
It is an extremely pressured environment, the need to conform and fit in is overwhelming. While in a choir, it is virtually impossible for the boys to have any interests or any free time to play and be a child. You mention that your son is being bullied a bit; in my opinion, the choir atmosphere tends to be inherently profoundly bullying, with little room for individuality or self expression.
While you are involved in a boys' choir, it is so absorbing that life hardly exists outside the choir. The boys cling to it, as the only reality they know. Your son sounds more attached to the choir than ours was, but I do think that you can probably help him to see all the positives about leaving, and keep him in the dark about why he is actually not continuing with it. (All you might need is one very tempting club or activity which directly conflicts with the choir schedule.) Once you get out of the choir, the relief can be immense and very liberating.
My husband and I are both musicians, as you are, and I think I understand your feelings, but truly, not being in a choir is a great blessing for most kids!
Finally, If your son turns out to be serious about his instrument, a choir hardly leaves room for him to develop instrumentally...