Well, now it is obvious that you are 100% in the wrong @Minimelanie! In fact I am now actively on your son's DDad"s side, as he obviously listens to, believes, and respects your DS. I know I am sounding harsh OP, and I am sorry that I feel the need to be. If someone had said something like that to me, when my children were the same age, I would have probably burst into tears, so I hope you are both stronger, and more mature, than I was all of those years ago. But, you did ask for our views, and that is mine.
I am only commenting because I want to implore you to always listen to your children, believe them whenever it is at all possible to, especially if they are talking to you about something that seems difficult for* them to say, or about something that upsets them, and even though he is still only little, please respect him as well. As for the bullying, *of couse children of five years old can be bullied, and very sadly, they can also be bullies, especially if that is the lifestyle they live in.
I think that you owe both your dear son, and his dad, apologies, and please remember that unless one parent is an abusive bully, they both have a right to make those sort of choices for their children. In an almost ideal world, and when possible, both parents should discuss things with each other when it is as important as their child claiming that they are being bullied.