Firstly well done for rescuing him. No animal deserves that.
I too believe that this type of breed can be amazingly loyal and gentle.
However, should any individual dog of this breed type choose to 'turn', either through fear, mistreatment, being startled or even residual mental health issues from past treatment, they have the physical power to kill. I am not saying that they will do this, just that they have the physical ability to do so. This is why people fear them and why I've taught my DD to recognise this physical power in any dog that has it and respect it with distance and healthy understanding.
There are of course Dogs without the physical power in their jaws to injure people to the same level and are therefore perseived in a very different way.
For me this is pure common sense and human nature - that people instinctually recognise or sense danger in an animal that has the power and strength to maim or kill should it choose to do so.
It in no way reflects the individual dog or how that particular animal will react of course, but you cannot expect people not to recognise the potential, especially when they do not know your dog personally.
So my answer is both:
YANBU - because you love your dog and (from your perspective and knowledge) he will do no harm.
YABU - because it's perfectly reasonable for people to instinctually feel caution around an animal with the strength and bite power to do so much harm should it choose to do so (despite it being unlikely).