I had a similar experience with DS1's clip-board carrying nursery teacher 14 years ago. His speech at the time wasn't very clear, and he was quite shy and clingy towards me, although he did love playing with the other children.
I don't think she used the words 'not very bright,' but she wearily relayed to me towards the end of his time there how he struggled with this or that aspect of education and socialisation.
I was stunned, but managed to point out that as far as I was concerned he was there to learn to interact with adults and other children, and he would be able to focus on letters and numbers at the appropriate stage - ie school. She back-tracked at this point and said, of course, it's ridiculous to judge little children on these criteria etc etc. Since her role at that nursery did seem to mainly involve walking round looking officious and carrying said clipboard, I took everything she said with a pinch of salt, although it was hurtful at the time.
He actually turned out to have glue ear, but took a few years to really come into himself and perform at his true level. I think he was just a bit dreamy and bewildered for a while. He's now at grammar school studying a-level maths, further maths, physics, history and politics. Wants to do a maths degree. Put that in yer pipe and smoke it, Mrs miserable nursery teacher.
In hindsight, I probably knew it wasn't a good nursery, but this review happened towards the end of the year, plus he was my first, and it was close to home, and attached to the highly-thought-of primary school yada yada.
You're at an early stage, though, so maybe you could look around for somewhere with a kinder, more appropriate approach to little ones - assuming, as others have said, you have no concerns about his understanding and general interest.