Yes, I would be worried.
Something similar happened at my dd's previous setting. I took her out forthwith when the staff told me "we didn't see anything, we didn't hear anything, so we don't think it's serious. Besides, she didn't have any marks and bruises when we checked."
Needless to say, I didn't return my dd to that setting. Just thinking about it makes me .
She now attends a wonderful nursery school. Because the children are more 'mature' and have better speech, any child that tries to hurt another child is swiftly reported to the nursery teacher, who is very good at dealing with these things. The nursery school also has a 'buddy' system, when the older children look at for the younger ones.
The staff at my dd's previous setting were basically telling me that my dd had made it up. I didn't appreciate this type of behaviour from the staff.
A month after attending the nursery school full time, my dd confided to the nursery teacher what happened at the previous setting. Action was taken, because I'm informed via the grapevine that things have changed at the previous setting. Whether it's because of the things my dd told her nursery teacher, or because there was another perhaps more serious incident at the previous setting, I don't know and I don't think I'll ever find out.
What I do know is that I did the right thing in changing settings.