Yes, it's the first thing you're taught in safeguarding. You have to get out of the habit, if something you see concerns you and wondering what to do about it, of thinking "what if I'm wrong?" and instead think "but what if I'm right?"
It's not a judgement, as some posters seem to think. It's not even an assumption. It's someone with half a brain seeing something and thinking "There is the slightest chance this MAY indicate something more." Of course there may be a whole box full of toys stored somewhere out of OP's sight. It would be fairly easy to bring the conversation around to toys/tidiness or asking if they could use x/y/z toy that their sister was giving away etc and see what this mum said. She may well say, "oh there is a cupboard upstairs on the landing FULL of toys. They really don't need any more but thankyou." As OP gets to know the family more she will soon get to know if the children sit there all the time during each visit, or if they disappear upstairs to get something to play with and bring it down to show OP.
Even in cultures which don't have plastic tat, or in poor countries where they couldn't afford it anyway, they usually make toys out of whatever materials they have to hand. In years gone by in Western countries, parents made dolls out of rags, or pegs, and encouraged imaginary play. Pull along toys made out of cotton reels for wheels, string etc etc. A mini army of soldiers made out of bits of wood with the figures painted on etc. Boats made out of an empty coconut shell or walnut shell, cocktail stick and paper for a sail. Dolls houses made out of cardboard boxes or orange crates. Lots of boys had home made gokarts.
It really is very unusual for children not to have playthings. I would definitely try to establish, sensitively, if there was SOMETHING for the children to play with in the house, even if it were hidden away on your visit. Children's books are also easy to come by from the library or for pennies, or passed on by friends and relatives. A house with no children's books whatsoever is a bit sad.