They are taking the mick. I'm ND also and sometimes worry but once you realise it takes the mick you grow a thick skin. I moved into same road as my parents - who have been childminders/Foster carers my whole life. Neighbours kids started knocking I let them in, but soon realised they were not interested in playing with my children but the garden equipment, free drinks and snacks and games machines. They would stay until 10pm They were only 8/9 year olds I felt so bad asking them to leave. Then they would knock and just ask for a drink - lol. I started to say no we are busy today and they backed off. If your children genuinely do not want to play let them answer and say they don't want to play and of course get a bolt for the gate. I bet your neighbours went through similar with them.