Catinboots - YANBU IMO.
You said your son isn't bothered whether he plays with this boy or not.
When he comes over why don't you just say that your boy can't play today. The boy will soon get the message.
If your cleaner is one of those rare (it seems to me) assertive, direct people, who asks you why little Johnny hasn't been able to play recently you can just say calmly that things have been busy recently and also you felt the two boys were a bad influence on each other - see below
I have a similar situation with two different children.
One, who comes over a lot to play and often causes arguments with my dcs. I have "trained" my dcs to always ask me if they are allowed to play before just letting her in and starting to play. If it doesn't suit that day, I say no and send the child home. Easy. If I do let her in and trouble starts, I send her home.
Also, my ds was having trouble with another boy and I spoke to the parents but rather than blame their child, I emphasised the fact that they were both lovely boys but seemed to bring out the worst in each other, although secretly I blamed the other boy as my boy is an angel . They agreed to greatly lessen contact between the boys.
Hopefully you can tackle the situation without losing your cleaner.
Why should you have to put up with a boy who sounds dangerous and a pain? No reason in my view. HTH.
PS If your child did fall off a roof or drown in a loch, I expect you would rather have lost your cleaner. Food for thought. good luck.