Hmmm. Does he seem particularly unhappy or distressed? Is he very distressed when his Mum leaves him?
I am not a minder - but my DS has cottoned on to this sort of thing, having told both my sister and my bf when they were babysitiing for him (and he loves both dearly, and has a great time with them) that he didn't like them, didn't want them to look after him, wanted me, and would they go home?
My bf distracted him (aka spoilt him rotten), my sister calmly and firmly said 'well Mummy will be back soon, in the meantime, I AM looking after you, I am NOT going home, so that's that, lets have a good time'. After a teeny mini sulk for show, he got on with it and has never said anything of the sort since.
This little boy may well be trying it on - I wouldn't assume that it is you, your house or activities that do not meet with his pleasure!! But would discuss it with his Mum before taking such a tack.