YANBU you shouldn't leave your child any where you don't feel comfortable.
I won't leave my DS with my parents cos my mum is schizophrenic and altho she is ok now, she can go AWOL very quickly and with a little bit of extra pressure (eg. taking care of GS) she could make herself ill.
Is that harsh? overprotective? PFB- symndome? Yes- but i don't want my DS exposed to some of the traumatic stuff i saw/ heard when i was growing up.
He's your son and you shoudn't feel obliged to leave him anywhere you don't want to.
Could you just say 'meet you at ....pm, then we can go for a coffee/ come to ours/ go to yours...'?