OP, it's your decision. She sounds awful, and I wouldn't want her doing childcare for me, because I wouldn't be able to trust her. I suppose her doing childcare is her idea, and your husband wants her to do it. Is that correct?
The thing is though, if she's looking after your child one day a week, she can just take your child on these trips then, and not tell you. You need to stop using her for childcare.
There's nothing wrong per se, with a grandparent wanting to take their GC out for the day, but it depends on how old the child is, the distance involved, whether the mother trusts the GP to keep the child to their routine, as much as possible, and whether the GP will also adhere to any other things the mother asks them to do, or not to do.
If you can't trust them, then absolutely you should say no.
You need to kick up more of a fuss than his mother, so your DH takes your side, instead of hers. You are the one he's living with, who he wanted to have children with, so he needs to back you up. Otherwise, he can go back and live with her.
And I say this as a grandmother, and when I was a young mum, I knew I could trust both sets of GPs, so my children often went out for the day, or even on holiday, with them. But the difference being, I trusted them, and the children were a few years older then.