Sorry OP, I share your frustration, I have exactly the same issue with my mum and it's been reassuring to read responses saying it's not ok.
We live a fair distance away and almost every time without fail, since my nephew was born, he tags along at the last minute when I arrange to meet my mum. It's been going on for years now, and I have raised it repeatedly as sometimes I really just wanted to spend time with my mum and frankly doing that with a baby / toddler / young kid in tow is hard work and means you have to change plans (my Dsis also never comes and meeting my mum at a soft play centre never filled me with joy). I was similarly accused of not liking kids.
Now I have DC of my own and there's still no change. I'll be visiting my mum for a weekend with DC and my Dsis will drop off nephew for the whole day or overnight because it will be 'nice' for the cousins to spend time together. Or worse, on the rare occasion I'm without DC the same thing will happen. My Dsis barely stays long enough to say hello before she's gone.
It feels like I'm being used as childcare as I'm left with my mum looking after 3 kids rather than 2, they don't all play well together and the worst bit is that unlike my nephew my DC never ever get time alone together with their granny.
I have no solution. At times it has put a big strain on the relationship with my mum, who I'm very close to, as I do feel second best to my sister and her needs (and my DC also come second). I slowed down visits a lot and told her why, but again, I know I'm just labelled as a miserable person who doesn't like spending time with kids (even now I have my own kids!). Sorry to not be able to give advice, I do hope you manage to find a better solution than I have!