You are doing the right thing. Listen to your own feelings as a mum, you don't need validation from anyone else. However you say grandmother doesn't defend his behaviour, so she also knows that he tramples boundaries and she's willing to visit you without him. She understands.
Children remember with disgust the unpleasant experiences they had when young of being forcibly kissed, held or people saying weird things to them. It's even worse if their parents and other adults don't defend them, and expect children to please adults even if it means being uncomfortable themselves.
Ignore the apologists who inevitably come on these threads to tell you you're overreacting or "he just loves his grandbaby". They either have never experienced boundary crossing themselves, or they have no healthy boundaries, or don't believe children have the right to say no to bodily contact.
My parents didn't defend me. I was expected to defer to the wishes of adults at all times. I had to endure endless unwanted kisses, hugs, sitting on people's laps and photography in what should have been private situations by relatives and family friends, some of whom were strangers to me. At a very young age I was raped by a family member, and I never forgave my parents for leaving me with them. Don't let people say this is terrible but very rare, because it isn't.
It's no exaggeration to say I hated my parents for not protecting me. There's better awareness now of children's right to bodily autonomy, which wasn't a thing when I was a child. All parents now should know that interactions between children and other adults should only be allowed if beneficial to both parties. Children aren't props for entertaining adults. If the child doesn't want the interaction, they mustn't be forced into it.
And I'd bet my lunch that most children don't enjoy being given smacking kisses on the cheek. It's something adults do because they like it.