yanbu. your child, your decision.
And am sorry to say this but I don't get this whole "children should have a relationship with their grandparents".
Of course it's lovely if they do, but if they don't it really isn't the end of the world because they know no better.
I grew up abroad and consequently had no relationship with any of my extended family. I now live in the same town as them all and wouldn't know any of my cousins on my dad's side if I passed them in the street, and I haven't visited my grandmother for three years.
No I don't remember times as a child being spoiled by them as these times didn't exist for me, but as I knew no different I haven't really missed out iyswim?
Having said that, my ds does have a good relationship with both my parents and ILs, but it is perfectly possible to have that without having to spend extended periods of time away from me and just with them.
ILs live 4 hours away and there's no way ds will be going to stay with them on his own any time soon - he'll be 5 next month.
but the relationship can still be cultivated with us staying there or them staying here etc.