It's fine. I don't know why that would be weird.
Your child doesn't have to see all of his grandparents on Christmas Day.
My grandparents lived 4 hours away when I was a child, both sets. Some years one or the other pair would stay with us, some years it would be a great grandparent, some years it was just us. They all had other grandchildren and children who lived all over the place but even if they hadn't, there's no set rule that everyone has to see everyone on Christmas Day.
We've never seen my stepdaughter on Christmas Day, we've always done Christmas with her a day early so she has a full day with each parent.
My sister and I live local to my parents so we all have Christmas together (with grandchildren, parents in law etc) but my brothers live 3-4 hours away in different directions and if my parents wanted to go to them that would be totally fine. If they stay home or go to their in laws who are another 3-5 hours in different directions that's also absolutely fine! It's all fine! Everyone can Christmas wherever they want to Christmas!
Do you expect your parents to have to stay local for every Christmas forever? They're grown ups, they can go anywhere. If they're very local to you they probably see your child often throughout the year, which is lovely and gives plenty of options for a non-Christmas Day festive gathering and Christmassy treats and days out.