My father died 3 years ago last month, and he is buried half a block away from us. We live in a tiny rural town, where the hospital/cemetery is up the end of our street and round the corner.
Know how often I've visited his grave? Twice since he was buried. Twice. Since Dec 2 2016 (day of funeral), so 3 times all up. Not because we had a bad relationship, we didn't. We were very close. But it causes more emotional upset to go there than not. What is the point? Graveyards are morbid places and I find visiting cemeteries to be morbid and depressing. Psychologically harmful, too. I would ask yourself what you get out of it. Also consider the psychological harm on your son. He is understanding that other schoolmates visit their grandparents and are actually able to talk with them. How do you think he measures that up when you say to him we are seeing grandma? He knows he can't see her, or talk to her. All he can do is stare at a grave. What's the point? What do you think that would do to a child, especially at the school playground when his classmates are talking about seeing their gran, talking to them, what their gran gave them for birthday/Christmas, and, he can't say anything, because you take him to stare at a grave? I would hazard a guess that he is starting to pick up from his classmates that it is not normal to stare at a grave, when talking to gran.
Sorry but I think you are being very, very selfish and harmful. If you can't leave your son with someone, don't go. Plain and simple. Or go only on special occasions like birthday/mothers' day. I also would question yourself on what you get out of it? Going every week? What is the point? You can't speak to her! I always remember the 'do not stand at my grave and weep, I am not there' etc poem. I don't believe my dad's soul is in the grave, I feel it everywhere around. So, the point of going to his grave is, I don't see what.
After a while you need to move on with your life, and I would question that maybe you could do with some counselling/psychological help if you can't physically/psychologically move on from her death. I wouldn't be taking children to cemeteries myself, at least not every week. I think you really need to stop going, you need to find a way to move on. You need to be concerned with the living, that is, your son. And the impact going to a cemetery every week and staring/talking to a grave has on your son.