Totally agree with @Puddlelane123 here
The OP is a young girl who having lost her mother in the most traumatic fashion (whilst pregnant) merely wants to make a short weekly trip to her grave to honour her memory. Knowing what I do about mothers in general, and certainly single mothers without family to help, this is quite possible the ONLY thing she does for herself all week. Of course she shouldn’t be made to feel guilty for that.
Please don't listent to people saying its twisted or you are damaging his health. You really aren't. He's 4 - 5. He's fine. He might have decided he doesn't want to go any more but that could be in the same way children decide they don't want to see Great Aunt Muriel because she has whiskers and wants to kiss them. Children have all kinds of random reasons for not wanting to do something.
It maybe a simple as he finds it cold or boring. That's completely different for what works for you.
Your priority actually is your mental health. If you aren't fit and well in the first place you could never have cared for your son in the way you have. If coping at such a young age while you were pregnant with the suicide of your mother (for which deepest condolences) involved visiting her grave once a week, then good for you for looking after yourself.
If its very important to you to go on Christmas day then explain that to your son and take him. If he's been going weekly for years, one more week isn't exactly going to cause an earthquake.
Just do what feels right to you and if that's listening to your son then find another way to honour your mother.
I would ask him gently in passing about why he said that. It could be some trivial child's reason that will reassure you.
Either way, please don't listen to some of the cruel and insentive comments here. Most of us were made to do stuff we didn't want to do as children on a scale from eating sprouts to visits to Great Aunt Muriel to extra home work. We are all still standing.