Hey OP,
I don't think you're doing anything wrong, especially with the frequency with which your going, nothing at all. Agree with some other posters, you should explore why your DS doesn't want to go anymore. If you think hes got a valid reason, then maybe you could respect that. (He is five though, I wonder how well thought out his feelings can be?)
I find it really interesting (although ofc undersrandable) that many people find graveyards morbid. I generally don't have the same issue - as a kid (ie from when I can remember) I used to be taken frequently by my Italian DGD who was widowed young to my DGM and other family graves. Never really found it morbid, it was always more a sign of respect, stand quietly for a minute before he'd tell me a funny story about someone, then change and water the flowers, and often stop and chat to someone else visiting their deceased. I guess that's Italy for you, and I do wonder whether that's a typical continental attitude.
He passed away last year, and was buried a few days before Xmas next to gran. On Xmas day I went over to check everything was ok, ended up chatting with one of his old friends who was visiting someone else, and went for a casual coffee after. I find it hard to associate going to the cemetery with morbidness because of this stuff - its just part of your day. My great aunts visit everyday to check and water the flowers and say hello to him, which is very typical. Ok they're quite Catholic but they don't go crying by his graveside, it's much more practical, happy and chilled.
That was always how I felt as a kid. There were many times I didn't want to go, but he'd drag me there because the main reason I didn't want to go was because I was lazy (and also looking back probably couldn't leave a small kid at home). Mum, on the other hand never used to come with us, and doesn't go now as she gets too sad, so who knows what's good for us?
Basically, you do you OP