How did it go at the vets? I would listen to their advice on this as he is such an elderly gerbil (and then I'd go with my gut instinct). I'm sure you'll know what's best for your boy and do the right thing for him.
Three years is a pretty good age for a gerbil. I had one gerbil that was about that age who had a paw removed that had a tumour on it and lived happily for another 4 or 5 months, but she was very fit and well at the outset. It felt the right thing because her quality of life was quite good after (RIP Ms Germaine Greer Gerbil.) I had another who had a scent gland tumour but he was in a lot of distress, quite old, not in the best of health and in that case, I decided it would be kinder for him to be PTS. God it was hard though (RIP Mr. Jack Hargreaves Gerbil, Esq.)
It's never easy, never, but you just have to go with the vet's advice and your gut feeling. Little fella will know you are doing the best you can for him, honest he will.