I would keep it too. The fact that he uses is occasionally shows that he likes it being there. If you want to get rid of it eventually, then you could slowly move it towards the door/cat flap with a view to eventually putting it outside. But don't move it far at once - maybe just a foot at a time - in case he loses track of where it is.
We have 2 trays (one upstairs and one downstairs - DP is a big sook and didn't want the kitten to have to go all the way downstairs to pee at night!) and when we tried to take one away the kitten was NOT impressed. He peed in a gym bag, the laundry basket, on clean laundry etc etc until we put it back. He hasn't peed anywhere naughty since we put it back! He is too wee to go outside, plus we want to wait until he is neutered, so we have to have at least one tray but they don't seem to take kindly to change. We re-arranged the living room yesterday and he has slept on top of his scratching post with his back to us or in the middle of the dining table ever since
If he's using it, if only on a weekly basis, it's good security for him because he knows it's there. Particularly as the weather worsens and he'll stay more inside - hopefully with you !! I'd leave it if you can. Although removing it may cause him to use outisde, it may unsettle him further.
Hope you're still sane, SirSugar. If you'd posted Catgate on this board you'd likely have got an 'affirmative action' posse mounting up.