I hadn't heard this phrase from anywhere but the internet. Haven't seen Game of Thrones. Google reveals there's a debate over it, plenty of forums coming up with the same discussion. The Google overview agrees with you that the phrase in its sarcastic usage originates from GoT Fandom taking the quote from the book.
However, if I might suggest, there is a flaw in your logic, which doesn't mean you are wrong, it's just a note of caution.
I have read a lot. But there are phrases from my youth that don't appear in literature and Google wouldn't give you literary references. It is possible to find the phrase, for example, 'itchy chin', meaning, 'Liar', on the internet, because it was a popular phrase amongst schoolchildren in the 80s, but there are no literary references at all. Basically, nobody has used it in a novel. Or any of the variants, Jimmy Hill, itchy beard, etc.
Now you might say, 'Oh, but it appears on the internet. Well, it's not unreasonable that it would, because it was at it's peak in the late 70s/ 80's/90's. So the generation, then, that has the most prominent nostalgia post presence on the internet. The further back you go, the harder it is to find phrases other than the most well known.
Not every saying/phrase makes it into print, or even on TV. I mean, when did you hear a kid say, 'itchy chin' on any UK TV show even of the time, never mind now. I was born in the early 70s, it's part of my childhood, it wasn't region- specific, but they didn't use it on Grange Hill, Byker Grove, or any of the contemporary dramas.
So, the only reason that phrase exists on the internet is because someone posted about it as a memory at some point. And the primary reason they'd have done that is the reason I first looked it up. To find out why we said it, because it isn't obvious why that term would be used in that context.
'Oh, you sweet summer child' isn't necessarily a phrase people would have shared on the net in a 'why did my Nan say this' type of way, because it's self-explanatory, really.
I suspect there's a whole host of sayings we really did hear that don't appear in books or on the internet. Because there's no reason for them to. They don't stand out in our minds enough to post about them, and authors found they weren't effective phrases to use in novels.
Still doesn't mean you are wrong, but believe it or not, not everything we heard our grandparents say, or even that we remember saying, is going to appear on the net or a novel.