It's definitely the way it's said. Yes maybe your grandparents said it, but I know my mum (who is also irish) is offended regardless of what nationality the speaker is!
My first summer working in a bar in England (Holloway) aged 17, couldn't hear the man's drink order due to the road works outside. He says "sorry love, it's the paddy's on the road" I felt embarrassed for the both of us and stupidly was shy to speak and reveal my irishness!
As a "proper adult" years later, I lived the last ten years in Cambridgeshire and not often but regularly enough, had "threw a bit of a paddy" said to me by educated people - twice I gently pointed out the phase is a bit racist on the irish, they were complete clueless and a bit mortified (I felt bad as it is total ignorance, but unlike people on here they didn't resort to justifying it!).
Was in the room listening to a telling of an English/irish/Scotsman joke, a retired man telling it with a twinkle in his eye. Didn't know I was irish. Not a funny joke either, just a blatant "the irish are stupid" joke.
Why are we debating whether or not it's offensive - offence doesn't need to be felt by the vast majority to be legitimate - obviously it is felt by plenty as per this whole thread.
Stereotypes die hard. Re the English/irish/scotsman joke. Somehow it's really really shit when an adult with a lovely posh accent tells a blatantly nasty joke about the irish though. And assumes the room is in cahoots with him.