I am old enough to remember the original Watergate from which all these 'gate' scandals take their name - it wasn't just the break in and attempted cover up, it was a symbol of a rotten to the core administration.
Similarly Partygate isn't about whether Johnson stuffed his face with a piece of birthday cake - it's about his failure to tackle Covid promptly, siphoning off public money to his already wealthy cronies, lying non stop, including in Parliament, illegally proroguing Parliament expecting the general public to be gullible enough to believe it and so on. I.e. equally rotten to the core as was Watergate.
Even Johnson's boast about his vaccine roll out, which was also spurious, is now losing traction since Omicron arrived and just about everyone has gone and caught it.