So am I. And I think you're correct in your assumption that discussion and rhetoric surrounding the poppy is becoming much more incendiary of late. I've not seen that vitriol directed at those who say they do choose to wear the poppy. However, there's been a significant amount of it levelled at those who do not.
There is constant social pressure to show visible signals of support for political issues, or to acknowledge certain public events (usually the sort that some people would like to see 'unite' the nation). This often entails indicating your support/acknowledgement by displaying some sort of symbol: rainbow lanyards, pronoun signatures, Palestinian/Israeli flags, coronation banners, banging saucepans at 8pm on a Thursday and, yes, the poppy. These are a convenient and immediate way to show that a person is on the 'right' side of whatever political/ideological belief they're signalling an adherence with.
I don't happen to want to. I'm a republican and oppose the monarchy, I'm dismayed at the mess in the middle east and deplore the methods of Netanyahu as well as Hamas, and I don't believe the rainbow stands for what it once did. But in the sector I work in specifically, it's undoubtedly easier to comply with these practices than it is to resist them.
Obviously, non-participtation by one individual isn't going to change anything, apart from perhaps causing a few eyebrows to be raised in your direction. But I hold with acting in accordance with my own conscience in a meaningful if private way. A fundamental democratic right is the right to remain silent. I'm not under any illusions that the rest of the world is queuing up to hear my opinion. Everybody's got one.
On this thread people have taken the trouble to explain their reasons for non-participation, whereas in an offline context they likely wouldn't (and shouldn't be expected to). It's a question of exercising that right as you see fit, and understanding that others will take a different personal decision from your own. When you live in a democracy this is inevitable.
It isn't a personal affront.