There's a good reason to have certain symbols, in the UK really it's the Crown, that are meant to be above political issues.
Of course there are plenty of people who see that as political too. And inclusion in terms of citizenship and the law is often imperfect.
But that is, explicitly, the meaning of that level of symbolism. In countries like the US, it might be the flag, or the constitution. But intended to represent the people, the nation, as a whole, including all people, whatever their political views. It's by that ideal that we're able to make criticisms of the reality.
I think it's fair to say that only symbols that represent that total ideal should be used. SO probably, the flag, and a few others.
The poppy becomes controversial, I think, because for many it is that kind of symbol, it represents people who have served the nation. And it's not quite enough to say, well, some are against what it stands for - some people are against flags, the monarchy, even the idea of the state itself in some cases. Yest those are the institutions through which their political views are given a place in society.
So it's perhaps a more difficult case to judge than some others.