Yeah exactly. I've never seen "woke" used with any apparent intent to mock Black people, or to use the word's origin in Black American culture as a way of associatively denigrating the target. Not because those using it aren't racist, because I'm sure some of them probably are, but because they'll generally have started using it long before being told its origins, if at all. The use of the word as a jibe may have unintentional racist implications, but IMO that's less the fault of those using it to ridicule than it is the fault of those who decided to co-opt it for themselves and take it mainstream with a new meaning.
I don't really use "woke" myself (don't have much need to, and don't want to make people feel uncomfortable given the word's background), but for me, knowing the history behind it gives it another layer of meaning — a reminder that the person being referred to as "woke" might be the kind of person who'll superficially act like they're supporting your movement, but is actually perfectly comfortable taking what you've worked for, using it for their own ends, and ruining it in the process.
Daft thing is, I actually agree with the vast majority of the social and political opinions of those who get called "woke" 
The main areas I differ are in free speech (and no, not freezepeach bollocks, just the old-fashioned kind that our democracies are founded upon, imperfect though it is), and the value of trying to recognise, in thought and language, reality as it is. @Terryer's "When you start trying to stick up for Nigel Farage having his bank account rescinded you know you are on the wrong side." is a perfect example of the kind of authoritarian, anti-free speech, chilling kind of argument we hear from those who get called "woke". It's not clear yet, from what I've seen (though I haven't been following the story), whether there have been genuine reasons behind his account closure, but yes, I would stick up for Nigel Farage if he hadn't done anything to genuinely merit his account being closed, even though I despise the conniving, manipulative, hypocritical, self-serving little arsehole.