I think you're right to be wary, GrumpiestOldWoman. The usefulness of such automated checkers depends on the situation, how sophisticated they are and how they were programmed.
By now we have numerous studies showing that such programs are typically coded by men unaware of their conscious and subconscious bias who end up programming something that discriminates against women.
So I would use the checker if I had to but make my own choices.
Chairwoman for instance is an incredibly powerful word. Most boards continue to be chaired by men, chairman is the default for most people. And even if you do use "chairperson", most people picture a male chairperson. So to write chairwoman when the person is an actual woman or you write about a hypothetical chairperson is an important counterweight to our ingrained assumptions.
ChateauxMargeaux is right that this erases women. Caroline Criado Perez devotes a lot of pages to this whole issue in her book. It's another one of those things that masquerades as progressive at first sight, but turns out not to be when you dig deeper.
And yes, this whole movement to change biased language was started by feminists, but they did not know then what we know now - without changing minds, changing words makes little difference and the best approach may well be varied - sometimes adding female versions is more effective and sometimes a neutral term works best.