But IMD seems to be based entirely around how men allegedly have it harder and are now apparently the only real victims here.
You're hugely misrepresenting the point of the day. If you genuinely believe that, you've either never bothered to take any notice, or you're surrounded by idiots who tell you this and you believe it.
"Men are the only victims, there is nothing wrong with anything in women's lives, men have it so much harder than women do"
vs
"Men can be victims too, maybe not in the same way as women, but both genders have problems. We have IWD to talk about problems specific to women, and IMD to talk about problems specific to men. Both are important, because ignoring the problems faced by ANY demographic is harmful, regardless of their relative status"
Saying "You shouldn't have IMD because men have it easier than women" is exactly the same as saying "You shouldn't have IWD because heterosexual women have it much easier than gay people".
I'm sure you could also find a demographic that has it worse than gay people as well. The point isn't that we ignore everyone while focusing on only the problems of the worst off in society. The point is that we address the problems facing ALL of us, together, as a whole.