Oppression of women in predominantly muslim /Middle Eastern countries is nothing to do with islam persay as both christian and muslim women in egypt for example, have very similar lives.
Religion on both these 'sides' is used as the big stick to keep the female in her place.
Islamaphobia is a made up word which nothing short of prejudice and racism.
People who admit to this are not 'afraid' of muslims. They hate them. Their reaction to seen a muslim is not the same as a persons suffering from a different phobia. They are not like an agoraphobic, afraid, shaking and prone to panic attacks when they see or find themselves in the midst of of groups of people or at being outside.
My mum has a phobia of bats, it means she literally shrieks and runs for the hills whenever she she sees one, be it on the telly, flying about, in a magazine or in any other form. My dad is scared to death of Snakes.
So in answer to the question, is there anything wrong with being Islamaphobic?
Yes, there is. Plenty.
Because this phobia doesn't exist. It's simple prejudice.