The reason it sounds extra odd is because that sentence introduces an indirect question (i.e. there is a question being talked about by someone. If I asked my friend 'Where are you?' and you narrated this, 'She is asking her friend where she is', the part 'where she is' is the indirect question). In very many examples of this type of indirect question we continue to use who, not whom, because the 'who' is the subject of the verb in the question clause (so if my friend is male it's not 'where him was', as the indirect question starts a new subclause with its own verb, regardless of the fact that I am the subject of the whole sentence). So 'I asked that man who he was' is correct, because 'who are you' is the original question, and the 'who' is doing the 'are' verb.
However, in this instance, the 'who' (pronoun) isn't the subject of the question clause either, because if you took the question out of the sentence and made it direct (i.e. reconstruct what was actually asked), you get 'Who/Whom do you ask?' - since you are the subject, the person doing the asking, not the pronoun (who slightly complicates things here however because when we use it as an interrogative in a direct question word, we very rarely do the subject/object bit with it - even more rarely than outside of a question). Or to turn it third person to test it against the more familiar example - Do you ask him or her? not Do you ask he/she.
I recognise that this is probably way more detail than you wanted, but I hope it's vaguely interesting nonetheless, and makes sense!