Not quite...
Some men have a non-working Y chromosome. For example, Swyer Syndrome (uterus, no functional gonads, assumed to be female at birth usually, taller than a typical woman).
Some people have a Y chromosome and testes but no androgen effects (CAIS). They are also usually assumed to be female at birth.
Whether these individuals should be included in women's sports is tricky, I feel. I would be inclined to say yes, as they don't have any effect of testosterone. Again, we'd probably get an over-representation of them in elite sports as they usually are taller than average.