'And even that's not universal, since in plenty indigenous cultures, having a penis didn't necessarily mean being a "man" at all.'
You seem to be very wed to gendered expectations.
I get that you're using the words man woman boy girl to mean sex role. I mean that's not what they've meant until recently but going with that.
You claim that some 'indigenous cultures' do not think having a penis means being a 'man'.
Given your definition of man is sex role enforced on males. Then if in a society the sex role for male is not how you see that sex role. It makes no odds. In that society the sex role for males (men) is the norm for that society. They are men. Just you understand sex roles to mean different things to what they do.
As an aside. Indigenous means originally from that land/ area. I don't know what you have in your head when you say 'indigenous' but it includes an awful lot of people.
EG in Europe
'Ethnic Europeans are indigenous to the region, having lived there for millennia. However, the UN recognizes very few indigenous populations of Europe, confined to the far north and far east of the continent.
Notable indigenous minority populations in Europe that are recognized by the UN include the Finno-Ugric Nenets, Samoyed, and Komi peoples of northern Russia; Circassians of southern Russia and the North Caucasus; Crimean Tatars of Crimea in Ukraine; Sámi peoples of northern Norway, Sweden, and Finland and northwestern Russia (in an area also referred to as Sápmi); Basques of Basque Country, Spain and southern France; and the Sorbian people of Germany and Poland.[85]'