I am not Japanese, but I lived there for 9 years, my husband is Japanese and my DD was born there.
As you rightly say, there is (traditionally) no dairy in the Japanese diet.
They got their calcium intake mainly from eating seaweed and small fish which were eaten including their bones/shells etc.
This didn't give a great deal of calcium overall and is one of the main reasons for the Japanese people being so small on average.
In recent years milk, yogurt and cheese consumption has increased dramatically. It is no coincidence that the height of the Japanese people has also grown just as dramatically.
In the UK, I am average to small. (5 foot 4)
Compared to DH's grandparents' generation, I am huge and tower over both men and women. A very large percentage of women in that generation are bent over with very severe osteoporosis.
In comparison to DH's parents' generation, I am large. I am taller than both his parents. His mother is about 5 foot and his father is about my height.
In comparison to our generation, I am taller than average. DH is 6 foot and is the tallest of all his friends. (He drank a lot of milk as a child) I am taller than most, but not all of my female friends.
In comparison to current high school students I am small. They are dramatically taller and much closer to modern western sizes.
This is not all due solely to dairy intake, but it can't be ignored. The diet has changed in many ways in recent years, but with regards to bone strength and growth, it must be a major factor.
In Japan, children who are allergic to dairy are given soya products, as is common here. Many of them have added calcium, so it doesn't mean that children have to miss out too much.