There is no scientific basis for what I'm writing, but just from my experiences. I'm thinking that it's a combination of genetics, culture and lifestyle.
I'm 100% Japanese, but I was born and raised in America. Meanwhile, my cousins and extended kin have lived in Japan their entire lives. Physically, my siblings (that were born & raised in America) and I are very different from our cousins. I'm built like a high school wrestler or a running back (5' 8", 160 lbs), and my younger brother did wrestling in high school. One of our cousins is taller than us, but he has the typical Japanese male build of being really skinny, and we outweigh him by at least 20 pounds (of muscle).
Here in America, people can tell that I work out, but since there are a lot of people bigger than me, it's not as noticeable. But when I go to Japan, I stand out physically.
Bodybuilding and weightlifting aren't as big in Japan as they are in America. Though those particular activities aren't mainstream, Japanese people are plenty athletic. A lot of them participate in high school sports, and in Japan, it's year round for each sport. At the moment, soccer is the craze in Japan. And soccer players aren't known for being big.
During one of my tenures in Japan, I did meet one Japanese guy who was physically bigger and stronger than me, and he did Judo.
Those are just some experiences.
K.C.