Countries like India are heavy users of diary products. Lactose intolerance is rare too. But always fare very poorly in the height rankings. So it cannot be concluded that it is diary products alone.
milk(calcium) just enables for bigger stronger bones, thus it provides that advantage when there is a pressure for it. The selective pressure for bigger body is present in cold climate and not present in India and even more - due to poverty one can see a pressure for smaller body there. Milk in India is used for another reason - it is a source of cheap (compare to meat) animal protein.
