I wouldn't say that such differences are inevitable, but when you do have a difference, and assuming similar distributions of scores for both men and women, the difference will be much more pronounced at the extreme end(s). This is true for height, and let's hypothesize that innate mathematical ability has a strong genetic component which could be similarly distributed.
If this hypothetical is true (to which there is some debate), you are going to have more male math profs (or specifically, more math researchers) merely because there will be more men than women at the extreme positive end of mathematical ability and therefore more men than women in positions that require such mathematical ability.
However, I'm not sure if I recall this 100% correctly, but women are also less likely to get math/physics PhDs even when high-school testing scores are controlled for. That's something else entirely and perhaps more important.
The question might be: what are we shooting for? Unfortunately, 99% of the criticism seems to be on an individual or anecdotal basis ("I like to work long hours, I succeeded in my profession" - no kidding, and you're here now, so obviously you got there) rather than a population basis, which is where decisions about policy take effect.
I'd like to think that Summers was pointing out that 50-50 parity isn't achievable (and perhaps not desirable), but some of the difference can be attributed to genetic variation, some can be attributed to choice (and therefore social learning), and some can be attributed to discrimination. We can't correct the first, we can make some inroads against the second by making the choice easier, although it'll always be tough (and will be tough no matter who makes it), and we should certainly fix the third inasmuch as we possibly can.
I'm not sure if that was Summers' point. But I'd like to hope.
Cheers,
Marco