I think this whole debate is grounded in a profound but deliberate confusion between empirical differences and empirical differences serving as signifiers of moral worth. From an empirical standpoint, people are different in their physical and physiological constitution, and thus it is likely that the differences in their psychomotor, affective or cognitive functions are affected by their heredity (i.e. genetic makeup) or interaction between the inherited and environmental factors. Denying that borders on obscurantism and philistinism.
But it is one thing to debate the factors affecting our psychomotor, affective or cognitive functions, and a very different thing to use some of these functions as signifiers or moral worth. The whole debate about IQ is so hotly contested not because of it significance for neuro- or cognitive science, but because "intelligence" has been defined as signifier or moral superiority. So properly speaking, the debate is purely moral and ideological and as such, has no rational solution. However, the contestants try to win by creating a red herring and changing the subject from the relation between signifier (difference) and the signified (moral superiority) to the ontological status of the signifier i.e. whether it is "objective" (based in biology) or "subjective" (i.e based in culture, values and perceptions.).
However, the whole debate is one big hoax which has neither any rational merits nor rational solutions. Even if there were biological determinants of cognitive functions i.e. men tending to have better math skills than women due to genetic/physiological factors, that difference per se does not imply any moral superiority. In the same vein, hair color is genetically determined, but it would be equally ridiculous to argue that, say, red-heads are morally superior to blondes.
The real issue is the notion of moral worth/superiority, not that of differences in our psychomotor or cognitive skills. An you cannot debate that rationally, but only by destroying the proponents of the point of view you find obnoxious - or better yet, stop debating it altogether.
Wojtek