This is a rather simplistic view, Henry. Skin color or facial features are one many possible 'social status cues' (as they are sometimes called in sociology) - but there are many other possible, such as speech pattern, body build, etc.
The bottom line is that "race" is a social construct - to use your example, an average Native American would probably make no distinction between a German and a Pole or a Jew - yet this was a life-and-death distinction in Eastern Europe not so long ago, between being the master-race and "tiermenschen" suitable only to be made into soap or lampshades. In the same vein, most Anglos in the US can't tell the difference between a Japanese, an a Korean or a Chinese - yet in Japan or in China ca 1940 that diffrence was again the matter of life and death.
Methinks, the biggest pitfall of the "race debates" is a certain form of "essentialization" of race - that is, a belief that it is something about physical characteristics of a group of poeple that is responsible for their social status. In that aspect, the racists and certain type of anti-racists share the same view (i.e. that the so-called "coloreds' are inferior because of their physical characteristics) - they differ only in the moral attitude toward that view. The racists see it as morally justified, the anti-racists in question, as moral abomination.
I think this stale imagery confuses the issue of social inequality instead of explaining it. The bottom line is that social order is envisioned as a hierarchy of perfection - a view enshrined in our knowledge structures, educationsl institutions, religion, etc. - and real people are fit into this apriori social order. The social/intellectual/physical characteristics that are used to justify the assignement of people to that hierarchical order are mere rationalizations not causes or reasons.
That is, as long as hierarchical order is seen as the epitome of rationality and perfection, any human characteristic can be used to assign people to diffrent levels in that hierarchical order. It could be skin color, gender, the shape of the nose, body build, accent, manners, size of feet, shape of ears or necks - anything can be a social status cue. Of course, the association between particular types of cues and particular social status becomes quckly entrenched in any particula culture, and treated as self-evident by most people. But that is also true of any 'stock knowledge."
The task of critical social science (as opposed to providing the needed conclusions to those in the position to pay for them -- practiced in econ and poli-sci departments) is to "deconstruct" (sorry for that dreadful term) those elements of stock knowledge, that is, to demistfy their supposedly "natural" character and show how they were produced by a particular set of social-historical cicrumstances to suit the needs of particular interest groups.
wojtek