Sure, but I would say that the level of that constraint depends on how hard-coded the capability is, or how mechanical it is. A computer is about as materialist as you can get ;-), but it can run any algorithm (setting aside issues of complexity, which are not relevant here, but do demonstrate what sort of constraints are applicable in such a general purpose computing device).
Assuming by "people who do language" you mean "people who speak or use" (as opposed to people who study language), it seems to me, there are different sorts of [physical] constraints that it might be helpful to differentiate between: e.g: constraints of our vocal system which makes it impossible to produce certain sounds. I think I can state safely that that's not your concern here (spoken language), but rather, in explicating Chomsky, you are pointing to the structure of language and its acquisition.
I want to revisit something I wrote earlier:
Chomsky offers "poverty of input" arguments, and others (the empirical types dear to Miles ;-)) point out the startling applicability of the UG to all languages (AFAIK -- every now and then some Deepak Chopra type person offers that the Gojambi, a remote tribe in the impenetrable forests of Ecuador, have no word for "I" and do not use this or that sentence construction at all; but most such claims seem to be either exaggerated or irrelevant to the UG thesis). Again AFAIK, the UG stands unrefuted. Further, the base work done by Chomsky w.r.t syntax shall always remain foundational, given its value to such fields as Computer Science.
An aside to an aside: what Chomsky seems to have brought to linguistics seems to be in the same spirit as other [positivist?] developments in the early 20th century: an intentional narrowing of the field of examination in order to be able to say concrete things. Contrast continental linguistic turn that preceded Chomsky (this btw, is why I find uncomfortable comparison of Chomsky to Freud, since they seem, in their methodology, quite the opposite).
Back to the matter: (here I am thinking and recounting aloud, not lecturing, so please humour me!) if human languages exhibit a startling similarity of structure, then there must be a reason for that -- something in human beings which forces language development into a particular path. To this reasoning Chomsky adds his belief in the poverty of input to shape the development or acquisition of language in a child, and he finds it inescapable that we have some prebuilt wiring for language.
Child behavioural psychologists and cognitive scientists seem to not buy into this. And I too find the "poverty of input" argument too close to Intelligent Design proponents for comfort. And neat experiments and theories have disproved various Pinker'ish claims about how something just had to be inborn (I don't have the relevant books/papers here at work, so will post info tonight). Similar claims are made about other human capacities, typically calling upon twin studies (which theories fail to notice that even separated twins share a developmental history: in the womb -- more on that on my blog if you care to search). This does not imply, as in Pinker's caricature, that his opponents believe the mind to be a blank slate. But on another front, Pinker and Co, understandably run with Chomsky's idea, taking it to a place he is unwilling to go: the language faculty as an adaptation.
There is one other idea that favours Chomsky's innate language "organ" notion, and that is Jerry Fodor's notion of modularity. If the mind is implemented, among other details, as a series of specialised modules, then that is one other reason to believe that one such module can be language processing and acquisition, with its own inbuilt structure that parallels, or if you prefer limits, the structure of language.
But Fodor himself offers a very neat idea (IMHO), quite in line with your (M.Smith's) idea of structural constraints, of why a UG might hold in the mental language faculty without having to be implemented as a physical mechanism. To wit: the CTM. Fodor points out that our mechanisms of thought (or belief system), in general, are surprisingly truth-preserving. Empiricists believe that is so because our belief system is honed by repeated encounters with reality. Fodor seems to think not: rather, our mind's actions are truth preserving like a computer, because these actions are the equivalent of computation. It seems to me, the same can be true of language, without the modularity.
--ravi