I don't think this "biological substate that contrains X" rhetoric is very helpful. You can just as easily turn it around: "Is it so far-fetched to think that social conditions, whatever they may be, might constrain our linguistic capacity"? Whether we privilege the "biological" or the "environment", we're artificially valorizing one element and treating the other as the element contrained/controlled by the valorized element.
Any human characteristic, language included, is an emergent property of the complex interaction of biological and environmental factors. So--your question is not helpful, because the question assumes a dichotomy between the "physical" and the "environment" that does not exist.
Miles