Perhaps because prior to the mid-seventies the academia was not marketized to the degree it is now - academic careers was not so heavily dependent on popularity, market success of intellectual commodity and fundraising capacity. Today, many if not most job postings in the academia list "fundraising" as a necessary qualification together with proper academic credentials and research experience.
I would go as far as saying that the funding source of the research determines the clarity of academic writing. The corporate-, government- or foundation- sponsored research tends to produce clear academic writing, whereas those researchers who support themselves by selling their intellectual commodity in the market tend to be more obscure. It is so, because in sponsored research the producer has already been paid in the form of a grant, so market success is not necessary. However, the sponsoring agency often makes specific requirement about dissemination results of the research - these are usually written for very specific audiences, executives, police makers, program officers, practitioners, who have little time and tolerance for bullshit.
On the other hand, if your funding and career depends on the market success of your intellectual product, you target audience are usually the literati, because they are the ones who buy and read books, and because executives, policy makers and practitioners read executive summaries produced by reputable (in their view) sources, rather than verbose academic prose written by those whose reputation is, at best, unknown. Therefore, verbosity and obscurity is a way of pleasing the taste for the exotic in the audience, and also ascertains that the ideas produces that way are sold rather than disseminated as a "public good."
Wojtek