By serendipity via link on MyDD, a Democratic Partyesque blog, http://www.chicagoboyz.net/archives/002543.html October 29, 2004 Bogus Lancet Study
Via The Command Post comes this study published in Lancet (free reg) which purports that 100,000 Iraqi have died from violence, most of it caused by Coalition air strikes, since the invasion of Iraq. Needless to say, this study will become an article of faith in certain circles but the study is obviously bogus on its face.
First, even without reading the study, alarm bells should go off. The study purports to show civilian casualties 5 to 6 times higher than any other reputable source. Most other sources put total combined civilian and military deaths from all causes at between 15,000 to 20,000. The Lancet study is a degree of magnitude higher. Why the difference?
Moreover, just rough calculations should call the figure into doubt. 100,000 deaths over roughly a year and a half equates to 183 deaths per day. Seen anything like that on the news? With that many people dying from air strikes every day we would expect to have at least one or two incidents where several hundred or even thousands of people died. Heard of anything like that? In fact, heard of any air strikes at all where more than a couple of dozen people died total?
Where did this suspicious number come from? Bad methodology.
>From the summary:
Mistake One:
"A cluster sample survey was undertaken throughout Iraq during September, 2004"
It is bad practice to use a cluster sample for a distribution known to be highly asymmetrical. Since all sources agree that violence in Iraq is highly geographically concentrated, this means a cluster sample has a very high chance of exaggerating the number of deaths. If one or two of your clusters just happen to fall in a contended area it will skew everything. In fact, the study inadvertently suggests that this happened when it points out later that:
"Violent deaths were widespread, reported in 15 of 33 clusters..."
In fact, this suggest that violent deaths were not "widespread" as 18 of the 33 clusters reported zero deaths. if 54% of the clusters had no deaths then all the other deaths occurred in 46% of the clusters. If the deaths in those clusters followed a standard distribution most of the deaths would have occurred in less than 15% of the total clusters.
And bingo we see that:
"Two-thirds of all violent deaths were reported in one cluster in the city of Falluja"
(They also used a secondary grouping system (page 2, paragraph 3) that would cause further skewing.)
Mistake Two:
"33 clusters of 30 households each were interviewed about household composition, births, and deaths since January, 2002."
Self-reporting in third-world countries is notoriously unreliable. In the guts of the paper (page 3, paragraph 2) they say they tried to get death certificates for at least two deaths for each cluster but they never say how many of the deaths, if any, they actually verified. It is probable that many of the deaths, especially the oddly high number of a deaths of children by violence, never actually occurred.
So we have a sampling method that fails for diverse distributions, at least one tremendously skewed cluster and unverified reports of deaths.
Looking at the raw data they provide doesn't inspire any confidence whatsoever. Table 2 (page 4) shows the actual number of deaths reported. The study recorded 142 post-invasion deaths total with with 73 (51%) due to violence. Of those 73 deaths from violence, 52 occurred in Falluja. That means that all the other 21 deaths occurred in one of the 14 clusters where somebody died, or 1.5 deaths per cluster. Given what we know of the actual combat I am betting that most of the deaths occurred in three or four clusters and the rest had 1 death each. Given the low numbers of samples, one or two fabricated reports of deaths could seriously warp the entire study.
At the very end of the paper (page 7, paragraph 1) they concede that:
"We suspect that a random sample of 33 Iraqi locations is likely to encounter one or a couple of particularly devastated areas. Nonetheless, since 52 of 73 (71%) violent deaths and 53 of 142 (37%) deaths during the conflict occurred in one cluster, it is possible that by extraordinary chance, the survey mortality estimate has been skewed upward. "
Gee, you think? It's almost as if military violence is not randomly distributed across the population of Iraq but is instead intelligently directed at specific areas, rendering a statistical extrapolation of deaths totally useless.
In the next paragraph they admit:
"Removing half the increase in infant deaths and the Falluja data still produces a 37% increase in estimated mortality."
That puts their final numbers just above the high end of the range reported by other sources.
This "peer reviewed study" is a piece of polemical garbage. Everybody is supposed to take away the bumper sticker summary, "Coalition kills 100,000 Iraqi civilians, half of them children," without reading the details. It tries to use crude epidemiological models like those used to study disease and applies them to the conscious infliction of violence by human beings. The result is statistical static.
(Update: Commentator Clashman below points out that the studies "conservative" estimate is actually around 66,000 instead of the 30,000 I had done in my head so the study is actually at least twice of what other sources place as the upper range at around 25,000)
(Update: Further related thoughts in )
(Update: I have posted more thoughts on evaluating the methodology of the study) Posted by Shannon Love on October 29, 2004 05:14 PM | TrackBack