Generally I accept that blogs are not about big 'I ams' but should be about debate.
I hope I can contribute to this debate and so it is probably worth saying that I am also a qualified epidemiologist who studied at the London School of Hygiene and Tropical Medicine (considered with Johns Hopkins to have experience and expertise in healthcare among conflict and displaced populations) and worked at the WHO for six years.
I approached this study with an open mind. The study should not be dismissed out of hand the authors are well regarded professionals that have produced a study that in the absence of reliable figures produced by an independent body (which they requested to be established after the 2004 report) attempts to estimate the increased mortality, especially the numbers of violent deaths subsequent to the invasion of Iraq.
I think it likely that the trends identified are correct (and agree with other assessments) the numbers however I consider to be likely to be highly overestimated.
On page one they refer to their earlier 2004 report that used a similar cluster sample survey to the current study. At that time they estimated;
-'98,000 excess deaths (95% CI 8000-194000)'
The figures in brackets are called Confidence Intervals and they express the percentage accuracy or confidence that the true figure lies within the range given i.e. they are 95% confidence that the true figure lies somewhere between 8000 and 194000. This is a huge range and to a reader of a study should be the first clue as to the probable accuracy. The other way of putting this is to say that there is a five% chance that the true figure is below or above the estimated range.
95% is the lower figure that is considered acceptable usually a 99% CI is striven for. With some 'weaker' techniques such a CI is not possible. The other thing to note is the huge range of where the true figure could actually be. The narrower that range of estimation the more reliable is the estimate. Here the upper point of the range at 95% CI is 24 times that of the lower point.
On seeing such a CI a critical reading of the study would want to know a lot more about the methodology and possible sources of bias and confounding effects as well as being wary of the kind of conclusions that could be drawn from such a study.
Criticisms of the study are of two kinds; methodological and ideological.
Methodology.
-The study quotes other reports used to count violent deaths;
-The Iraq Body Count which estimated that up to Sep 26 2006 between 43,491 and 48,283 Iraqis have been killed since the invasion. -Estimates from the Iraqi Ministry of the Interior were 75% higher than those based on the Iraq Body Count from the same period.
[Now this is a strange way to present these figures 75% higher means the original no. + Three quarters of that no. i.e. the figure from the Iraqi ministry is 76109-84,495 estimated killed since the invasion]
-An Iraqi non-governmental organisation, Iraqiyun, estimated 128 000 deaths from The time of the invasion until July, 2005, by use of various sources, including household interviews. [NB the research technique in this study uses household interviews but comes out with a much higher estimate]
-The US Department of Defence keeps some records of Iraqi deaths, despite initially denying that they did.-
Recently, Iraqi casualty data from the Multi-National Corps-Iraq (MNC-I) Significant Activities database were released. These data estimated the civilian causality rate at 117 deaths per day between May, 2005, and June, 2006, on the basis of deaths that occurred in events to which the coalition responded.
[Again this is a strange way to present the data which if added up over the calendar year which it apparently records (365 days) equals 42,705 deaths in events that the coalition responded to}.
Now there is a perfectly valid discussion by the researchers about the way in which 'passive' (hospitals, morgues etc) surveillance techniques typically produce lower estimates than 'active' techniques (going out and counting). But the conclusion of the report argues for the credibility of its findings by;
'Claiming similar patterns of mortality over time documented in our survey and by other sources corroborate our findings about the trends in mortality '
Well yes the trend is up obviously but the big difference between the Johns Hopkins study and all others is the numbers.
There are three 'passive' measures of numbers that they refer to, while presenting the data in such a way that one cannot immediately see that the figures from the Iraq Body Count, the Ministry of the Interior and the MNC I all give estimates between 42,000 and 84,500.
Even the Non Governmental group's multi-source and 'active' approach gives a figure that is about one fifth of their total (128,000 up to July 2005).
What can explain the discrepancy in these figures? Why does the Johns Hopkins study produce figures far higher than any other measure?
The answer is primarily methodological. The biggest source of error is the estimation of the pre-invasion numbers of deaths.
They realise this is a potentially fatal flaw in the conclusion;
-'The striking similarity between the 2004 and 2006 estimates of pre-war mortality diminishes concerns about people's ability to recall deaths accurately over a 4-year period'
This is contestable. The same method produces the same estimate based on personal recall of events before a threshold event. Human memory and recall is notoriously unreliable and is subject to mental awareness of key events the JFK assassination phenomenon, everyone recalls what happened that day although an ordinary day they can't recall.
Basically all of the possible biases that they must discuss for academic credibility on page 7 are likely to have occurred. It is likely that the sampling was not really random, that it sampled more heavily areas prone to violence (thereby overestimating the national rate) and the method of recall in interviews was flawed.
-'further calling back to households not available on the initial visit was felt to be too dangerous (page 7)'
The clustering technique they used in order to have statistical power and to be free from bias requires that a 'random' sample survey 100% of households in that cluster. For understandable reasons this did not happen but this is a serious source of likely bias.
-'not sampling two of the Governates could have underestimated the total number of deaths, although these were areas were generally known as low violence areas' (page 7).
This is the most disturbing of statements as it shows that the 'ideological' bias of the researchers is to avoid underestimation of deaths. They tell us that excluding these areas does not affect the findings as 'they were known as low violence areas'
Well this shows that we have an overrepresentation of areas known for high violence. 12 clusters were taken from the highest violence areas of Baghdad 4-12 times the number of clusters taken for other areas. Urban areas for safety reasons were sampled more than rural or hard to reach areas yet we know that more violent attacks take place in urban areas..
Their method of clustering they would say is based upon population size, but it does not avoid the central problem of violent death in Iraq, it is not random and evenly distributed throughout the population. It is specifically targeted in some areas and regions.
To exclude lower violence areas and include more high violence areas and then to project a national population figure based upon this must as a logical result of the study design overestimate the true rate. In fact this study design must systematically overestimate the numbers of those dying violently.
Evidence of editorial or researcher bias is not hard to find either;
-'Nevertheless, surveillance tallies are important in monitoring trends over time and in the provision of individual data, and these data track closely with our own findings
Mortality rates from violent causes have increased every year post-invasion. By mid-year 2006, 91 violent deaths had occurred in 6 months, compared with 27 post-invasion in 2003 and 77 in 2004, and 105 for 2005, suggesting that violence has escalated substantially.
The attributed cause of these deaths has also changed with time. Our data show that gunfire is the major cause of death in Iraq, accounting for about half of all violent deaths. –'
Deaths from air strikes were less commonly reported in 2006 than in 2003–04, but deaths from car explosions have increased since late 2005.'-
NB any reasonable observer would consider that this fits with an increase in sectarian and jihadi violence but the gloss put on this is extraordinary;
-'The proportion of violent deaths attributed to coalition forces might have peaked in 2004; however, the actual number of Iraqi deaths attributed to coalition forces increased steadily through 2005.'
In other words even though the proportion of deaths attributed to coalition forces is going down and therefore the number of deaths that were not coalition attributed is going up the data is presented as the Coalition killing more people.
-'Deaths were not classified as being due to coalition forces if households had any uncertainty about the responsible party; consequently, the number of deaths and the proportion of violent deaths attributable to coalition forces could be conservative estimates.
Distinguishing criminal murders from anti-coalition force actions was not possible in this survey'-
So we are told that the no of air-strike reported deaths is down the number of car bombs is up and now half of all violent deaths are caused by gunfire there is no way of distinguishing 'criminal' (whatever that means) from anti-coalition force actions (i.e. how do they define anti-coalition force actions if they can't distinguish types of violent death).
Claiming in these circumstances that the number of violent deaths attributable to coalition forces 'could be conservative' estimates is seriously misleading. There is no attempt at an even handed approach to the likely bias in respondent attribution. There is no 'could be a conservative or exaggerated estimate'.
One doesn't have to be an expert in questionnaire design to see that, given the context (sectarian violence, political motivation, denial etc) to consider that respondents may overestimate coalition responsibility at the expense of other causes.
Yet we have fairly robust indicators of other causes that the authors gloss over or misrepresent.
-'Across Iraq, deaths and injuries from violent causes were concentrated in adolescent to middle age men. Although some were probably combatants, a number of factors would expose this group to more risk—e.g., life style, automobile travel, and employment outside the home'.
This is really discreditable 'some were probably combatants'. They have simply refused to even attempt to enumerate the number of Iraqi combatants despite the data being skewed to wards military age young men and instead lifestyle reasons might better explain the data (what smoking crack?)
-'The circumstances of a number of deaths from gunshots suggest assassination or executions. Coalition forces have been reported as targeting all men of military age.-'
This is perhaps the most egregious assertion despite the fact that their own data is screaming increase in sectarian and Jihadi violence they basically imply that the coalition is now engaged in targeted assassinations of young men.
For the record I was opposed to the Iraq war not least because I have worked in healthcare among conflict and displaced people and know that any social destabilisation on this scale leads to a big increase in civilian deaths.
Having worked at the UN I considered that an invasion without UN auspices would mean that the basic post invasion measures needed for Public Health would not be put in place (that expertise lies with the UN specialist agencies that are not political or military in mission but humanitarian WHO, UNDP, UNICEFF.
I think that there is a humanitarian disaster continuing in Iraq but this paper is ultimately flawed and worse it is actually misleading if not downright dishonest Posted by mettaculture at October 12, 2006 01:48 PM