The Iraq Data Debate: Civilian Casualties from 2006 to 2007

The Iraq Data Debate: Civilian Casualties from 2006 to 2007

Gen. David Petraeus’ assertions about falling casualties in Iraq are supported by a range of objective sources. But his testimony to Congress does not establish whether the decline is attributable to the surge or to sectarian cleansing.

Last updated September 28, 2007 8:00 am (EST)

Backgrounder
Current political and economic issues succinctly explained.

More on:

Iraq

Wars and Conflict

This publication is now archived.

Introduction

There is a growing debate over the data used to support claims of progress in Iraq. In particular, it has been widely asserted that Gen. David Petraeus and MNF-I (Multi-National Force-Iraq) have produced artificially optimistic data on civilian casualties. Petraeus has argued that civilian fatalities climbed over the course of 2006, but fell forty-five percent over the last eight months as surge brigades have arrived in Iraq. [1] These findings have been challenged on a variety of grounds. Some accept the observation that casualties have declined, but argue that much of this is due to sectarian cleansing rather than improved security: where the intended targets have already been driven out, violence becomes unnecessary but the neighborhood is no more secure for targeted minorities. [2] But others dispute the observation of decline itself. New York Times columnist Paul Krugman, Spencer Ackerman of the American Prospect, and John Podesta, Lawrence Korb, and Brian Katulis of the Center for American Progress, for example, argue that violence is not falling. National security expert Rand Beers argues that claims of a decrease cannot be verified by independent sources. [3]

Iraq data are inherently messy and all empirical claims need to be treated with caution. But two broad points seem clear nonetheless. First, sectarian cleansing is an important factor in Iraq ’s violence, but it is hard to know how important it has been relative to the surge in reducing civilian casualties. No claim for the relative importance of the surge and cleansing for Iraqi civilian casualties can be sustained from available data. Second, MNF-I is not alone in finding a reduction in civilian deaths since 2006. Multiple, independent sources find similar trends, and there is very little evidence to suggest any upward trend in violence in 2007. Given this, the Petraeus testimony is not inaccurate or uncorroborated in the way many have claimed. But neither is it complete: while the testimony does not explicitly attribute the casualty reduction to the surge as opposed to sectarian cleansing or other causes, its weight of emphasis implies a primary role for the surge. A more complete assessment would have addressed potential alternative causes explicitly, and would have clarified the limitations on what can be known about the surge’s effects.

Figure 1*

Iraq Casualties

Sectarian cleansing’s unknown contribution to casualties

There has clearly been a great deal of sectarian cleansing in Iraq, and cleansed neighborhoods are presumably harder for hostile militias to penetrate for murder or bombing attacks than mixed ones. Homogeneous neighborhoods may thus be safer than mixed ones (though imperfectly: Sadr City has been overwhelmingly Shiite since before 2003, but has been a frequent target of Sunni bombings all the same). But this does not necessarily mean that overall casualties will fall as neighborhoods within a city are cleansed.

The most violent districts are often those at the frontiers, where cleansing is actively underway. As neighborhoods are cleansed, these frontiers move on to the next block, but this merely moves the fighting; it does not end it. Eventually this process can indeed yield a cruel form of peace between sects within a given city, but only when that city as a whole becomes uniformly Shiite, Sunni, or Kurd (which Baghdad, for example, is still not). Even this need not end intra-sectarian violence, hence its impact on overall civilian casualties is unclear. And it is far from clear that cleansing even a particular city ends inter-sectarian violence nationally: Iraqi militias have shown a willingness to travel between cities and towns to expand their scope well beyond the borders of any one locality, hence a cleansed town may only lead to the opening of new violence in the next community until the process exhausts the country or is halted by security forces.

These moving frontiers of sectarian violence thus clearly change the geography of civilian deaths, but may or may not change their numbers very much. Hence the role and importance of cleansing for Iraqi civilian casualty trends are unclear. And in parallel with this cleansing effort has been the gradual arrival of the surge forces and their increasing use in direct population security—not to mention other effects, such as Muqtada al-Sadr’s decision to order his militia to stand down as the surge was announced in early 2007, or the decision by Sunni Sheiks in Anbar Province to stop fighting Americans and turn on al-Qaeda in Iraq instead (which has now radically reduced AQI’s lethality in Anbar). The relative causal role of these events cannot be parsed very accurately on the basis of publicly available information. Given this, about the most that can be said is that some unknown mix of improved security, voluntary decisions by key militias to stop fighting, a changing scope of internecine violence as cleansing frontiers have shifted, or still other factors collectively explain any change in civilian casualties. This leaves open the possibility that improved security was the chief cause—or that it was a smaller contributor than the others.

Petraeus’ statement did not discuss the potential role of sectariancleansing in Iraqi civilian casualty trends. [4] It should have. An explicit discussion would have provided a more comprehensive picture, and reduced the risk that the testimony could mislead its audience by overlooking competing explanations of the identified trends. But it should be noted that Petraeus did not actually claim that security alone explains reduced casualties. On the contrary, his statement consistently frames the surge as “one reason” for the observed decline in violence, and he consistently describes it as “helping reduce” the violence. This formulation clearly implies other contributors beyond the surge, and carefully avoids unique causal attribution that the data could not sustain. And a thirty-three percent increase in U.S. combat brigades designed to provide direct population security in Iraq surely had some effect on civilian casualties. Gen. Petraeus’ statement was thus incomplete, and potentially misleading in its emphasis, but not inaccurate as such. [5]

The pattern of declining casualties

As for the second point, most available independent data sources show a broad pattern similar to that asserted by MNF-I: civilian casualties increase until some point near the end of 2006; this trend changes in 2007 and casualties then decline. Figure 1 presents these data graphically. [6]

Unsurprisingly, the data are noisy. It is very difficult to collect complete, accurate figures on deaths in Iraq: reporting is haphazard; communications are difficult; many sources (such as hospital officials) are under threat by sectarian militia or insurgents and face pressure to manipulate data; Muslim burial customs encourage very rapid interment by immediate family, complicating casualty reporting; other bodies are discovered, if at all, only long after the killings took place, obscuring the date of the actual murders. No two sources thus agree completely, and the differences can be large—over two thousand five-hundred deaths separate the highest (3,389, according to icasualties.org) from the lowest (842 from AP) figure for September of 2006, for example.

But although there is disagreement on individual months’ totals, the trend in the data as a whole is quite similar across sources. All sources with data for 2006 show civilian casualties increasing for the year, often quite sharply (icasualties.org, for example, presents a five hundred-seventy percent increase for the year; AP reports a five hundred-ten percent rise), and all of these display a maximum value in late 2006 (in September for icasualties.org, for example; December for AP; November for UNAMI). All sources with data for 2007 and 2006 show a net decline in 2007 from the 2006 peak. The 2007 decline is sometimes steep (IBC, icasualties.org, or Washington Post), and sometimes shallow (AP or Reuters), but with the sole exception of McClatchy (which has no 2006 data), all others show some apparent decline for the year in 2007. Neither the increase nor the decrease is uniform: values fluctuate up and down from month to month as well as across sources. And most sources (all but AP and Reuters) show a double peak, with local crests in the vicinity of July-September 2006 and again somewhere between the following October and February. One, AP, shows a trend in the latter months of 2007 that is essentially flat. But the high value in all five series occurs somewhere between July and December 2006, with no value in 2007 equaling that source’s 2006 maximum, and with all displaying a downward slope for 2007 as a whole.

Nothing in these data suggests that MNF-I is an unreasonable outlier contradicted by all other sources. In eighteen of the twenty months presented, the MNF-I figure is between the high and low values presented by other sources. The MNF-I pattern of increasing casualties in 2006 followed by decreasing casualties in 2007 is consistent with the broad trends in the available data as a whole. MNF-I’s data show steeper slopes than some sources, but less steep than others: while the MNF-I rate of increase for 2006 is the sharpest of the available sources, its rate of decline for 2007 is exceeded by icasualties.org, IBC, the Washington Post, and Brookings. [7] On balance, MNF-I’s casualty totals for 2006 are slightly higher than the mean for the other data sources available (by about four percent), and slightly lower for 2007 (also by about four percent), but the differences are small. It is hard to sustain a claim that MNF-I is radically out of step with other, independent, data sources, or that those sources contradict MNF-I in any substantial way—if one considers the data as a whole, rather than focusing on selective subsets taken out of context.

The use and misuse of data

But if one is willing to take selective subsets out of context, then it is possible, in fact, to support any conclusion from these data. This is because the data are noisy and non-monotonic: values rise, then they fall, and both the rise and the fall are subject to apparently random fluctuations from month to month along the way. A misleading two-point comparison can thus easily be structured to imply that casualties are getting worse, not better, in 2007; or that the drop in 2007 is much greater than it is; or that there has been no change at all. If one chooses July and August of 2007, for example, then casualties go down for the MNF-I and Washington Post data, but they go up, during the surge period, for the AP, Reuters, and icasualties.org data (and all but the Washington Post data show a twenty to thirty-five percent one-month increase for June to July 2007, including MNF-I). But this is at least as likely to be an artifact of month-to-month instability as a sign of an underlying real reversal of security: much as the stock market bumps up and down even in the midst of long term trends to the contrary, so do these data, and the clear trend for 2007 as a whole is down.

Alternatively, many have compared 2007 data with figures for the same month from 2006; in each case, the 2007 figures are higher. But this hardly means that casualties are now increasing—on the contrary, the slope is now downward for all sources save the latter months of AP. The downward slope for 2007 is shallower than the upward slope for 2006, hence comparisons separated by twelve-month spans will show that the later values have not yet declined to the earlier levels. But if the current trend continues, they will. And there is nothing magic about twelve-month intervals; if one compares casualties at quarterly intervals, for example, the later values are generally lower than the earlier ones for 2007, not higher. For cyclic businesses in the civil economy, twelve-month comparisons are important because they compare like quantities (Christmas season sales this year and last year, for example). But there is no reason to suppose that Iraq is a seasonally cyclic business, hence there is no special significance to same-month comparisons per se.

None of this means that current strategy will necessarily succeed or that current policy is necessarily sound. These trends could flatten out or reverse in coming months. The up-tick in some sources for July or August 2007 could eventually prove to have been the beginning of a new and unfortunate long-term trend if they continue in this direction rather than bumping back downward again. And even if favorable casualty trends continue, at the current rate of decline they have a long way to go before they reach tolerable levels: even at the current rate of decline, it will take a year for casualties to fall even to five-hundred a month. One of us has argued that total withdrawal is a defensible option for Iraq, [8] and these data do not suffice to overturn such a finding; a much deeper analysis is required to sustain any particular position on the war. There is plenty of room for debate on US policy in Iraq . But not all assertions about casualty trends are defensible. And the frequent assertion that MNF-I’s data are unrepresentative is unsound.

*In Figure 1, Iraq Body Count (IBC) data for June 2006 was entered incorrectly as 1300. The correct value is 2352. Data shown for IBC, downloaded on September 14, 2007, are averages of the "high" and "low" values for each month.

Note on the authors

Stephen Biddle is Senior Fellow for Defense Policy at the Council on Foreign Relations and served as an advisor to General Petraeus in Baghdad in spring 2007. The views expressed here are his own. Jeffrey Friedman is Research Associate at CFR.

Endnotes

[1] General David Petraeus, Report to the Congress on the Situation in Iraq, Multi-National Force-Iraq, 10-11 September 2007, p. 3 and associated slides.

[2] See, e.g., Paul Krugman, “Time to Take a Stand,” New York Times, September 7, 2007; MoveOn.org, “General Petraeus or General Betray Us? Cooking the Books for the White House” (https://pol.moveon.org/petraeus.html, accessed September 17, 2007); Office of Speaker Nancy Pelosi, “Iraq Reporting Fact Check” (www.speaker.gov/blog?p=736); John Podesta, Lawrence Korb, and Brian Katulis, “Key Questions on the White House Report and Petraeus-Crocker Testimony,” September 7, 2007 (www.americanprogress.org/issues/2007/09/Iraq_questions.html); Nancy Youssef and Leila Fadel, “What Crocker and Petraeus Didn’t Say,” McClatchy News Service, September 11, 2007.

[3] See, e.g., Krugman, “Time to Take a Stand;” Spencer Ackerman, “Iraqi Civilian Casualties: 2007 More Deadly than 2006” (www.tpmmuckraker.com/archives/004116.php, accessed September 17, 2007); Podesta et al., “Key Questions;” Rand Beers, “Drop in Violence?” (www.nationalsecuritynetwork.org/node/194/print, accessed September 17, 2007).

[4] Gen. Petraeus and Amb. Ryan Crocker each discussed sectarianism per se extensively, and Gen. Petraeus in particular did treat the dynamics of sectarian cleansing on at least two occasions (in response to questions from Sen. Ben Nelson and Rep. Adam Smith he discussed the tendency of cleansing frontiers to continue moving outward to new neighborhoods until stopped by security forces, rather than halting of its own accord once the original communities had been purged). But neither witness explicitly addressed the potential causal link between cleansing and Iraqi civilian casualty trends. Several lawmakers raised this point, but only in the context of multiple-part questions with the focus on other issues; neither witness spoke to this part of the questions, nor did any of the lawmakers follow up on this issue, hence it went unaddressed. The role of cleansing in casualty causation has played a more prominent role in the subsequent debate over the testimony than it did in the hearings themselves.

[5] Nor do the data fail to provide a valid (if partial) test of Gen. Petraeus’ main argument regarding the security effects of the surge. One cannot prove that any security improvement is attributable to the surge. But these data could have disproved MNF-I’s claims if they had shown an increase in casualties rather than a decrease. The fact that the data failed to falsify the claim does not prove the claim, but if offers corroborative evidence that should increase our relative prior confidence in that claim’s validity. This type of test is common in empirical social science. Rarely can data analysis prove a positive claim; such “failure to falsify” is the most that can normally be accomplished. On the logic of increasing confidence by surviving tests that fall short of proof, see Gary King, Robert Keohane, and Sidney Verba, Designing Social Inquiry (Princeton: Princeton University Press, 1994), pp. 100-105. Strictly speaking, nothing in Gen. Petraeus’ testimony claims anything more than this for the data presented.

[6] The data are taken from: Petraeus, Report to Congress, attached slides; Iraq Body Count, an independent group of volunteers who collect cross-checked media reports of violent deaths in Iraq; icasualties.org, an independent organization which collects data on civilian casualties from media sources; Reuters (Alastair Macdonald, “Iraqi Data Suggests Civilian Deaths Still Rising,” Reuters, November 1, 2006; “Iraq Civilian Deaths Hit New High in January,” Reuters, February 1, 2007; “Iraq Civilian Deaths Down in April: Govt Figures,” Reuters, May 1, 2007; Mussab Al-Khairalla, “Civilian Death Toll in Iraq Spikes in May,” Reuters, June 2, 2007; Dean Yates, “Iraqi Civilian Deaths Rise,” Reuters, September 1, 2007), which obtained its data from Iraqi Ministry of the Interior sources; The Associated Press, via personal communication from Ms. Lynn Dombek, Research Director, AP, 14 September 2007, which tracks casualties on a monthly basis with data drawn from police, hospital officials, morgue workers and verifiable witness accounts (and which considers its findings to be likely underestimates); the UN Assistance Mission for Iraq (UNAMI, “Human Rights Report: 1 May–30 June 2006,” July 18, 2006; International Federation of Red Cross and Red Crescent Societies, “Operations Update: Iraq, Humanitarian Emergency,” June 14, 2006; UN Assistance Mission for Iraq, “Human Rights Report: 1 July – 31 August 2006,” September 20, 2006; UN Assistance Mission for Iraq, “Human Rights Report: 1 September-31 October 2006”; UN Assistance Mission for Iraq, “Human Rights Report: 1 November-31 December 2006”, January 16, 2007); Michael E. O’Hanlon and Jason H. Campbell, "Iraq Index: Tracking Variables of Reconstruction & Security in Post-Saddam Iraq," September 10, 2007 (www.brookings.edu/iraqindex), which presents authors’ estimates; Washingtonpost.com, "Civilian Safety in Iraq", accessed September 14, 2007 (http://www.washingtonpost.com/wp-dyn/content/graphic/2007/09/05/GR2007090500440.html), which cites unofficial data from the Iraqi Ministry of Health; and McClatchy News Service, via personal communication from Mr. Mark Seibel, Managing Editor for International News, Washington Bureau, McClatchy, and Ms. Leila Fadel, Baghdad Bureau Chief, McClatchy, 17 September 2007 (Leila Fadel, "Security in Iraq still Elusive," McClatchy Newspapers, 9 September 2007 cites unnamed “Iraqi government” sources for comparable data).

[7] Slopes were computed by OLS regression on the pertinent years’ data for each source.

[8] Stephen Biddle, “Evaluating Options for Partial Withdrawals from Iraq,” Testimony Before the House Armed Services Committee Subcommittee on Oversight and Investigations in Alternatives for Iraq, Hearings Before the Committee on Armed Services, U.S. House of Representatives, One Hundred Tenth Congress, First Session, July 25, 2007; idem, “Go Deep or Get Out,” Washington Post, July 11, 2007.

More on:

Iraq

Wars and Conflict

Close

Top Stories on CFR

Indonesia

Prabowo Subianto was named the winner of the Indonesian presidential election. But it is unclear which version of Prabowo—the more moderate candidate from the campaign trail or the self-styled strongman—will govern Indonesia.

Russia

The mass casualty theater attack in Moscow was a reminder that affiliates of the Islamic State have reorganized and infiltrated even powerful states.

India

With India's development continuing to gain steam, one of the biggest challenges will be to avoid the mistake that others have made when they failed to recognize their newly acquired global systemic influence and adapt accordingly. Both China and Big Tech show that it is never too early to start managing one's own rise.