I'm getting queries about various things in the article, especially the fact that there is not very much about the statistics. I know you're disappointed. I too was disappointed when my editor informed me that there really wasn't room for 2,000 words on the fascinating topic of survey interview technique and small sample confidence intervals. Unfortunately, there's only so much space in the magazine; I had fifteen hundred words, and statistical arguments, dry and complicated, are extremely hard to fit into a small space.
There are also various odd quibbles about usage or turns of phrase. I'm not writing for academics; I'm writing for a general audience. The fact that something is obvious to you does not mean that it is obvious to laymen, and unfortunately, that + space constraints mean journalists have to simplify a lot, especially technical questions. This is, believe it or not, as unsatisfying to us as it is to professionals in various fields. But we don't have the space to write a textbook, and unlike professors, we can't flunk our readers if they drop out because they're bored and confused.
Luckily, I have this blog, and you, dear readers, to bombard with the accumulated dross of hours of interviews and a multi-foot stack of printouts. I'm heading to Puerto Rico on Sunday for vaction, leaving you in the hands of a crack team of guest bloggers who will make you sad to think that I'm coming back. Before then, I'll lay out as much as possible of the various arguments about the studies, the problems and advantages of cluster sampling, the specific issues in Iraq, and the more theatrical public controversies. The rest, I'll finish up when I get back, or third degree sunburn, whichever comes first.
Unfortunately, this takes a while: the whole thing is complicated, and thankfully, very few people have spent as much time absorbed in the question as I did while writing this story.
What I hope to do is the one thing that, as far as I know, no one has done so far: lay out a moderately detailed explanation of the history and issues of counting the Iraq war dead. I want it to be accessible to people who have no, or only a passing familiarity, with conflict epidemiology and the specific problems in Iraq, or even basic statistics. If you're already familiar with what I'm writing (or don't care about conflict epidemiology, which is somewhat understandable), just skip those bits. The blogging about Burnham et. al., and to a lesser extent some of the other counts, has been . . . well, let's just say, extremely passionate. However, as far as I know (and I think I managed to find all of the major blogs that were writing on the topic, as well as any significant article written in an English language publication), no one has laid out the entire subject in any sort of orderly fashion; it's mostly just critics responding to critics responding to critics . . . by which point anyone who isn't completely obsessed with survey technique has lost any understanding of, or interest in, what's going on. Print can't do what I can, which is utilize essentially unlimited space, hyperlinks, and reader feedback on what's unclear. I make no guarantee that at the end of it, I'll have made the thing any plainer . . . but darn it, I'm sure going to give it that old Hoover High Try.
Meanwhile, I'm afraid I'm going to ignore the various insistent questions about whether I thought of . . . whatever. So far, no one's asked me something that I (or really, all the people I interviewed) haven't asked and tried to answer. In other words, I'm getting there. If you do hit on something new, I'll look for an answer, throw it out to my sources, or say "Here's a good question". But if I start trying to answer all these queries out of order, we'll get bogged down in a labyrinthine set of disjointed explanations.
I have to challenge your first question, "How many Iraqis have died because of the American invasion?"
The deaths in Iraq since 2003 aren't "because of the American invasion." Most of the "excess deaths" since 2003 have not been at the hands of Americans. The fascists and fanatics caused most of those deaths, and they continue to inflict the largest part of Iraq's bloodshed.
The Nazis killed most of their death-camp victims after the U.S. entered World War II. Would you ask then, "How many Jews and Gypsies died because the Americans entered World War II?"
The same man who supervised the collection of data on the Johns Hopkins-Lancet war-death study also supervised the earlier study that found, before the 2003 Iraq invasion, that as many as 10,000 Iraqis were dying each month because of the sanctions Saddam had brought on his nation, and because Saddam was diverting Oil-for-Food cash to palaces and payoffs.
That's the study Cindy Sheehan was referring to when she said, in Ireland, that President Clinton was responsible for more Iraqi deaths (more than 1 million dead, from sickness and starvation) than President Bush. Ramsey Clark cited the same study when he argued that sanctions should be ended and Saddam left alone.
My point: How about subtracting the estimated 120,000 a year who were dying during Saddam's reign?
... guest bloggers who will make you sad to think that I'm coming back.
You mean they could pass a Turing test?
I, for one, look forward to seeing you come back and answer a few questions . Have fun on Spring Break!
"The fact that something is obvious to you does not mean that it is obvious to laymen, and unfortunately, that + space constraints mean journalists have to simplify a lot, especially technical questions. This is, believe it or not, as unsatisfying to us as it is to professionals in various fields. But we don't have the space to write a textbook,..."-MM
we're on the web, right?
Pass a note to Ed. : That's what Endnotes are for.
http://www.talkjustice.com/links.asp?453054167
Meghan, I love your blog, but I'm a little worried about your new article (which I haven't read yet) about how all these scientists are wrong about how many Iraqis died as a result of our invasion.
Obviously, you think the Lancet study is wrong, you've posted about that many, many times. But methinks you may have awakened a sleeping giant. I really don't think the authors of those peer-reviewed and published studies are going to kowtow to you and admit they were wrong and you (who have no training or expertise in the subject) are right, without a strenuous (maybe even a name-calling) fight.
But best wishes in any regard. Love you. Have fun in PR.
Meaghan,
As the longest ongoing blogger to your posts, I must say, "tootle-loo." Unfortunately, my doctors have concluded that I am too ill--both physically and mentally--to post to your blog anymore. It was my favorite hobby--next to reading Libertarian websites.
Now, what to do, what to do? Do you or your readers have any ideas?
From now on if any one posts under my name, you must remember that it isn't me--it's an impostor. I am the victim of identity theft. I beg of all your readers, do not listen to me anymore.
I'm actually having no trouble at all posting. Maybe the Atlantic's servers are based in Vietnam to save $.
Anyway, I did appreciate the article's notes about the bias effects of initial exposure to a certain number. I think the bias effect of the press's predilections for very large numbers is even stronger than you suggest: no one will fund an NGO working on a minor catastrophe, and no one will read an article about a small but unfortunate event, so there's a strong incentive to ensure that refugee numbers hit at least 1 million, etc. It would be nice if every number came with a really good confidence interval attached; the numbers on the dead in eastern Congo seem particularly sketchy. The really frustrating missing data points in the Vietnam War are the number of Vietnamese soldiers killed; the number of Vietnamese civilians killed; and the number of people interned in reeducation camps after the war, and how many of those died. Between the competing propaganda interests of the organizations that did the counting, there's no way anyone will ever arrive at good numbers on any of those.
Lady Lucy, thanks for your concern, but the piece was researched and fact checked pretty carefully for precisely that reason, and I will be very surprised if Burnham et al dispute any of my factual claims, unless they choose to quibble over whether the UN study found three or four times fewer deaths than their original survey. The position I staked out is not on the side of their fiercest critics, nor of the study's supporters, but the middle ground occupied by other researchers, including from other Iraq studies and from someone who has seen both the Burnham and WHO datasets. If they attack, I'm pretty prepared to contest their claims--especially since they declined to make themselves available for interviews after repeated requests.
Brooksfoe, you anticipate one of my talking points! Actually, it's already in the article, but I'm going to mention it at greater length: casualty figures suck, and yes, the Congo is one of the worst offenders, followed by other areas in Africa--I expect any day to hear that one billion Darfuris have been killed in Sudan. I'm not sure who's worse: the well meaning NGOs who think 1 million dead Africans=1 dead American or the journalists who take the absolute highest figure they can find even if it's the serial number on the back of their DVD players.
Anonymouse, I sure hope you aren't serious. Because if you are . . . Anonymous, where shall I go--what shall I do? We'd be lost without you.
Email me if you are serious, so that I can at least express my deep regret in person. And if you aren't . . . well, you oughtn't to tease an old lady like that.
Look, people, how hard is it to properly spell "Megan?" Especially when it's written in large letters at the top of the page?
I mean, seriously - I am no fan of the proprietress of this here blog (and that is a huge understatement), but if you guys love her so much, would it be too difficult to remember that there's no 'h' and only one 'a' in her first name?
"I too was disappointed when my editor informed me that there really wasn't room for 2,000 words on the fascinating topic of survey interview technique and small sample confidence intervals."
Did you try screaming, "Philistines! You're strangling my muse!" at him?
[Lady Lucy, thanks for your concern, but the piece was researched and fact checked pretty carefully for precisely that reason, and I will be very surprised if Burnham et al dispute any of my factual claims, unless they choose to quibble over whether the UN study found three or four times fewer deaths than their original survey]
Strange that you refer to Burnham et al (2006) as "the earliest survey" without mentioning Roberts et al (2004) then, particularly as Roberts et al (2004) found a very similar rate of violent, nonviolent and total excess death to that in the IFHS survey.
Njorl: I also staged a hunger strike. I was undone by a two-for-one sale on the new Deep India gujarati-style Dal entrees.
Dsquared: Burnham et al is shorthand for the Hopkins team who did the study, not a reference to either of the Lancet studies. Gil Burnham is senior researcher upon whose reputation both studies hang in the public health community now, since as you know Richard Garfield of Columbia declined to participate in the second effort, and has made public statements disagreeing with its results. I'm sorry if the context didn't make it clear, but I thought it obvious that the actual article could not file a dispute with my editor.
I've blogged a fair amount on the topic here:
http://willscommonplacebook.blogspot.com/search/label/Iraq
[as you know Richard Garfield of Columbia declined to participate in the second effort, and has made public statements disagreeing with its results.]
as I don't know, because it isn't true.
And the claim that "Burnham et al is shorthand for the Hopkins team who did the study, not a reference to either of the Lancet studies" is completely tangential. You in fact refer to "the Lancet study" in the singular throughout the article, referring to Roberts et al (2004) only once as "a smaller study" (which you then claim "had been quickly contradicted by a larger UN survey suggesting that it had overstated excess mortality", which is also untrue, not least as the ILCS study to which you refer did not make an estimate of excess mortality, only of directly war-related mortality in which measure it agreed nearly exactly with Roberts et al (2004)).
Getting there, dsquared, getting there. Let's just say that the conflict epidemiologists I interviewed who were not in one of the two warring camps disagreed rather strenuously with your, and Les Roberts, claims on this subject.
dsquared says:
"the ILCS study to which you refer did not make an estimate of excess mortality, only of directly war-related mortality in which measure it agreed nearly exactly with Roberts et al (2004))."
It is true that ILCS did not make an excess mortality estimate (Megan seems to get that wrong), but reading the link you provide, the claim that it "agreed nearly exactly" with the 2004 Lancet is just fanciful. The blog you link to seems to fabricate this "nearly exact" match by way of making false calculations and groundless assumptions that lead to the match.
It notes correctly that ILCS estimates 24,000 war-related deaths, but then says, "The corresponding number from the Lancet study is 33,000 (the rest of the excess deaths are from increases in disease, accidents and murders)."
How and why is this the "corresponding number"? There are several problems here. First, the number of violent deaths estimated by Lancet 2004 was 57,600 (as given by co-author Richard Garfield: http://www.epic-usa.org/An_Interview_with_EPIC_A.html)
Somehow the blog you cite shaves off 24,600 (42%). It would appear that he's arbitrarily removing all "murders" that Lancet attributed to crime, but then this doesn't even add up. I believe there were 21 violent deaths, and 7 were classed as criminal murders (33%). So his 33,000 is wrong.
But then the whole premise he's using is wrong. On what basis does he assume all "murders" would be excluded from ILCS? It states the ILCS question had a multiple choice breakdown for when people reported deaths: "Disease / Traffic Accident / War related death / Pregnancy or childbirth / Other."
There is no logical category for "murders" except for "War related" or possibly "Other". Can we really assume that Iraqis who were seeing an explosion in violent crime post-invasion would say none of the killings of their family members were "War related", but instead just all "Other"?. Maybe in some cases they would put some murders into "other", but the blog you cite just assumes that they're all excluded from "war related". This assumption has no foundation.
Then he goes on: "When you allow for the fact that the Lancet study covered eighteen months rather than one year, the ILCS gives a slightly higher death rate." But "allow" how? This requires some kind of calculation. Due to the errors and baseless assumptions already discussed above, there's probably little point in finding out how he did this "allowance", but he never says how.
Also he gets the ILCS time-period wrong. The ILCS only began field work in March (one year), but it ended fieldwork in May (14 months). These means some areas would be interviewed at one year, others at 14 months. The difference between 12, 13 or 14 months, when the comparison is to 18 months, is not a small difference.
So he just mixes baseless assumptions with faulty calculations in order to produce a "nearly exact match", and lo and behold, he produces it.
In any case, if this phony match were true then the ILCS and Lancet 2004 did match, that would mean that both studies confirm that Lancet 2006 overestimated violent deaths during that period by at least a factor of 2: http://www.rhul.ac.uk/Economics/Research/conflict-analysis/iraq-mortality/L1_versus_L2.html
But since the match is not true, instead only Lancet 2004 suggest that, while ILCS suggests a much greater overestimate.
As I said in this note, I really don't want to get into a debate in the comments threads before I lay out some of this stuff, so I won't respond to the specifics, but I didn't err; I simplified for space in a way that doesn't materially change the import. Why I think it doesn't materially change the import I will go into later.
[ It would appear that he's arbitrarily removing all "murders" that Lancet attributed to crime, but then this doesn't even add up. I believe there were 21 violent deaths, and 7 were classed as criminal murders (33%). So his 33,000 is wrong.]
No, you've got this wrong; the difference corresponds to two violent deaths classified as "Other" and one execution carried out by the Saddam regime.
["When you allow for the fact that the Lancet study covered eighteen months rather than one year, the ILCS gives a slightly higher death rate." But "allow" how?]
How do you normally calculate an annual rate from an eighteen-month rate? Multiply by two and divide by three, divide by 1.5, divide by eighteen and multiply by 12, the choice is nearly endless.
[Also he gets the ILCS time-period wrong. The ILCS only began field work in March (one year), but it ended fieldwork in May (14 months). These means some areas would be interviewed at one year, others at 14 months.]
Once more, you are wrong. The fieldwork period is irrelevant to the question asked, which refers to the period ending twelve months after the invasion. This would have been rather a glaring error for ILCS to have made, which is why they didn't make it.
See, Tim Lambert is a careful bloke. On the evidence here, you're not.
[Let's just say that the conflict epidemiologists I interviewed who were not in one of the two warring camps disagreed rather strenuously with your, and Les Roberts, claims on this subject.]
hahaha "Let's just say that the people who weren't in one of the warring camps, were in a specific one of the warring camps". I'm presuming that you're referring to Jon Pedersen here and his questions about recall bias apply equally to the IFHS survey, which is why he's never put an oar into this debate. You did ask about statistical and methodological issues with respect to the IFHS survey, didn't you? I'd hate to think you were a hack.
"No, you've got this wrong; the difference corresponds to two violent deaths classified as "Other" and one execution carried out by the Saddam regime."
I don't have anything wrong here. What assumptions he might have made to get it wrong is not so relevant. The 33,000 is still wrong, and it is still groundless to remove all the "murders" for a comparison (even if you remove the right number).
Additionally, another point I did not note is that he forgot to consider Anbar. ILCS included it and the version of Lancet 2004 that he's comparing did not. So really he should have either removed Anbar from ILCS or put it back into Lancet before messing up the rest.
"How do you normally calculate an annual rate from an eighteen-month rate? Multiply by two and divide by three, divide by 1.5, divide by eighteen and multiply by 12, the choice is nearly endless."
...nearly endless, and all arbitrary. Any such method would almost certainly make your results wrong, as the rate of violent deaths will vary over time. None of those would be any good unless there was some reason to believe that the rate of deaths remained the same over the whole period. But if you think that's what he did, so be it.
"Once more, you are wrong. The fieldwork period is irrelevant to the question asked, which refers to the period ending twelve months after the invasion. This would have been rather a glaring error for ILCS to have made, which is why they didn't make it."
Once again you claim I've got something wrong and I do not. Your Lambert page quotes the ILCS question:
“Has any person(s) who was a regular household member died or gone missing during the past 24 months?”
The past 24 months would have been from the time of the interview until 24 months prior.
"Tim Lambert is a careful bloke."
He certainly was not at in this instance, unless we wish to say he carefully fabricated a phony match between the two studies with false calculations and baseless assumptions.
Manual trackback.
Maybe the fact that the Atlantic can't find the space to discuss statistical methods and analysis is one reason that it shouldn't be in the business of publishing critiques of scientific studies that typically employ statistical techniques for their analysis of data.
And then there's this other problem. You have no training in scientific methods, statistical analysis (unless you count that six-sigma seminar you went to while getting your MBA), or epidemiology, while the authors of the Lancet articles and peer-reviewers who accepted those same articles presumably do. So what are you talking about again?
"I have to challenge your first question, "How many Iraqis have died because of the American invasion?"
The deaths in Iraq since 2003 aren't "because of the American invasion." Most of the "excess deaths" since 2003 have not been at the hands of Americans. The fascists and fanatics caused most of those deaths, and they continue to inflict the largest part of Iraq's bloodshed."
If one approaches this with a near-mindless literalism, and a certain disturbing lack of comprehension (or an hysterical oversensitivity to certain moral issues), Frank is more or less correct here. In any other sense, he's wrong, and falls into the surprisingly large category of people - often, but not exclusively, conservatives - who seem to have a bizarrely simplistic notion of causation. Lakoff, iirc, has suggested that conservatives tend to stress direct (vs. liberalish systemic) causation, and this may be a (very extreme) example of this. The fact that a number of incredibly idiotic decisions - including both invading Iraq and the parade of occupational incompetence that followed - directly and fairly predictably led to a situation where rather unpleasant people are killing many of their fellow Iraqis/Arabs/Muslims simply seems to be beyond this person, which is both strange and frightening.
vapid: Yes, the Cult of the Expert is very much a part of the meme-complex that is the Lancet Cult. Show us something we don't know already.
PS. for all: I predicted to a friend that dscared would return to the safety of the womb once his bluster was challenged, and sure enough, to the Deltoid Mother he fled. Iraq mortality “experts” of his calibre can always exercise that option.
kimbu, this is all you got? vapid? dscared? changing one letter of a screen name to make something else? you must have been quite a wit in gradeschool.
btw, do you trust your medical choices to megan as well? it's not a cult, it's a bunch of highly trained people who review each other's work for no pay to make certain that it passes scientific muster - not an editor with no expertise rubber stamping a magazine piece to make deadline.
and really, try to make a point next time.
vapid: I can't help the fact that you and dscared are easy to skewer. Are you saying that credulous credentialism isn't vapid? That dscared hasn't vanished in a puff of ink-squid?
I also complained that you contributed nothing new. What was your response? To repeat yourself.
I have news for you (yes, only for you): Reiteration doesn't establish your case.
Megan,
While to trying to convince the reader of your argument, you actually failed to provide any hard evidence. If you work for some kind of a Zionist-Fascist organisation, why don't you be honest?
The reality is, you are trying to justifying the mass murder of millions of Iraqi civilians. You are a complicit.
Gali, Italy
Or was it merely dscared’s squid-ink?
;-)
how's this? statistics are important, and can be confusing to some. they are typically what scientific articles rely upon to come to conclusions instead of mere opinion. I don't see any evidence that megan has any understanding of statistical theory or methods and in fact just chose to leave this discussion out of a piece evaluating the scientific conclusions of a peer reviewed paper. that's bullshit
vapid: Those sentiments were implicit in your first email. But at least you’re trying to work through the definitions now, which is better than the assumption-fest you began with.
Statistical theory may well be just as correct as two plus two equalling four. Does that confirm the Lancet estimates? Only if you’re under the impression that that depends solely on whether statistical theory is correct and that 2 + 2 = 4 – or indeed the naiïve notion that peer review weeds out errors and fraud.
I suggest you look for information that doesn’t confirm your biases. Can you hold on to your beliefs despite such facts? Good. Then your beliefs probably aren’t as lacking in substance – as vapid – as the thoughts you’ve expressed so far.
good lord. JACKASS (see what I did, I changed ALL the letters in your name and added a few to make that)
"Only if you’re under the impression that that depends solely on whether statistical theory is correct and that 2 + 2 = 4 – or indeed the naiïve notion that peer review weeds out errors and fraud."
I'll leave the above to you to decode yourself.
If you weren't so busy trying to find some way not to feel bad about this war and then minimizing the damage that the US has done abroad you might have a firmer grasp on things. But as it is, you will only go to the opinions that make you feel better about what you likely supported from the beginning and then ignorantly hold forth on topics you just can't understand.