Evidence? Who wants evidence?

Ezra Klein recently called for a national medical effectiveness agency:

The final point is that we don't currently have good drug effectiveness information that could help insurers make value-based decisions. As such, they make cost-based decisions. The insurers are behind a national body that would produce this research, as well they should be. Zirkelbach is suggesting that the tiering system, with its somewhat arbitrary nature, is all they can do in the meantime.

I was thinking the same thing when I wondered why the FDA doesn't actually do the clinical trials; surely, the drug companies could pay for the trials the FDA conducts, which would remove the taint of self-interest from the proceedings, and ensure that results don't get buried.

But there are, as this post notes, some political problems associated with assembling that kind of data; how does that information get used? The question I worry more about, though, is does that information get used?

I've been reading (actually, listening to) Supercrunchers, by Ian Ayres. The gist of it is that the advent of huge datasets and better statistical methods is enabling us to replace intuition with vastly superior data-based analytics.

He covers everything from predicting wine quality, to Direct Instruction, to evidence based medicine. And the major unifying theme: in every case, experts have fiercely resisted quantitative methodology. This is why DI and EBM are still not widespread, even though they clearly offer vastly superior results: the professionals who need to implement them don't want to.

Ayres attributes this to overconfidence: even when we know that the script beats the human, we still think that we can beat the script. But he does not, in my opinion, linger long enough on the major source of resistance: the script makes professional jobs less fun. Being the voice, and arms, of a computer is a lot closer to being a data entry clerk than what doctors and teachers envisioned when they entered the profession. So despite overwhelming evidence, both efforts have stalled.

If we're going to have a federal institution do anything, it should be facilitating the roll-out of these kinds of data-driven systems. But the political power is on the side of the providers. I wonder with efforts like the one Ezra proposes whether it wouldn't be dogged by similar problems.

Comments

You misconstrue the difficulty with EBM, which is not that professionals (docs) dont want to use it--perhaps some don't. But the broader problem is the difficulty of applying the results of clinical trials to individuals. For this to work, judgment based upon individual circumstances is still necessary, after all the guidelines based upon trials have been read and digested. Hence the continued need for professional discretion.

Providers are absolutely right to fight the imposition of guidelines insofar as these override clinical judgment; of course we should make use of EBM, but as a complement to judgment, not as a replacement for it.

Posted by Tom H | April 16, 2008 12:41 PM

Please reconcile this with your post advocating light regulation on lending. Here, it's an outrage that evidence is ignored by so-called experts. In that post, it's an outrage that evidence (behavioral finance studies showing that people often make consistenly bad financial choices) could be used to constrain people's array of decisions.

Posted by BudFox2 | April 16, 2008 12:53 PM

"I was thinking the same thing when I wondered why the FDA doesn't actually do the clinical trials; surely, the drug companies could pay for the trials the FDA conducts"

If the FDA conducted the trials (a) they would get done muuuuuuuch more slowly and (b) the FDA would have to follow standard procurement rules, which would contribute to (there are of course other retardant factors as well) the slowness and would otherwise be cumbersome and generally lead to lower quality results.

Posted by ostap | April 16, 2008 1:02 PM

ostap--

The FDA would do what the drug companies do, outsource the work to clinical studies companies. The reason the drug companies run the trials is so they can control the publication of results. It would not be in their interests to have adverse event reporting go straight to the FDA, for example.

They would surely not want a third party getting first crack at the results. What they want is a crippled FDA that certifies their drugs as safe and effective, and that that certification can be used as a shield from law suits.

Posted by jayackroyd | April 16, 2008 1:29 PM

The section on Direct Instruction gave some evidence that even the intuition about making teaching less fun could be wrong.

Page 164 (Hardback) - "Teachers interviewed after implementation of DI in Broward County, Florida, said the 'approach actually allowed more creativity, because a framework was in place within which to innovate,' and added that classroom innovation and experimentation were a lot easier once DI had helped students acquire the necessary skills."

What's really the case is that these sorts of procedural methodologies sound a lot less fun. In reality, what it means is that persistent sources of problems will be dealt with more effectively, freeing up intelligent professionals (whether teachers or doctors) to do things with higher value.

Posted by TW Andrews | April 16, 2008 1:37 PM

"For this to work, judgment based upon individual circumstances is still necessary, after all the guidelines based upon trials have been read and digested. Hence the continued need for professional discretion."

No, what it means on average is that the statistics will yield better results on average for the individual patients. It means that doctors are worse for the individual cases, however sometimes they may do better.

If somebody told me I had a 70% chance to get better by one method, and a 50% to get better by another method I'd choose the 70% option every time. However, there will be some cases where the 50% option would have been better individually but there's no good way to determine that until after the fact.

Posted by JordanT | April 16, 2008 2:19 PM

A comment on EBM. I am a fan, relatively speaking and intimately involved with the creation of many of these guidelines for trauma care for the Eastern Assn for the Surgery of Trauma(see www.EAST.org for examples)but there are some major implementation problems.
1) Applicability (as the first commenter stated) -- if you apply all the medication recommendations to the average 75 year old you get a list of 8-15 medications, all of which interact in unpredictable ways. No one has done the studies on polypharmacy (imagine how many groups you would need even to study someone on 4 medications)
2) What I call the "parachute paradox" -- how do you get someone to sign up for a randomized study entitled "the utility of parachutes in falls from heights, a blinded study". According to EBM we haven't proved that appendectomy is useful for appendicitis, but it'll never be studied -- like most operative procedures.
3) Complexity -- I'm a critical care/trauma surgeon, most of what we do in the ICU has not been shown to improve mortality but has been shown to improve some sort of physiologic variable which "should" improve mortality if you get big enough numbers to study. However how do you get enough people with exactly the same injuries, medical history, body mass index and etc. Not to mention it is extremely hard to obtain consent early in the hospital course (if you're sick enough to need the ICU you're often too sick to consent and often we can't find family).

None of this should prevent the continued push for evidence and the creation of EBM guidelines, but it is not a simple matter to fit people into the rules.

Posted by BladeDoc | April 16, 2008 2:20 PM

I did my graduate work in this area, specifically using mathematical models to ascertain drug efficacy, and I'm starting a post doc which will deal with many similar and tangential topic.

There is a wealth of information out there for many of these compounds. I mean there are standard practices established which are more or less required to get published.

What I think is really needed is someone to interpret this information and distill it down into decision making criteria.

Of course this doesn't really help in terms of alleviating legal responsibility. I've talked to industry folks (R&D people from places like Pfizer) and they frequently comment that the FDA shifts their goal posts. The problem is that this shift occurs during development. It adds a significant amount of uncertainty, and that's not good when a lot of money is at stake.

Posted by John Harrold | April 16, 2008 6:03 PM

JordanT:
"No, what it means on average is that the statistics will yield better results on average for the individual patients. It means that doctors are worse for the individual cases, however sometimes they may do better."

well, actually thats not what it means. Average effects in trials can obscure lack of homogeneity among patients in a treatment trial such that action if taken as suggested by average effects for the group will sometimes be predictably harmful for individuals least likely to benefit from the treatment. This is explicated by DB at http://www.medrants.com/index.php/archives/3368. See also Rod Hayward's article referenced there.

It simply will not do to blindly follow guidelines in individual cases for whom the guidelines are a bad fit. Clinical judgment is still necessary.

Posted by Tom H | April 16, 2008 6:45 PM

EBM would be a great resource for doctors. The patient could fill in the questionnaires and all the data could be put into the system. Then the doctor would get a list of results. Given the differences in human bodies and backgrounds, etc... it would be up to the doctor to make the final determination.

EBM can help, though, because it's just not possible for a human doctor to keep up with all of the research and science. An EBM database, though, could be continually updated with the latest results so that all doctors who used it would at least have access to this information.

But doctors are (rightly) afraid that EBM will be used as an excuse to eliminate them or dramatically reduce their role. This might make medicine cheaper, but I'm not sure it would be better, overall.

Posted by Earnest Iconoclast | April 16, 2008 8:57 PM

Has anybody applied statistical analysis to statistical analysis? What is the track record of people who thought they knew better than the experts?

On the other hand, what is the track record of most experts?

Posted by Joseph Hertzlinger | April 17, 2008 2:27 AM

Over at d-edreckoning there is a series of articles about a small Kansas school district that adopted DI with impressive results, the series starts with this article:
http://d-edreckoning.blogspot.com/2008/04/gering-public-schools-school-district.html

There is also a promotional video:
http://www.nifdi.org/gering_video.html

Bottom line is that they were able to dramaticaly raise the reading scores, particularly for the hispanic english as a second language population.

Posted by Robert Sperry | April 17, 2008 1:48 PM