In the process of interviewing health care economists for the article I'm writing, David Cutler of Harvard told me that education and health care are the two segments of the economy where measurement of outcomes is hardest. It's unsurprising, then, he told me, that payment schemes in those two systems are so haphazard.
But even when there is good evidence, the systems don't use it. One of the most extreme examples is back surgery. The evidence seems to be completely overwhelming that back surgery is a terrible idea for back pain in all but a small minority of cases. Yet we do a phenomenal amount of back surgery. I presume that the surgeons performing this surgery are not venally crippling their patients in order to charge them exorbitant amounts of money. So why do they do something that shows so little evidence of improving the lives of their patients? And why do insurance companies and government programs pay for it?
The uncharitable explanation I have received is that doctors are incredibly good at detail and memorization and pattern recognition, and generally very bad at abstract analysis; they are also, because med school is so hard to get into, usually very arrogant.
The charitable (well, more charitable, anyways) explanation is that doctors are like the rest of us: they have an unreasonable belief that they are above average. (80% of people think that they are better-than-average drivers). And so even though they recognize that back surgery doesn't work on average, they think that they can beat the odds. They also naturally resist anything, like evidence-based medicine, that routinizes their jobs and cuts down on discretion. And we let them because medicine is important--so important that we can't stand thinking about the catastrophic possible errors, and so we prefer to turn our care over to priest-gods to whom we impute improbable, nay inhuman, wisdom.
Alex Tabarrok points out another example of this operating in education:
What we need to save inner-city schools, and poor schools everywhere, is a method that works when the teachers aren't heroes. Even better if the method works when teachers are ordinary people, poorly paid and ill-motivated - i.e. the system we have today.In Super Crunchers, Ian Ayres argues that just such a method exists. Overall, Super Crunchers is a light but entertaining account of how large amounts of data and cheap computing power are improving forecasting and decision making in social science, government and business. I enjoyed the book. Chapter 7, however, was a real highlight.
Ayres argues that large experimental studies have shown that the teaching method which works best is Direct Instruction (here and here are two non-academic discussions which summarizes much of the same academic evidence discussed in Ayres). In Direct Instruction the teacher follows a script, a carefully designed and evaluated script. As Ayres notes this is key:
DI is scalable. Its success isn't contingent on the personality of some uber-teacher....You don't need to be a genius to be an effective DI teacher. DI can be implemented in dozens upon dozens of classrooms with just ordinary teachers. You just need to be able to follow the script.
Contrary to what you might think, the data also show that DI does not impede creativity or self-esteem. The education establishment, however, hates DI because it is a threat to the power and prestige of teaching, they prefer the model of teacher as hero. As Ayres says "The education establishment is wedded to its pet theories regardless of what the evidence says." As a result they have fought it tooth and nail so that "Direct Instruction, the oldest and most validated program, has captured only a little more than 1 percent of the grade-school market."
It's borderline criminal that even where evidence is available, we don't hold our teachers and doctors to the same standards of performance as would operate in a normal business.
But is efficient transmission of information really the biggest issue in the inner city schools? DI doesn't sound like it addresses the problem of students who are more interested in doing whatever pops into their heads from second to second than learning anything. Maybe teaching can be done from a script, but can classroom control?
"The evidence seems to be completely overwhelming that back surgery is a terrible idea for back pain in all but a small minority of cases. Yet we do a phenomenal amount of back surgery"
Reminds me of a story Charlie Munger tells about a particular eye surgery that had become completely obsolete in light of new methods. Munger relates how a famed ocular surgeon continued to teach the procedure to classes of new student, despite it's obvious flaws. Munger asked him why and he replied, cause its such a great procedure to teach!
Okay, I admit it. My ideological bias against DI, based on the description you provide here, is so great that I am disinclined to believe what you are saying. I will need to see it in action and see a hell of a lot of data before I take this claim seriously.
At most, I would believe that it would make sense to structure part of one's study day as DI, and the rest in a more independently designed format where the teacher retains greater control.
What's the alternative to back surgery that produces better results, anyway? It does seem true that once you start the back surgeries, they never stop, but is just toughing it out a viable option?
Anecdotal evidence time: Back pain put my dad into a state where he practically couldn't do anything at all, including sleeping in a bed, walking more than half a block, etc. etc. He got a surgery (or more accurately, he got the most recent of a series of surgeries) and now he's not doing too badly for the old dude that he is. So, the surgery bought my dad a couple of better years in terms of quality of life. What else could he have done that would have had as good a result, even if he's going to need more surgery in a couple of years?
Now, you can argue that the taxpayer shouldn't have been responsible for paying for all these surgeries (and of course we/you/they were) but you make it sound like back surgeries just plain don't work at all. Is that what you meant?
As far as schools is concerned we need a critical mass or responsible concerned parents for school to be effective in inner city schools.
Anything else is nice, but few things can motivate a kid like a loving, concerned, involved parent.
Teachers aren't heros as your citation pointed out, and no amount of procedure can make up for that fact.
Physical therapy, antidepressants, doing nothing, and even cutting disability payments for back pain all work at least as well as back surgery in trials. Or so the number crunchers say. The placebo effect is powerful. Plus people who get surgery can't see what would have happened if they didn't do the surgery, which is apparently usually that even without PT, it probably would have gotten better.
There are a minority of cases where back surgery is effective; perhaps your father was one of them. But generally, it is no more effective than "wait and see".
Tabarrok is mostly right about Direct Instruction. However, it is one of two reading programs that have a substantial amount of evidence for effectiveness. The other is Success for All and SFA actually has a greater research base than DI. (Full disclosure- I'm a curriculum/technology developer for SFA).
Anyway, there has certainly been resistance to DI and SFA from a segment of the education community because of both programs' more "scripted" format. But the story of why both programs aren't more widely adopted is bigger than that.
One of the Bush Administration's only initiatives with actual potential was Reading First, which tied federal funding to the adoption of reading programs with research backing. Had the program been implemented according to the letter and spirit of the legislation, DI and SFA would be the most widely used programs in the country. Instead, of course, Reading First became another casualty of the Bush Administration's crony capitalism. The Dept. of Education secretly drew up lists of "acceptable" programs that shockingly included programs published by large commercial textbook publishers, that had no research base, and excluded DI and SFA. There was a major Inspector General investigation into this scandal and some heads did roll.
Maybe teaching can be done from a script, but can classroom control?
Heh, the first education professional chimes in with an objection.
I said:
Maybe teaching can be done from a script, but can classroom control?
and then Kevin said:
Heh, the first education professional chimes in with an objection.
Not at all. It was an honest question. All I hear about the local terribly managed horrible results inner city school system is how out of control some(maybe a tiny minority, but who knows) of the students are, and how the concern the teachers feel for their physical safety gets in the way of teaching. etc. etc.
If you tell me a teaching system works in an environment where learning actually has some chance of occurring, then I could very well agree with you. But I don't think that's necessarily what you have in the inner city schools.
BP,
As a former teacher (who taught inner city kids) I can tell you that the best behavior management tool is a good lesson plan. When the teacher has a clear idea of what is to be taught and how it is to be taught, and the methods he/she is using are engaging and effective, you will see a dramatic decline in disruptive behavior.
What about me? I've had lower back pain for a long time, and gotten the following treatment:
-several physical therapists
-a chiropractor
-minor surgery (a steroidal injection into the spine)
-massage therapy (through the chiropractor)
Nothing worked. At least you can't claim in my place "it would have gone away anyway".
A recent study reported that acupucture had a ~50% success rate vs. ~25% for physical therapy. But that's a placebo too, right?
BP_Beckley's point is good too: DI seems to assume the student is under control. How replicable is the ability to accomplish that?
feral1 -
I think if you read the IG reports you won't find that there was any evidence of a secret list. What you will find is that the law required the Department to only approve grants for curricula that were based on sound research. The organizing statute for the Department also prohibited the Department from approving or disapproving specific curricula. A bit of a catch-22 for the Department. They couldn't tell states which curricula would be approved they could only tell them whether their application was approved or not with only the most generic comments for advice.
Add to that heavy-handed lobbying from home-state politicians for their state's grants to be approved, an understaffed Reading First office, and poor contracting procedures and you have the result. Not an evil-Bush administration conspiracy but a much more common Washington story -- a well-intended but poorly written law that was unevenly executed.
The real surprise is that the program appears to be a success at least by government program standards. It is at least demonstrating some positive results - more than can be said for most prek-12 federal education programs.
Warning: thread hijack in progress...
I used to have bad lower back pain from a compressed vertebrae. I tried chiropractic (which helped with upper back, but not lower), acupuncture and something called the Egoscue method. No luck. Then I tried a local once-a-week, 15 minute physical therapy service called America's Back and the pain is pretty much gone. Amazing. I stopped it for about nine months because they closed the local office. The pain came back, so I bit the bullet and now make the weekly one-hour drive to their office. Pain is gone again. I've considered approaching them about franchising, if for not other reason than to guarantee I'll always have the treatment, but I'm not sure how profitable they are.
Here's their site: http://www.americasback.com/
Back to DI: I don't know anything about this technique, but it sounds similar to methods advocated by (I kid you not) my childhood soccer coach. He used to design training programs for mission-critical programs like the NASA Space Shuttle, so it's based on a lot of real-world use (though there is admittedly a world of difference between an astronaut and an inner city kid of average IQ). His book is here: http://www.amazon.com/Eden-Conspiracy-Educating-Accomplished-Citizenship/dp/0966501004
Derek,
I'd like give a shout-out to the Alexander Technique. I went to a session back when I was a music major, and it worked pretty darn well.
I also recommend military-style situps.
Back to the healthcare discussion at the beginning of the post...
But isn't the problem with something like evidence-based medicine that it centralizes decisionmaking? Obviously, patients would want to choose doctors who were aware of what treatments worked best, based on research and experience. But calls for complete reliance on evidence-based medicine, particularly among state-funded health systems, often prevent experimental treatments from taking root or deny high-risk (but perhaps high-reward) procedures altogether. And they tend to move the decisionmaking process into the hands of budget analysts who may not have sufficient information.
And if evidence-based medicine becomes the preferred policy, wouldn't governments and insurance companies (payers of all stripes) have a distinct incentive to disallow treatments unless they had a certain threshold of demonstrable success? Who decides that threshold?
Maybe the only way to avoid that approval question is for patients to purchase treatments on their own.
Derek,
Damn you to hell...
From your comment I thought America's Back was ALREADY franchised...I don't think I can make the 8 hour drive (one way) once a week...
John,
I've read the IG report and while it was good start,it did not present the whole picture.
See some excerpts from a Washingto Post article posted below to get a fuller picture-
Billions for an Inside Game on Reading
By Michael Grunwald
Sunday, October 1, 2006; B01
President Bush's No Child Left Behind Act was premised on three revolutionary goals....
The third was to bring facts and evidence to the notoriously squishy world of education policy, promoting teaching methods backed by "scientifically based research" instead of instinct and fad. This was the least-publicized goal, but arguably the most vital; the phrase "scientifically based research" appeared more than 100 times in the landmark 2001 law.
The centerpiece of the new research-based approach was Reading First, a $1 billion-a-year effort to help low-income schools adopt strategies "that have been proven to prevent or remediate reading failure" through rigorous peer-reviewed studies. "Quite simply, Reading First focuses on what works, and will support proven methods of early reading instruction," the Education Department promised.
Five years later, an accumulating mound of evidence from reports, interviews and program documents suggests that Reading First has had little to do with science or rigor. Instead, the billions have gone to what is effectively a pilot project for untested programs with friends in high places.
Department officials and a small group of influential contractors have strong-armed states and local districts into adopting a small group of unproved textbooks and reading programs with almost no peer-reviewed research behind them. The commercial interests behind those textbooks and programs have paid royalties and consulting fees to the key Reading First contractors, who also served as consultants for states seeking grants and chaired the panels approving the grants. Both the architect of Reading First and former education secretary Roderick R. Paige have gone to work for the owner of one of those programs, who is also a top Bush fundraiser...
On Sept. 22, the department's inspector general released a report exposing some of Reading First's favoritism and mismanagement. The highlights were internal e-mails from then-program director Chris Doherty, vowing to deny funding to programs that weren't part of the department's in-crowd: "They are trying to crash our party and we need to beat the [expletive] out of them in front of all the other would-be party crashers who are standing on the front lawn waiting to see how we welcome these dirtbags."...
But the report barely scratched the surface of the incestuous process that dominated the formation of Reading First. The initiative didn't promote scientifically based reading instruction, the third goal of No Child Left Behind...
Bush administration officials frequently say that Reading First does not play favorites or intrude on local control, that states and districts are free to choose their own textbooks and programs -- as long as they're backed by sound science. But aggressive muckraking by the newsletter Title 1 Monitor and reading advocates at the Success for All Foundation have eviscerated those claims, and the inspector general's report officially contradicted them, accusing the department of breaking the law by promoting its pet programs and squelching others. In his internal e-mails, Doherty frequently admitted using "extralegal" tactics to force states and local districts to do the department's bidding. A report by Success for All documented how state applications for Reading First grants that promoted the preferred programs were the only ones approved.
In fact, the vast majority of the 4,800 Reading First schools have now adopted one of the five or six top-selling commercial textbooks, even though none of them has been evaluated in a peer-reviewed study against a control group. Most of the schools also use the same assessment program, the same instructional model, and one of three training programs developed by Reading First insiders -- with little research backing.
"They kept denying it, but everybody knew the department had a list," said Jady Johnson, director of the Reading Recovery Council of North America. "They're forcing schools to spend millions on ineffective programs."...
Success for All is the phonics program with the strongest record of scientifically proved results, backed by 31 studies rated "conclusive" by the American Institutes for Research. And it has been shut out of Reading First. The nonprofit Success for All Foundation has shed 60 percent of its staff since Reading First began; the program had been growing rapidly, but now 300 schools have dropped it. Betsy Ammons, a principal in North Carolina, watched Success for All improve reading scores at her school, but state officials made her switch to traditional textbooks to qualify for the new grants.
"You can't afford to turn down the federal money," Ammons said. "But why should we have to give up on something that works?"...
The answer lies in the Reading First grant process, which was almost comically skewed. Michigan was the first state approved, after it simply proposed to adopt the five best-selling textbooks. But when Rhode Island officials proposed to require "high-quality reading programs that meet the test of having a scientific research base," they were rejected. Doherty told them to check out Michigan's list, so they cut and pasted it into their application, while suggesting that districts could still adopt other programs justified by research. They were rejected again. So they limited their program to the textbooks. Only then were they approved. Similarly, Oklahoma unsuccessfully proposed to require reading programs backed by three years of longitudinal data before it got the hint and proposed the Michigan list.
So instead of advocating scientifically based reading programs, Reading First has promoted programs with "key elements" endorsed by a national reading panel, which could describe almost any program. It may not be a coincidence that the initiative was essentially outsourced to a few experts with a dizzying array of apparent conflicts of interest.
For example, when the department needed reviewers to evaluate reading assessment programs, it contracted with a University of Oregon team led by Edward Kame'enui, Roland Good and Deborah Simmons. Good had developed an assessment called Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and Kame'enui, Good and Simmons had all served on the design team for Voyager Passport, a remedial program built around DIBELS. Ultimately, DIBELS was the only assessment used in Reading First, and Voyager was the most popular supplemental program...
But it is clear that Reading First has been a terrific boon for the textbook publishing industry, and for the department's favored programs. For example, the company that developed Voyager Passport was valued at about $5 million in a newspaper article before Reading First; founder Randy Best, whose Republican fundraising made him a Bush Pioneer, eventually sold it for $380 million. He then put Lyon and Paige on his payroll.
BP Beckley,
I agree with the poster who said that the best classroom management technique is a good lesson plan. Also, I'm not sure about the numbers, but my impression is that DI gets used with disadvantaged children, when it's used at all. In fact, its debut was with the anti-poverty experiment "Project Follow Through" which you can read about on Wikipedia.
brooksfoe,
Doing DI for reading and math for half a day and then doing non-DI stuff is how one's supposed to do it, I gather. For more information, Ken DeRosa at d-edreckoning.blogspot.com is your guy.
The Washington Post could just as easily said:
"...Reading First; founder Randy Best, who donated tens of thousands of dollars to Democratic Party PACs, eventually sold it for $380 million" (according to opensecrets.org).
But I guess that would have ruined their "Bush is evil" narrative.
Oh noes! How are we ever going to control our students??
Err issue every teacher a Desert Eagle .50 and the permission to use it? Failing shooting the unruly out of hand, perhaps bring back corporal punishment - schools had much success with leather belts and rulers applied to wrists and other tender parts. Essentially you need to understand that you are the adult in the room and that you must take control. If you can't, QUIT TEACHING!! You are just making the problem worse.
"Priest-Gods"? Did you mean "Priest-Kings"?
Megan - Are you Gorean? Please tell us. I think this is important for the readers to know. Is it a game for you or a lifestyle? Again, we should be told.
(And, yes, I'm embarrassed to say I even know something about this.)
it seems like every side on the debate, from free market libertarians to single payer advocates, desire more restrictions put on doctors' ability to make decisions.
Doctors have way too much power in the system.
But is efficient transmission of information really the biggest issue in the inner city schools? DI doesn't sound like it addresses the problem of students who are more interested in doing whatever pops into their heads from second to second than learning anything.
The experience from the guys who developed and implemented DI in inner city schools is that kids are motivated to do what they are good at.
Normal scenario:
- kids start at school at kindergarten, and the teacher starts teaching the kids to read. The teacher is probably inexperienced and has never been properly trained in how to teach reading.
- the teacher's curriculum has not been properly developed and therefore it omits some things a kid needs to know to be able to read (one of the things the developer of DI discovered when developing his curriculum is that some kids didn't know the meanings of words like "above" and "below"). Some kids in the class have already been taught these things by their parents, or are naturally very intelligent, and manage to figure it out. Others who are less fortunate, don't.
- the less fortunate kids therefore don't understand the lesson. And then they don't understand the following lessons. And they see other kids in the class understanding it all. And the school is sending out the message that reading is ultra-important, but the kid can't read because they don't understand the lessons because they lacked a bit of crucial knowledge the curriculum assumed.
- Frustration sets in.
- An individual kid could therefore go two ways. One is to feel absolutely miserable about their failure to develop the ultra-important skill of reading and become convinced that they can't learn anything. The other is to reject the idea that reading is ultra-important.
- In any case, if they can't read, most of the rest of school is going to be deeply confusing.
And another sad story can play out. Kid enters the school, can already read, do basic maths, etc. The teachers at the school are flat out dealing with all the kids who don't have such a good background. Kid gets bored, decides school is a waste of time, starts causing riots out of boredom. (Not all well-prepared kids do this, but some do).
Is it surprising that in a few years time these kids are more interested in doing whatever pops into their heads at the moment than learning anything?
But, if the curriculum is well-designed so it all builds on itself and the school is set up to support this, then this sad scenario doesn't happen. Instead what happens is:
- the teacher may be inexperienced (every teacher must be new once) but has been taught how to teach reading, and has DI, which the whole school is operating to.
- the teacher, or someone else at the school, tests every kid to find out what they already know, and places them in the curriculum accordingly
- the DI is set up so for each lesson the kid has already been taught everything they need to know to understand the lesson
- the teacher is constantly testing the kids' understandings on each point, so if the kid doesn't understand a point the teacher immediately knows that it's time to reteach that point.
- the script has many places for the teacher to praise the kids for doing so well. (the developer, when he was doing it, would also tell the kids that they were doing ultra-hard work and he didn't expect them to be able to do it, but I don't know if that got into the scripts. It certainly gave the kids a self-esteem boast when they then could answer his questions right).
- so the kids are constantly having the reinforcement of understanding their lessons and being praised for doing so well. School is a place where they accomplish something every day in a way that is valued by the school. Plus the kids become convinced that they can learn something.
The well-prepared kid is placed further along the sequence, so they are constantly learning new things as well.
So which set of kids is going to be more motivated to learn? Ones who know they have successfully learnt a lot in the past, who believe they are good learners, who understand all the background information necessary to understand the school lessons? Or kids who either haven't learnt to read or have been bored stiff for the last few years?
Tracy W,
That was a really beautiful summary of the thinking behind Direct Instruction. Note how, although the curriculum is scripted, it is actually highly individualized and sensitive to the particular needs of the children.
Very interesting... as always! Cheers from -Switzerland-.
Thank you for the DI stream. as a student teacher the level of DI is an interesting question. I am interested in the lower grades and have learned that if you are going to teach someone something for the first (few) times you have to lay the Correct pattern... doesn't matter how old the student is. THEN they can be asked to have an opinion - because they have some language. The best way i have seen to teach reading, to young or underachieving students, is singing. (and may i say 'naughty Mr. Bush')....
from Australia