The gold standard of health insurance studies is the 1982 RAND study, which was a randomized controlled study of health care usage. One of the surprising findings was that when you pushed the cost down, usage went up, but health didn't.
However, Alex Tabarrok points to a major flaw in the study:
Of the 1,294 adult participants who were randomly assigned to the free plan, 5 participants (0.4 percent) left the experiment voluntarily during the observation period, while of the 2,664 who were assigned to any of the cost-sharing plans, 179 participants (6.7 percent) voluntarily left the experiment. This represented a greater than sixteenfold increase in the percentage of dropouts, a difference that was highly significant and a magnitude of response that was nowhere else duplicated in the experiment.What explains this? The explanation that makes the most sense is that the dropouts were participants who had just been diagnosed with an illness that would require a costly hospital procedure. … If they dropped out, their coverage would automatically revert to their original insurance policies, which were likely to cover major medical expenses (such as hospitalizations) with no copayments.
Who drops out of your study is an important factor that is far too often overlooked, especially by journalists. Weight loss studies are particularly bad for this--the people who aren't losing weight are, naturally, the ones who tend to drop out, which vastly overstates the success figures. But all sorts of medical studies are prone, including, it seems, this one.
Good catch. For mortality endpoints you might be able to do a death records search of the drop-outs and include them in an "intention to treat" analysis that would still be broadly valid (despite a bias to the null). But this makes the primary study uninterpretable "as is" as the censoring was clearly not at random if there is a difference between the exposure groups.
Seven percent is quite large and could hide shockingly large differences . . .
It appears that the flaw isn't nearly as serious as you might think. The RAND researchers did follow up on the drop-outs and check on their health. See the comments to Alex's post for details.
The "gold standard" is a study over 20 years old?
A twenty year-old study that no one living in an industrialised country with socialised health care and higher health output measures than the US has ever heard of?
Some gold-standard.
While there was some assessment of the drop-outs, this was incomplete and so remains a threat to study validity.
There is a more general question of whether the study answers the question that people think it does but that is a different argument.
If only RAND had done another study since 1982. Of course, it's not like things have changed much since then....
While there was some assessment of the drop-outs, this was incomplete and so remains a threat to study validity.
Not a serious threat.
If only RAND had done another study since 1982. Of course, it's not like things have changed much since then....
If you think there is evidence that things have changed in such a way as to conflict with the basic finding of the study, please produce it. Of course, medical science and technology have advanced since 1982, but as far as I'm aware there have been no advances in health care that would significantly influence its impact on aggregate morbidity and mortality rates. The effects of health care on health and longevity are still swamped by lifestyle and environmental factors like the rate of smoking and the amount of exercise people get.