It’s these things which convince me the “philosophy is useless” crowd is dead wrong.
fundamentally there is only one thing different about this deworming trial and the rest of social science and medicine: Miguel and Kremer had the decency, generosity, strength of character, and intellectual confidence to let someone else peer under the bonnet.
This kind of statistical replication is almost vanishingly rare. A recent study set out to find all well-documented cases in which the raw data from a randomized trial had been reanalysed. It found just 37, out of many thousands. What’s more, only five were conducted by entirely independent researchers, people not involved in the original trial.
These reanalyses were more than mere academic fun and games. The ultimate outcomes of the trials changed, with terrifying frequency: One-third of them were so different that the take-home message of the trial shifted.
The current system of pursuing science grew organically, and only rarely were the fundamental assertions challenged. To turn a phrase from Bruce Scheiner, the result is a sort of “science theatre,” or the appearance of reliable conclusions that disguise fundamental flaws.
The best available evidence — from dozens of studies chasing results for completed trials — shows that around half of all clinical trials fail to report their results. The same is true of industry trials, and academic trials. What’s more, trials with positive results are about twice as likely to post results, so we see a biased half of the literature.
This is a cancer at the core of evidence-based medicine. When half the evidence is withheld, doctors and patients cannot make informed decisions about which treatment is best.
While I don’t always agree with philosophers of science, their work is vital to shoring up these issues. The frequentism vs. Bayesian debate may seem pedantic, but its outcome determines how we gather and analyze data. The same also goes for activists like Ben Goldacre, who push hard to ensure science is more than good theatre.
I befriended some campaigners, we assembled a group of senior academics, and started the AllTrials.net campaign with one clear message: “All trials must be registered, with their full methods and results reported.”
Dozens of academic studies had been published on the issue, and that alone clearly wasn’t enough. So we started collecting signatures, and we now have more than 85,000 supporters. At the same time we sought out institutional support. Eighty patient groups signed up in the first month, with hundreds more since then. Some of the biggest research funders, and even government bodies, have now signed up.
This week we’re announcing support from a group of 85 pension funds and asset managers, representing more than 3.5 trillion euros in funds, who will be asking the pharma companies they invest in to make plans to ensure that all trials — past, present, and future — report their results properly. Next week, after two years of activity in Europe, we launch our campaign in the U.S.
Science is changing, and I couldn’t be happier about it.