All of my talk about the nuts and bolts of probability theory and hypothesis testing might seem a bit abstract. Certainly, there’s no shortage of scientists who dismiss it as much ado over nothing.
We discuss three arguments voiced by scientists who view the current outpouring of concern about replicability as overblown. The first idea is that the adoption of a low alpha level (e.g., 5%) puts reasonable bounds on the rate at which errors can enter the published literature, making false-positive effects rare enough to be considered a minor issue. This, we point out, rests on statistical misunderstanding: The alpha level imposes no limit on the rate at which errors may arise in the literature (Ioannidis, 2005b). Second, some argue that whereas direct replication attempts are uncommon, conceptual replication attempts are common—providing an even better test of the validity of a phenomenon. We contend that performing conceptual rather than direct replication attempts interacts insidiously with publication bias, opening the door to literatures that appear to confirm the reality of phenomena that in fact do not exist. Finally, we discuss the argument that errors will eventually be pruned out of the literature if the field would just show a bit of patience. We contend that there are no plausible concrete scenarios to back up such forecasts and that what is needed is not patience, but rather systematic reforms in scientific practice.
Pashler, Harold, and Christine R. Harris. “Is the Replicability Crisis Overblown? Three Arguments Examined.” Perspectives on Psychological Science 7, no. 6 (2012): 531–36.
But the debate over p-values, power levels, and valid statistical tests carries real, far-ranging consequences.
Over the last decade, there’s been a lot of talk about reproducibility problems in science—about published results that turn out to be false alarms. In fields like psychology, neuroscience, and cell biology, these errors can send scientists down unproductive paths, waste time and money, and pollute headlines with misleading claims. “But I get much more exercised about reproducibility problems in clinical genetics, because those have massive and real-time consequences for thousands of families,” says MacArthur.
People get abortions on the basis of mutations that are linked to severe congenital diseases. They get mastectomies on the basis of mutations in breast-cancer genes. They get monitoring devices surgically implanted in their chests on the basis of mutations in heart-disease genes. “This is absolutely an issue, and it’s led to all sorts of problematic decision-making,” says Rehm.
Ed Yong has a long piece outlining why.
“All we need to do to get there is convince researchers around the world to share their data, build the world’s largest repository of genetic and clinical information, and develop functional tests for every gene in the human genome,” he adds. “Easy.”