As I mentioned last time, psychology isn’t the only branch of science facing a reproducibility crisis. Biomedicine is also on fire at the moment, and eagle-eyed readers may have spotted an example I glided past. Schimmack (2012) developed a metric called the “Incredibility Index,” then took it for a spin on two studies: Bem’s now-infamous one, which I’ve had a lot of fun with, and a study on self-control and blood-glucose levels.[1]

Total power values for all three estimation methods are less than 1% and all IC-indices are greater than 99%. This indicates that from a statistical point of view, Bem’s (2011) evidence for ESP is more credible than Gailliot et al.’s (2007) evidence for a role of blood-glucose in self-regulation.[2]

Ouch, especially since Google Scholar suggests the latter study has been cited 873+ times. Judging from some of what I’m reading, biomed researchers have it worse than psychologists. I mean, really:

The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers. Those who have the power to act seem to think somebody else should act first. And every positive action (eg, funding well-powered replications) has a counterargument (science will become less creative). The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.[3]

Despite the dour tone above, the symposium report is worth reading. It’s also interesting to see Bayesian methods get a bit more respect, too:

Pre-registration describes a different way of publishing research articles, in which the introduction, hypotheses, methods and analysis plans are sent to the journal before the experiment is carried out (a ‘Stage 1 submission’). This submission is then subjected to peer review before data are collected.  … Submitted studies must also meet statistical power requirements, and include sufficient a priori power analyses or Bayesian sampling and analysis strategies.

Hopefully science can clean up its act more, and return to being something that gets solid results.


[1] Gailliot, Matthew T., et al. “Self-control relies on glucose as a limited energy source: willpower is more than a metaphor.” Journal of personality and social psychology 92.2 (2007): 325.

[2] Schimmack, Ulrich. “The ironic effect of significant results on the credibility of multiple-study articles.Psychological Methods 17.4 (2012): 551.

[3] Horton, Richard. “Offline: What is medicine’s 5 sigma?.” The Lancet 385.9976 (2015): 1380.