The preregistration revolution

(2018) - Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, David T. Mellor

Journal: Proceedings of the National Academy of Sciences
Link:: http://www.pnas.org/lookup/doi/10.1073/pnas.1708274114
DOI:: 10.1073/pnas.1708274114
Links::
Tags:: #paper #Pre-Analysis
Cite Key:: [@nosekPreregistrationRevolution2018]

Abstract

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

Notes

"As prediction error decreases, certainty about what will occur in the future increases. This view of research progress is captured by George Box's aphorism: "All models are wrong but some are useful" (1, 2)." (Nosek et al 2018:2600)

"To an audience of historians (15), Amos Tversky provided a cogent explanation of the power of hindsight for considering evidence: All too often, we find ourselves unable to predict what will happen; yet after the fact we explain what did happen with a great deal of confidence. This "ability" to explain that which we cannot predict, even in the absence of any additional information, represents an important, though subtle, flaw in our reasoning. It leads us to believe that there is a less uncertain world than there actually is...." (Nosek et al 2018:2600)

"Lack of clarity between postdiction and prediction provides the opportunity to select, rationalize, and report tests that maximize reward over accuracy" (Nosek et al 2018:2601)

"Rejection of the null hypothesis atP< 0.05 is a claim about the likelihood that data as extreme or more extreme than the observed data would have occurred if the null hypothesis were true" (Nosek et al 2018:2601)

"For some kinds of analysis, it is possible to define stages and preregister incrementally. For example, a researcher could define a preregistration that evaluates distributional forms of variables to determine data exclusions, transformations, and appropriate model assumptions that do not reveal anything about the research outcomes. After that, the researcher preregisters the model most appropriate for testing the outcomes of interest. Effective application of sequential preregistration is difficult in many research applications. If an earlier stage reveals information about outcomes to be tested at a subsequent stage, then the preregistration is compromised" (Nosek et al 2018:2602)

"robust option is to blind the dataset by scrambling some of the observations so that distributional forms are still retained, but there is no way to know the actual outcomes until the dataset is unblinded (47, 48)." (Nosek et al 2018:2602)