Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

(2015) - Lucas C. Coffman, Muriel Niederle

Journal: Journal of Economic Perspectives
Link:: https://pubs.aeaweb.org/doi/10.1257/jep.29.3.81
DOI:: 10.1257/jep.29.3.81
Links::
Tags:: #paper #Pre-Analysis
Cite Key:: [@coffmanPreAnalysisPlansHave2015]

Abstract

The social sciences—including economics—have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the “file drawer problem,” which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's “file drawer.” Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias—either replication attempts bent on proving or disproving the published work—or modest amounts of poor replication attempts—designs that are underpowered or orthogonal to the hypothesis—replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications.

Notes

"A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins" (Coffman and Niederle 2015:81)

"recent empirical literature suggests the behavioral problems that pre-analysis plans attenuate are not a pervasive problem in experimental economics" (Coffman and Niederle 2015:81)

"pre-analysis plans have quite limited value in cases where more than one hypothesis is tested, piloted, or surveyed, and also where null results may not be reported" (Coffman and Niederle 2015:81)

"pre-analysis plans may discourage the use of novel research designs and hence inhibit studies of robustness of previous findings." (Coffman and Niederle 2015:82)

"pre-analysis plans limit the freedom of researchers concerning which hypothesis to investigate" (Coffman and Niederle 2015:82)

"researcher is restricted on how to test the hypothesis" (Coffman and Niederle 2015:82)

"the researcher often also precommits to a data collection plan. In particular, the researcher cannot stop collecting data only when a desired level of statistical significance has been reached." (Coffman and Niederle 2015:82)

"obtain a theory-driven estimate for this question" (Coffman and Niederle 2015:84)

""positive predictive value."" (Coffman and Niederle 2015:84)

"pα first parameter is the statistical significance threshold for a positive result." (Coffman and Niederle 2015:84)

"second parameter is the "power" of the study" (Coffman and Niederle 2015:84)

"bi A smaller means a more powerful study." (Coffman and Niederle 2015:84)

"π third parameter is the proportion of studies that are testing true hypotheses (or the expected probability of a hypothesis being true)." (Coffman and Niederle 2015:85)

"fourth parameter is u, the study bias, which is the probability with which a study that would have been reported false without any bias is instead reported positive (for any reason)" (Coffman and Niederle 2015:85)

"The final parameter is k, the number of substitute studies that were (or could be) investigated. To be precise, we assume that out of k possible investigations, only the first positive one is reported, and all others are either never investigated or simply never reported" (Coffman and Niederle 2015:85)

"[_ − β k(1 − ]_ π _____________ k = 1_ u)_ __________ Positive Predictive Value- − β ] π + (1 − α) − ] k k k k 1 (1 u) (1 (1 u)" (Coffman and Niederle 2015:86)

"A common criticism of pre-analysis plans is that they inhibit exploratory work (for example, Gelman 2013)." (Coffman and Niederle 2015:88)

"However, we also know that allowing empirical or fieldwork such degrees of freedom can produce high false positive incidence rates (as in Simmons, Nelson, and Simonsohn 2011)." (Coffman and Niederle 2015:88)

"First, we can allow the researcher to offer reasons in defense of the reasonableness of, say, add-on treatments, language changes, or a unique method for analyzing the data" (Coffman and Niederle 2015:88)

"Second, we can use robustness tests, in which important and/or surprising results should be replicated with a variety of modest alterations whenever possible" (Coffman and Niederle 2015:88)

"Miguel et al. (2014) rightly point out that pre-analysis plans can encourage exploratory work by lending credibility to surprising findings" (Coffman and Niederle 2015:89)

"When a hypothesis is registered, it does not necessarily lay out, or commit to, any specifics regarding data collection or method of analysis" (Coffman and Niederle 2015:89)

"Most prominently, hypothesis registries will help eliminate the file-drawer problem, in which null results are more likely to remain unpublished." (Coffman and Niederle 2015:89)

"Though a registry would not directly decrease the number of substitute studies, it would give us a better sense of the number of substitute studies run for a given class of hypotheses" (Coffman and Niederle 2015:90)

"First, many researchers would not feel comfortable sharing the details of their hypothesis and design before they have published their work" (Coffman and Niederle 2015:90)

"Second, if a listing in a hypothesis registry does not result in a published paper, it would not be clear why" (Coffman and Niederle 2015:91)

"Third, the hypotheses in the registry would not necessarily be organized in a helpful way, and, as with Google Scholar and other literature search tools now, navigating the registry for work related to a specific hypothesis would not be straightforward." (Coffman and Niederle 2015:91)

"Suppose a study finds a statistically significant result, and further suppose that the hypothesis is actually true. How much more confident do we become that the result is correct after one replication? Five replications?" (Coffman and Niederle 2015:92)

"First, researchers may be motivated (for a variety of noble and ignoble reasons) to prove or disprove a published result" (Coffman and Niederle 2015:93)

"Second, a failure to replicate a result can arise out of a poor test of the original hypothesis" (Coffman and Niederle 2015:93)

"A Journal of Replication Studies has three purposes. First, such a journal would offer an outlet for publication to meaningful, well-designed, and well-run replications" (Coffman and Niederle 2015:94)

"Second, the journal could perhaps signal what articles are higher priority for replication attempts." (Coffman and Niederle 2015:94)

"Third, the Journal of Replication Studies could also collect replications (failed or not) that exist within other original papers" (Coffman and Niederle 2015:95)