Open science challenges, benefits and tips in early career and beyond

(2019) - Christopher Allen, David M. A. Mehler

Journal: PLOS Biology
Link:: https://dx.plos.org/10.1371/journal.pbio.3000246
DOI:: 10.1371/journal.pbio.3000246
Links::
Tags:: #paper #Pre-Analysis
Cite Key:: [@allenOpenScienceChallenges2019]

Abstract

The movement towards open science is a consequence of seemingly pervasive failures to replicate previous research. This transition comes with great benefits but also significant challenges that are likely to affect those who carry out the research, usually early career researchers (ECRs). Here, we describe key benefits, including reputational gains, increased chances of publication, and a broader increase in the reliability of research. The increased chances of publication are supported by exploratory analyses indicating null findings are substantially more likely to be published via open registered reports in comparison to more conventional methods. These benefits are balanced by challenges that we have encountered and that involve increased costs in terms of flexibility, time, and issues with the current incentive structure, all of which seem to affect ECRs acutely. Although there are major obstacles to the early adoption of open science, overall open science practices should benefit both the ECR and improve the quality of research. We review 3 benefits and 3 challenges and provide suggestions from the perspective of ECRs for moving towards open science practices, which we believe scientists and institutions at all levels would do well to consider.

Notes

“In the case of RRs, hypotheses and methods are peer reviewed on the basis of scientific validity, statistical power, and interest, and RRs can receive in principal acceptance for publication before data is collected [29,30]. RRs can thereby increase the chances of publishing null findings” (Allen and Mehler, 2019, p. 2)

“Preregistering hypotheses and methods renders so-called hypothesizing after the results are known (HARKing) [31] impossible and prevents manipulation of researcher degrees of freedom or p” (Allen and Mehler, 2019, p. 2)

“Once data collection has started, new learning about analysis techniques, subsequent publications, and exploration of patterns in data cannot inform confirmatory hypotheses or the preregistered experimental desig” (Allen and Mehler, 2019, p. 2)

“Table 1. Open science practices. Some methods introduced or suggested by the open science community to improve” (Allen and Mehler, 2019, p. 2)

“Continuous learning during the course of an investigation is difficult to reconcile with a hard distinction between confirmatory and exploratory research but may be the price of unbiased science [4,11]. Open” (Allen and Mehler, 2019, p. 3)

“We therefore support the view that mistakes should be handled openly, constructively, and, perhaps most importantly, in a positive nondetrimental way [38,39]. Mistakes can and will happen, but by encouraging researchers to be open about them and not reprimanding others for them, open science can counter incentives to hide mistakes.” (Allen and Mehler, 2019, p. 3)

“When developing full a priori analyses, pipelines anticipating all potential outcomes and contingencies should be attempted” (Allen and Mehler, 2019, p. 3)

“The additional requirements of open and reproducible sciences often consume more time: Archiving, documenting, and quality controlling of code and data takes time.” (Allen and Mehler, 2019, p. 4)

“Systems that reward open science practices are currently rare, and researchers are primarily assessed according to traditional standards” (Allen and Mehler, 2019, p. 4)