Preregistration of Analyses of Preexisting Data

(2019) - Gaëtan Mertens, Angelos-Miltiadis Krypotos

Journal: Psychologica Belgica
Link:: http://www.psychologicabelgica.com/articles/10.5334/pb.493/
DOI:: 10.5334/pb.493
Links::
Tags:: #paper #Pre-Analysis
Cite Key:: [@Mertens2019]

Abstract

The preregistration of a study’s hypotheses, methods, and data-analyses steps is becoming a popular psychological research practice. To date, most of the discussion on study preregistration has focused on the preregistration of studies that include the collection of original data. However, much of the research in psychology relies on the (re-)analysis of preexisting data. Importantly, this type of studies is different from original studies as researchers cannot change major aspects of the study (e.g., experimental manipulations, sample size). Here, we provide arguments as to why it is useful to preregister analyses of preexisting data, discuss practical considerations, consider potential concerns, and introduce a preregistration template tailored for studies focused on the analyses of preexisting data. We argue that the preregistration of hypotheses and data-analyses for analyses of preexisting data is an important step towards more transparent psychological research.

Notes

"Several reasons for the limited replicability of findings in the psychology literature have been suggested, including the use of underpowered studies (Bakker, van" (Mertens and Krypotos 2019:338)

"Dijk, & Wicherts, 2012; Schimmack, 2012), inadequate inferences from statistical tests (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), a lack of a unified theoretical framework (Muthukrishna & Henrich, 2019), and problematic incentives (Lilienfeld, 2017)." (Mertens and Krypotos 2019:339)

"Preregistration of research plans can help to (partly) prevent these problematic research practices (Asendorpf et al., 2013; Munafò et al., 2017; Nosek et al., 2015; van 't Veer & Giner-Sorolla, 2016; Weston, Ritchie, Rohrer, & Przybylski, 2019)." (Mertens and Krypotos 2019:339)

"Preregistration refers to the specification of a study's hypotheses, methodology, and statistical analyses before inspecting the research data. Preregistration takes typically the form of a document that is made publicly available on a timestamped repository or website" (Mertens and Krypotos 2019:339)

"may also lead to higher chances of acceptance by a journal, as the authors can prove that all their analytical plans were determined a priori, protecting the authors from rejection based on negative findings (Allen & Mehler, 2019)." (Mertens and Krypotos 2019:339)

"Empirical research can broadly entail two different epistemic goals: exploration and confirmation (De Groot, 1969). Particularly, exploratory research is often hypothesis generating and curiosity driven: new ideas and theories can develop on the basis of collected data and/or exploratory data analyses. Confirmatory research, on the other hand, involves testing specific predictions (hypotheses) derived from theories." (Mertens and Krypotos 2019:340)

"However, a problematic practice in which researchers often engage is to first explore the data and then formulate hypotheses that correspond with the obtained results, without clarifying this order or even actively distorting the order of operations (Kerr, 1998; Simmons et al., 2011). This practice invalidates the commonly used statistical procedures in the null-hypothesis testing framework, because the generation of hypotheses and testing of hypotheses are no longer independent (i.e., the formulation of the hypothesis and the 'test' of the hypothesis are based on exactly the same data), inflating the number of false-positive results (De Groot, 2014; Simmons et al., 2011; Wagenmakers, Wetzels, Borsboom, van der" (Mertens and Krypotos 2019:340)

"Maas, & Kievit, 2012)." (Mertens and Krypotos 2019:340)

"a well-known problem that studies that report significant results have higher chances of getting accepted for publication by a scientific journal, compared to studies that report non-significant results (Coursol & Wagner, 1986; Levine, Asada, & Carpenter, 2009; Rosenthal, 1979)." (Mertens and Krypotos 2019:341)

"Data analyses usually entails many different steps (cleaning of the data, selecting a statistical model, selecting variables and covariates) in which researchers have to make" (Mertens and Krypotos 2019:341)

"decisions. These decisions (also known as researchers' degrees of freedom; Simmons et al., 2011) can influence the final results" (Mertens and Krypotos 2019:342)

"Therefore, timestamps included in the data files cannot be used to determine whether the data-analysis plans were specified before analyzing the data. This is an important issue because without timestamping, researchers can easily antedate their data-analysis plans while, in fact, data-analysis were not conducted independent of data-evaluation." (Mertens and Krypotos 2019:342)

"First, even if preexisting data are readily available, they may lack proper documentation that could allow researchers to easily understand the structure of the data" (Mertens and Krypotos 2019:344)

"Second, researchers should be aware that preregistration does not automatically mean that the preregistered analytical framework was appropriate and the results cannot be refuted" (Mertens and Krypotos 2019:344)

"Third, researchers may object that preregistration of analyses of preexisting data require a substantial time investment, interfering with the researchers' limited research time, and that it hampers the rapid development of science." (Mertens and Krypotos 2019:344)

"Table 1: Template questions for the preregistration of analyses of preexisting data." (Mertens and Krypotos 2019:345)