REPLICATION IN SOCIAL SCIENCE

() - Jeremy Freese, David Peterson

Journal:
Link::
DOI::
Links::
Tags:: #paper #Methods #OpenScience #General
Cite Key:: [@Freesea]

Abstract

Across the medical and social sciences, new discussions about replication have been transforming research practices. Sociologists, however, have been largely absent from these discussions. The goals of this review are to introduce sociologists to these developments, synthesize insights from science studies about replication in general, and detail the specific issues regarding replication that occur in sociology. The first half of the article argues that a sociologically sophisticated understanding of replication must take into account both the ways that replication rules and conventions evolved within an epistemic culture and how those cultures are shaped by specific research challenges. The second half outlines the four main dimensions of replicability in quantitative sociology- verifiability, robustness, repeatability, and generalizability- and discusses the specific ambiguities of interpretation that can arise in each. We conclude by advocating some commonsense changes to promote replication while acknowledging the epistemic diversity of our field.

Notes

“sociology’s two flagship journals have published 27 articles over their history with “replication” in the title, but only one in the last 25 years.” (Freese and Peterson, p. 4)

“Sociologically, that one’s work could be replicated provided a social control mechanism that discouraged shoddy or fraudulent work and was articulated as part of the norm of “organized skepticism” taken as key to the successful functioning of science (Merton 1973; Zuckerman 1977).” (Freese and Peterson, p. 5)

““The job of the social sciences is not to show that replication is futile or impossible, but to show how to pursue replicability in the face of its recalcitrance” (Collins 2016:78). For Collins, even as replication involves interpretive ambiguities and social negotiations, it remains “the only criterion [we have] of what is to count as a natural regularity (or social regularity)” (quoted in Ashmore 1989:138).” (Freese and Peterson, p. 7)

“(1) Verifiability involves taking the results of an original study as the object of inquiry and typically asks limited questions regarding whether the same results are obtained by doing the same analyses on the same data. (2) Tests of robustness conduct a reanalysis on the original data using alternative specifications to see if the target finding is merely the result of analytic decisions. (3) Tests of repeatability involve collecting new data to determine whether key result of a study can be observed by using the original procedures to collect new data. Finally, (4) for inquiries into generalizability, the original study provides a premise for other work that trying to evaluate whether similar findings may be consistently observed across different methods or settings” (Freese and Peterson, p. 11)

“For some, this might seem so pedestrian as to not really count as replication (Collins 1991); to others, it is “pure replication” (Brown et al. 2014)” (Freese and Peterson, p. 12)

“The convention in sociology has largely been to treat the availability of data for verification as an ethical matter to be handled by individual requests” (Freese and Peterson, p. 13)

“This work seeks to assess whether key results are consistently observed across alternative ways the original analysis could have been done.” (Freese and Peterson, p. 15)

“f course, replication involves more than reanalyzing old data. Concerns about a potential “replication crisis” in experimental psychology and biomedical research have focused on whether reportedly effects can be also observed by other scientists collecting new data” (Freese and Peterson, p. 17)

“Nevertheless, across quantitative social science discussion” (Freese and Peterson, p. 19)

“of replication, repeatability per se is noticeably de-emphasized relative to the re-examinations of original data described earlier, on the one hand, and inquiries directed to the generalizability of findings on the other (see, e.g., Wilson et al. 1973; Lucas et al. 2013).” (Freese and Peterson, p. 20)