Limitations of Fixed-Effects Models for Panel Data

(2020) - Terrence D. Hill, Andrew P. Davis, J. Micah Roos, Michael T. French

Journal: Sociological Perspectives
Link:: http://journals.sagepub.com/doi/10.1177/0731121419863785
DOI:: 10.1177/0731121419863785
Links::
Tags:: #paper #Methods #FixedEffects
Cite Key:: [@hillLimitationsFixedEffectsModels2020]

Abstract

Although fixed-effects models for panel data are now widely recognized as powerful tools for longitudinal data analysis, the limitations of these models are not well known. We provide a critical discussion of 12 limitations, including a culture of omission, low statistical power, limited external validity, restricted time periods, measurement error, time invariance, undefined variables, unobserved heterogeneity, erroneous causal inferences, imprecise interpretations of coefficients, imprudent comparisons with cross-sectional models, and questionable contributions vis-à-vis previous work. Instead of discouraging the use of fixed-effects models, we encourage more critical applications of this rigorous and promising methodology. The most important deficiencies—Type II errors, biased coefficients and imprecise standard errors, misleading p values, misguided causal claims, and various theoretical concerns—should be weighed against the likely presence of unobserved heterogeneity in other regression models. Ultimately, we must do a better job of communicating the pitfalls of fixed-effects models to our colleagues and students.

Notes

“We provide a critical discussion of 12 limitations, including a culture of omission, low statistical power, limited external validity, restricted time periods, measurement error, time invariance, undefined variables, unobserved heterogeneity, erroneous causal inferences, imprecise interpretations of coefficients, imprudent comparisons with cross-sectional models, and questionable contributions vis-à-vis previous work. Instead of discouraging the use of fixed-effects models, we encourage more critical applications of this rigorous and promising methodology.” (Hill et al., 2020, p. 357)

“Omitted variable bias is the primary statistical challenge in nonexperimental research (Allison 2009; DeMaris 2014; Wooldridge 2010).” (Hill et al., 2020, p. 357)

“Fixed-effects models for panel data were developed to address the issue of omitted variable bias in nonexperimental research (Allison 2009; Fox 2016; Treiman 2009; Wooldridge 2010).” (Hill et al., 2020, p. 358)

“A central limitation is that researchers generally fail to offer any serious discussion of the shortcomings of fixed-effects models for panel data” (Hill et al., 2020, p. 361)

“Because fixed-effects estimates are based on characteristics that change over time, variables and cases that do not change either (1) do not contribute much information to the analysis or (2) are altogether omitted by design. When the analytic sample is essentially limited to variables and cases that change, sample size is reduced, variation across cases is constrained, and statistical power is a cause for concern (Beck and Katz 2001; Longhi and Nandi 2014; Treiman 2009).” (Hill et al., 2020, p. 361)

“The primary reason for this predicament is statistical power. According to Paul Allison (2009:9), “standard errors for fixed effects coefficients are often larger than those for other methods, especially when the predictor variable has little variation over time.”” (Hill et al., 2020, p. 362)

“When variables and cases that do not change over time are omitted by design, the analytic sample is altered in substantive ways. Treiman (2009:378) argues that when you only model changing units, the findings apply to a “selected subgroup only.” For example, Allison (2009:3) notes that “those whose education level does change over the adult years may be quite unlike those whose education level does not change.”” (Hill et al., 2020, p. 362)

“Nevertheless, Jeffrey Wooldridge (2010:326) explains that In cases where the key variables do not vary much over time, FE [fixed effects] and FD [first difference] methods can lead to imprecise estimates. We may be forced to use random effects estimation in order to learn anything about the population parameters.” (Hill et al., 2020, p. 362)

“Fixed-effects coefficients are also less reliable when the number of time periods is limited. Stephen Nickell (1981:1418) explains that A typical set of panel data has a rather large number of individuals and a rather small number of time periods and it is in just these circumstances that the biases, which are essentially of the Hurwicz type, are most serious.” (Hill et al., 2020, p. 362)

“Systematic measurement error is another concern (Angriest and Pischke 2009; Hsiao 1985). The problem is that measurement errors like misreporting and miscoding are compounded if they are systematic and repeated across time.” (Hill et al., 2020, p. 363)

“Allison (2009:2) notes that “fixed effects methods are pretty much useless for estimating the effects of variables that don’t change over time . . .”” (Hill et al., 2020, p. 363)

“The unobserved heterogeneity that may be present in fixed-effects models is a black box. According to Angriest and Pischke (2009:243), “a strike against the fixed effects setup is the fact that the exact nature of the unobserved variables typically remains mysterious.” Andrew Bell and Kelvyn Jones (2015:134) note that fixed-effects “models that control out, rather than explicitly model, context and heterogeneity offer overly simplistic and impoverished results that can lead to misleading interpretations.”” (Hill et al., 2020, p. 363)

“Although unobserved heterogeneity due to time-varying characteristics that are unmeasured or unknown to the analyst is the primary limitation of fixed-effects models, researchers tend to focus more on the heterogeneity that is controlled by design than to the heterogeneity that is not.” (Hill et al., 2020, p. 363)

“Although many researchers employ fixed-effects models with panel data to generate causal inferences, causal assumptions are rarely addressed (Imai and Kim 2017; Sobel 2012).” (Hill et al., 2020, p. 364)

“Previous work (Angriest; French and Popovici 2011; Imai and Kim 2017; Treiman 2009; Vaisey and Miles 2017; Wooldridge 2010) has emphasized three key causal identification assumptions: (1) no unobserved time-varying confounders (classic unobserved heterogeneity), (2) past outcomes do not directly affect the explanatory variables (reverse causality), and (3) past explanatory variables do not directly affect current outcomes (lagged treatments).” (Hill et al., 2020, p. 364)

“Fixed-effects coefficients are often imprecisely interpreted or in some cases misinterpreted. Interpretations are also frequently incompatible with the chosen theory or research question. First and foremost, interpretation depends on the type of fixed-effects model (one-way or two-way).” (Hill et al., 2020, p. 364)

“Although it is common to compare fixed-effects coefficients with standard cross-sectional coefficients to assess the degree of omitted variable bias in cross-sectional models or as a robustness check, we find these comparisons worrisome or in some ways nonsensical. Fixed-effects models and cross-sectional models are based on alternative estimation techniques and different samples.” (Hill et al., 2020, p. 365)

“The diffusion of fixed-effects methods has created a new research paradigm. Researchers now use fixed-effects techniques to assess the validity or robustness of seminal cross-sectional studies. These papers often begin with vague claims of omitted variable bias in previous cross-sectional research. They then frame the fixed-effects model as a more rigorous test. Fixed-effects estimates are directly “compared” with cross-sectional estimates coefficient-by-coefficient.” (Hill et al., 2020, p. 366)

“All researchers, including those who employ fixed-effects models for panel data, should provide serious and honest discussions of the limitations of their methodology.” (Hill et al., 2020, p. 366)

“Researchers should pay more attention to how their sample size is reduced in fixed-effects models.” (Hill et al., 2020, p. 366)

“Researchers should be cautious and critical about any substantive changes in their analytic sample.” (Hill et al., 2020, p. 366)

“When researchers employ a small number of time periods, they should seriously consider the potential for biased coefficients.” (Hill et al., 2020, p. 366)

“Researchers should pay more attention to issues related to data quality and measurement error.” (Hill et al., 2020, p. 367)

“Researchers should be forthcoming about whether they are truly interested in the effects of variables that do not change over time.” (Hill et al., 2020, p. 367)

“Researchers should be more discriminating and critical about the variables that are included in their models and those that are likely omitted by design.” (Hill et al., 2020, p. 367)

“Researchers should directly acknowledge and discuss the possibility of important time-varying variables that are missing from the fixed-effects specifications.” (Hill et al., 2020, p. 367)

“Researchers should be careful when making any claims about causality. Fixed-effects models are not direct substitutes for natural or controlled experiments.” (Hill et al., 2020, p. 367)

“Researchers should be unambiguous and precise when interpreting their fixed-effects coefficients” (Hill et al., 2020, p. 367)

“Researchers should avoid directly comparing fixed-effects coefficients with coefficients derived from cross-sectional methods.” (Hill et al., 2020, p. 367)

“When previous cross-sectional findings are not replicated, the implications of the fixed-effects analysis should be contingent upon the veracity of the estimates and the presentation of a compelling theory that specifies the likely candidates and implications for unobserved heterogeneity.” (Hill et al., 2020, p. 367)