Using Heterogeneous Choice Models to Compare Logit and Probit Coefficients Across Groups

(2009) - Richard Williams

Journal: Sociological Methods & Research
Link:: http://journals.sagepub.com/doi/10.1177/0049124109335735
DOI:: 10.1177/0049124109335735
Links::
Tags:: #paper #Methods #Probit #Logit
Cite Key:: [@williamsUsingHeterogeneousChoice2009]

Abstract

Allison (1999) notes that comparisons of logit and probit coefficients across groups can be invalid and misleading, proposes a procedure by which these problems can be corrected, and argues that “routine use [of this method] seems advisable” and that “it is hard to see how [the method] can be improved.” In this article, the author argues that as originally proposed, Allison's method can have serious problems and should not be applied on a routine basis. However, this study also shows that his model belongs to a larger class of models variously known as heterogeneous choice or location-scale models. Several advantages of this broader and more flexible class of models are illustrated. Dependent variables can be ordinal in addition to binary, sources of heterogeneity can be better modeled and controlled for, and insights can be gained into the effects of group characteristics on outcomes that would be missed by other methods.

Notes

“Allison (1999) notes that comparisons of logit and probit coefficients across groups can be invalid and misleading” (Williams, 2009, p. 531)

“Allison’s method can have serious problems and should not be applied on a routine basis” (Williams, 2009, p. 531)

“Unlike linear regression coefficients, coefficients in binary regression models are confounded with residual variation (unobserved heterogeneity)” (Williams, 2009, p. 532)

“Differences in the degree of residual variation across groups can produce apparent differences in slope coefficients that are not indicative of true differences.” (Williams, 2009, p. 532)

“n this article, I argue that heterogeneous choice (also known as locationscale) models provide a superior means for dealing with the problems Allison (1999) presents. I show that Allison’s solution actually involves a special case of these models, the heteroscedastic logit model. While this more limited method works well in some situations, in other cases, it can produce biased and inefficient estimates and can lead researchers to either overstate or understate the statistical and substantive significance of the differences that are found.” (Williams, 2009, p. 532)

“In assessing his proposed solution, it is useful to realize that his approach involves (but is not limited to) a reparameterization of the heteroscedastic logit model” (Williams, 2009, p. 538)

“The heteroscedastic logit model,inturn,ispartofalargerclassofmodelsthatisvariously known as location-scale models (McCullagh and Nelder 1989) and heterogeneous choice models (Alvarez and Brehm 1995; Keele and Park 2006).” (Williams, 2009, p. 538)

“With heterogeneous choice models, the dependent variable can be ordinal or binary.” (Williams, 2009, p. 538)

“Allison’s (1999) approach has some obvious limitations. With Allison’s method, the dependent variable can be only binary, not ordinal.” (Williams, 2009, p. 540)

“Hoetker (2004) did a series of simulations in which he examined the problems raised by Allison (1999) and how well Allison’s method addressed them. He found that ‘‘in the presence of even fairly small differences in residual variation, naive comparisons of coefficients can indicate differences where none exist, hide differences that do exist, and even show differences in the opposite direction of what actually exists’’ (Hoetker 2004:17). At least in the simulations he ran, he found that Allison’s method accurately detected differences in residual variation and false differences in coefficients and that it also accurately detected true differences in coefficients.” (Williams, 2009, p. 540)

“In short, the problem is not just that Allison’s (1999) methods are ‘‘conservative,’’ as both he and Hoetker (2004) claim. In some plausible situations, the tests appear to be conservative not because of a lack of statistical power but because the parameter estimates are biased downward.” (Williams, 2009, p. 547)

“presents a series of models for these data, all estimated with the oglm (Williams 2006b) routine in Stata” (Williams, 2009, p. 548)

“A key assumption of the model is that while the thresholds differ across values of j, the bs do not. This is referred to as the parallel lines assumption. One of the key advantages of the ordered logit model is that there are well-established tests for whether the parallel lines assumption is violated, and as Long and Freese (2006) point out, if the parallel lines assumption is violated, alternative methods for ordinal regression should be considered. Both Long and Freese and Williams (2006a) find that the assumptions of the ordered logit model are indeed violated with these data. In particular, a Brant test (Brant 1990; Long and Freese 2006) reveals that the variables yr89 and male do not meet the parallel lines assumption. While this in and of itself does not necessarily mean that a heterogeneous choice model is called for, oglm’s stepwise selection procedure also identifies yr89 and male as statistically significant variables for inclusion in the variance equation. This implies that residual variability in attitudes toward working mothers differed by year and by gender, both of which are substantively plausible” (Williams, 2009, p. 551)

“Even when coefficients do not differ across groups, as in our example, heterogeneous choice models can yield insights into the effects of group characteristics that would be overlooked in misspecified models. That is, the estimated effects of group characteristics (as well as other variables) can differ once heterogeneity is taken into account.” (Williams, 2009, p. 553)

“There is no need to limit the variance equation to a single dichotomous grouping variable.” (Williams, 2009, p. 554)

“Note further that while I have primarily focused on group differences in residual variances, group differences are only one possible source of heteroscedasticity” (Williams, 2009, p. 554)

“The variance may itself be of substantive interest. The variance equation makes it possible to examine the determinants of variability.” (Williams, 2009, p. 554)

“Heteroscedastic logit and probit models work only with dichotomous dependent variables. Heterogeneous choice models also allow for ordinal dependent variables.” (Williams, 2009, p. 554)

“As Keele and Park (2006) note, ordinal variables contain more information and models using them are much less prone to problems than are models with dichotomous dependent variables.” (Williams, 2009, p. 555)

“There are well-established diagnostic procedures that can indicate when the assumptions of the ordered logit model are violated.” (Williams, 2009, p. 555)

“with ordinal variables (that have three or more categories), it is not necessary to make the questionable assumption that at least one coefficient is the same across groups; the multiple cutoff points make it possible to identify the model and allow coefficients to differ across groups.” (Williams, 2009, p. 555)

“It is, however, necessary to make the assumption that the cutoff points are the same for both groups” (Williams, 2009, p. 555)

“The specialized programs that Allison wrote are no longer necessary because today major software packages include routines for estimating heterogeneous choice models.” (Williams, 2009, p. 555)