Testing the Accuracy, Usefulness, and Significance of Probabilistic Choice Models: An Information-Theoretic Approach
Testing the Accuracy, Usefulness, and Significance of Probabilistic Choice Models: An Information-Theoretic Approach
Key takeaways
Bibliography: Hauser, J.R., 1978. Testing the Accuracy, Usefulness, and Significance of Probabilistic Choice Models: An Information-Theoretic Approach. Operations Research 26, 406–421. https://doi.org/10.1287/opre.26.3.406
Authors:: John R. Hauser
Collections:: Methods
First-page: 407
Disaggregate demand models predict the choice behavior of individual consumers. But while such models predict choice probabilities (0 < p < 1), they must be tested against (0, 1) choice behavior. This paper uses information theory to derive three complementary tests that help analysts select a “best” disaggregate model. “Usefulness” measures the percentage of uncertainty (entropy) explained by the information the model provides. It provides theoretic rigor and intuitive appeal to the commonly used likelihood ratio index and leads to important practical extensions. “Accuracy” is a new two-tailed normal test that determines whether the (0, 1) observations are reasonable under the hypothesis that the model is valid. “Significance” is the standard chi-squared test to determine whether a null model can be rejected. This paper also extends the information test to examine the relationships among successively more powerful null hypotheses. For example, in a logit model one can quantify (1) the contribution due to knowing aggregate market shares, (2) the incremental contribution due to knowing choice set restrictions, and (3) the final incremental contribution due to the explanatory variables. Further extensions provide “explanable uncertainty” measures applicable if choice frequencies are observed. Market research and transportation analysis empirical examples are given.
content: "@hauserTestingAccuracyUsefulness1978" -file:@hauserTestingAccuracyUsefulness1978
Reading notes
Imported on 2024-05-06 13:39
⭐ Important
- & In a given instance individual i either rides, drives, walks, or stays put. Suppose a model predicts that i will ride the bus with probability 0.7 and i does ride the bus. To assess the validity of such a model, a test must quantify how much "rightness" or "wrongness" there was in the prediction. Furthermore, suppose a model makes individual predictions, but for 1,000 individuals. Analysts need a test to measure that model's predictive ability and to select a "best" model. (p. 407)
- & The chi-squared test can reject a null model, but it cannot give an indication of how well a model predicts nor can it compare two models unless one model is a restriction of the other (p. 408)
- & most common disaggregate test used to measure a model's predictive ability is the likelihood ratio index [19]. This test, p2 = 1-L(X)/Lo where L(X) is the log-likelihood of the tested model (explanatory variables X) and Lo is the loglikelihood of the null model, acts like a pseudo-R2 since p2 =0 when L(X) =Lo and p2 =1 when the model predicts perfectly, otherwise 0< p (p. 408)
- What??:
- & ests. Disaggregate tests are not used alone because they are theoretically sensitive to the problem that limpij<o [log p j] = (p. 409)
- & The probability model can be viewed as an information system. In other words, the "observable occurrence," e.g., the attributes of the choice alternatives, provides information about "unobservable events," i.e., about the choice outcome (p. 410)
- & The accuracy of the model can be calculated by comparing the empirical information, I(A; X), with the expected informatio (p. 411)
- & THEOREM 1. The entropy of a system is numerically equal to the information that would be observed, given perfect knowledge, i.e., H(A) = I(A; perfect knowledg (p. 411)