What Improves with Increased Missing Data Imputations?

Key takeaways

Variability in estimates will increase with higher fractions of missingness ceteris paribus and decrease with increased imputations ceteris paribus.

Refer to Table 2 and 3 to understand number of imputations.

with missingness = 0.3 returns stop at table 2 at around 30 imputations. For missingness=0.9 >100 seems needed.

Using Bodner's 10% criteria looking at table 3 confirms for missingness = 0.3 then around 36 is ok. For missingness around 0.90 then m=258.

content: "@bodnerWhatImprovesIncreased2008" -file:@bodnerWhatImprovesIncreased2008

Reading notes

Annotations

(08/05/2024, 18:04:30)

“a prominent guideline suggests that more than 10 imputed data values are seldom needed. This article calls into question the optimism of this guideline and illustrates that important quantities (e.g., p values, confidence interval half-widths, and estimated fractions of missing information) suffer from substantial imprecision with a small number of imputations.” (Bodner, 2008, p. 651)

“a researcher can draw categorically different conclusions about null hypothesis rejection, estimation precision, and missing information in distinct multiple imputation runs for the same data and analysis with few imputations.” (Bodner, 2008, p. 651)

“Rubin (1987) illustrated that using between 2 and 10 imputations loses little estimator efficiency (relative to an infinite number of imputations) when the fraction of” (Bodner, 2008, p. 651)

“missing information (defined later) is modest and generates confidence intervals and hypothesis tests close to their nominal coverage and significance levels, respectively. However, when the fraction of missing information is very large, more than 10 imputations may be needed (Schafer, 1997a).1” (Bodner, 2008, p. 652)

“Hershberger and Fisher (2003), using results from sampling theory where the “true” number of imputations m is a quantity to be estimated, found that the needed m can often be surprisingly large (e.g., several hundred) for a specified level of precision. In response, von Hippel (2005) challenged the sampling theory approach, clarified Rubin’s earlier work on this topic, and was generally thankful for practical reasons that so many imputations were in fact not needed.” (Bodner, 2008, p. 652)

“he main point made and illustrated here is that important inferential quantities such as null hypothesis significance test (NHST) p values and confidence interval (CI) half-widths can exhibit substantial variability across independent multiple imputation runs with small m for the same data and analysis.” (Bodner, 2008, p. 652)

“CIs seek to make probabilistic statements about population parameter values from sample statistics. With complete sample data (i.e., data sets with no missing values), inferential procedures must account for uncertainty due to the sampling variability of these statistics. With incomplete sample data (i.e., data sets with missing values), inferential procedures must also account for uncertainty due to the unknown missing data values. The following procedures for complete and incomplete data are designed to accomplish these tasks.” (Bodner, 2008, p. 653)

“the question again becomes how large should m be? This section reviews two methods to determine m and discusses their relative merits. To overcome the stated limitations of these methods, a simulation experiment is conducted in the next section to help guide decisions for choosing m.” (Bodner, 2008, p. 662)

“Rubin’s (1987) suggestion that few imputations are needed was based on the efficiency of point estimates. The relative efficiency in using m rather than an infinite number of imputations is approximately .1 C œ=m/ 1. With œ D :19, estimation of ™ is 94% as efficient using m D 3 compared to m D 1.” (Bodner, 2008, p. 662)

“However, two issues merit further consideration. First, use of Rubin’s equation requires knowledge of œ, which is typically unknown. Furthermore, the results of the two examples suggest that estimation of œ can be imprecise with few imputations. Second, discussions of estimator efficiency involve the expected variance of the sample statistic.” (Bodner, 2008, p. 662)

“However, Rubin’s method does not address imputation variability and therefore is not useful to determine a value of m to minimize its effects.” (Bodner, 2008, p. 663)

“Royston (2004) provided an iterative method for choosing m based on a scaled coefficient of variation (CV D SD=M 100) of O hm. To use the method, one first generates w replicate sets of m imputations and then computes O hm for each set; next one computes the CV for these w values of O hm. If the CV is greater than some criterion (Royston suggested CV D 5%), one increases m and repeats the process until the CV is less than the criterion. The number of imputations m is the value of m when this criterion is met.” (Bodner, 2008, p. 663)

“First, Royston offered no guidance in the specification of w, an initial value of m, or the increase in m given failure to reach criterion, and does not discuss what to do if NHST p values or O œms are of interest.” (Bodner, 2008, p. 663)

“Second, Royston’s iterative method is odd for choosing m because it requires imputing the data substantially more than m times.” (Bodner, 2008, p. 663)

“A series of simulation experiments were conducted with the goal of providing researchers guidance for choosing a number of imputations to reduce to reasonable levels the effects of imputation variability on estimated 95% CI halfwidths, NHST p values, and fractions of missing information.” (Bodner, 2008, p. 663)

“In the first phase, the number of imputations and the true fraction of missing information were varied systematically to explore the impact of these factors on these estimated statistics. In the second phase, criteria were sought representing reasonable levels of imputation variance for these statistics along with the minimum values of m necessary to achieve these criteria at two levels of confidence.” (Bodner, 2008, p. 664)

“A sample size of n D 100 was chosen to reflect the median sample size reported in a recent survey of psychological research (Bodner et al., 2004) where n1 and n0 (n1Cn0 D n) values for Y are observed and missing, respectively. Two factors were varied, the number of imputations (i.e., ms D 3, 5, 10, 20, 30, 40, 50, 75, and 100) and the fraction of missing information (i.e., œs D :05, .1, .2, .3, .5, .7, and .9). Manipulation of œ in this univariate setting was achieved by setting n1 D n.1 œ/. Thus, there were 63 conditions in the simulation experiment.” (Bodner, 2008, p. 664)

“The asymptotic results of Harel (2006) demonstrate that variability of O œm around œ is maximal for œ D :5 and approaches zero for œ near 0 and 1. One can easily manipulate Harel’s (2006) formulas to find the value of m to estimate œ D :05 and œ D :5 with equal precision; in brief, a much larger m is needed for fixed precision to estimate the latter œ compared to the former œ.” (Bodner, 2008, p. 667)

“a difference of .1 between O œm and œ was sufficiently precise irrespective of œ and that greater precision is not typically of substantive interest.” (Bodner, 2008, p. 667)

“First, as one would expect, a larger number of imputations is needed to contain a larger proportion of the O hm and O œm values in the specified intervals (i.e., moving from 80% to 95% containment). Second, for O hm the minimum m increased approximately at an exponential rate with increasing œ; for O œm the minimum m was maximal for œ D :5 and decreased as œ differed from .5.8” (Bodner, 2008, p. 668)

“For relatively accurate estimates of CI half-widths and NHST p values, the needed number of imputations increases with œ. For relatively accurate estimates of the fraction of missing information, the needed number of imputations increases as œ approaches .50. If œ was known, researchers could use the results presented in Table 3 to choose a reasonable value for m for relatively accurate estimates of CI half-widths and fractions of missing information. Researchers focusing on NHST p values can reduce substantively important imputation variability in O pm with respect to null hypothesis rejection when N pm is not near ’ by choosing a value of m from Table 3 that controls the amount of imputation variability in O hm at a specified level of confidence.” (Bodner, 2008, p. 670)

“However, because œ is seldom known with incomplete multivariate data, one must estimate its value to use this table.” (Bodner, 2008, p. 670)

“n lieu of a very precise estimate of œ with which one can use Table 3 to choose m to bring imputation variability in O hm and O pm to reasonable levels, the following conservative approach is suggested. First, compute a conservative estimate of œ based on one minus the ratio of the number of cases available for the estimation of a given parameter using listwise deletion (e.g., nL) to the total sample size (i.e., O œL D 1 nL=n)” (Bodner, 2008, p. 670)

“Next, consult Table 3 to choose m to minimize imputation variability in O hm to the desired confidence level using linear interpolation of the needed m between œ values as necessary and rounding to an integer value. The analysis is then conducted with the computed number of imputations.” (Bodner, 2008, p. 671)

“illustrate the method, consider the data used in Example 1 where nL D 19 of the n D 28 patients had observed cholesterol measurements 14 days after myocardial infarction. Therefore, O œL D 1 19=28 D :32 for inference for •14. Assuming the desired confidence is 95% that O hm is within 10% of N hm, then m should be between 24 (for œ D :30) and 59 (for œ D :50) from Table 3. Linear interpolation of these ms yields” (Bodner, 2008, p. 671)

“When œ is nontrivial and m is small (e.g., œ > :2 and m 10), important quantities of interest (e.g., O œm, O hm, and O pm) can suffer from considerable imputation variability.” (Bodner, 2008, p. 672)

“In many cases increasing m beyond 10 does not take much additional computer time or researcher effort given the prevalence of high-speed computing and automated routines for combining results for multiple imputation inference (e.g., PROC MIANALYZE in SAS).” (Bodner, 2008, p. 672)