Annex 2 - Comments on the paper of Anders Christianson “Avoid the Need to Impute!” By Graham Kalton

I think that Anders has raised a good topic fro discussion in The Survey Statistician. I should like to broaden it to the following question: Should survey estimates that fall below an “acceptable” level of accuracy be suppressed? Anders focuses on high nonresponse rates as an indicator of accuracy, whereas I am extending the issue to all dimensions of accuracy. There are clearly differences in views and practices as to whether estimates of low accuracy should be suppressed, differences that I suspect are related to the different circumstances and types of surveys involved. For this reason I do not think there is a simple universal answer to the question. In general I am opposed to suppression, but there are circumstances where it can be justified.

In essence, my argument is that estimates of any quality add to the information available to users. Rather than protecting users from themselves by suppressing estimates that producers judge to be of inadequate quality, I believe that producers should provide users with the estimates together with the information that is needed for users to assess the accuracy for themselves. It is for the users to decide whether the estimates are fit for use for their particular purpose. In doing so, users should of course consider alternative sources of information that may contribute to meeting their needs, and make use of the current estimates in the light of that information. If good alternative data are available, a user may choose to ignore the current inaccurate estimates, suppressing them for him – or herself. However, if there are no other reasonably closely related alternative data available, then the user may prefer to rely on the inaccurate estimates, rather than operating without statistical data. Users who need to make policy decisions or allocate funds based on statistical evidence must take some action. Even though the estimates may be highly inaccurate, they may nevertheless provide the best information available (see example in Kalton 2001).

My argument relies, of course, on users being able to make informed judgments. There are two aspects to this. First, users need to have access to the information necessary to assess the accuracy of estimates. Producers need to provide that information to users, after carrying out whatever studies are needed to compile the information. Second, users need to have the skills necessary to make an informed judgment. The case is often made that users lack those skills and will misuse the estimates if they are not suppressed. I acknowledge that this may often be the case, but I think that the solution lies in extensive efforts to educate users to understand measures of accuracy and to be concerned about them.

One widely used form of suppression is to suppress estimates with coefficients of variation (CVs) of greater than, say, 30 percent. Here, the level of inaccuracy (i.e, the standard error) can be measured and presented straightforwardly to the user. Thus, the case for suppression appears to be the concern that users will ignore the standard error. They need to be educated not to do so.

The situation with high total or item nonresponse rates is different in that these rates serve only as indicators of a potential for inaccuracy (i.e., non response bias), but no direct measure of inaccuracy can be provided. Furthermore, attempts can be made to reduce nonresponse bias by means of weighting adjustments for total nonresponse and imputation for item nonresponse. The effectiveness of these compensation procedures cannot be assessed. My general view in this situation is that the survey data (in the case of a high total nonresponse rate) or a particular item (in the case of a high overall item nonresposne rate) should not be suppressed, but that users should be given clear warnings about the risk of bias in the survey estimates.

To the extent that nonresponse bias studies can be conducted, they should be, and the results presented to users. Also, I argue that considerable efforts should be made to compensate as effectively as possible for high levels of missing data, either in weighting adjustments or through imputation. It was in this context that I referred in the ISI session to the paper by Wayne Fuller, Marie Loughlin and Harold Baker (1994), in which the authors applied extensive regression weighting adjustments to try to compensate for potential nonresponse bias in a survey that achieved a response rate of only 37 percent.

Another issue with suppression in the case of nonresponse is determining the threshold for the response rate below which the survey data are to be suppressed. Anders reports that the threshold was set at 83 percent in the minimum performance standards established for the Swedish TV surveys in 1973. If that threshold were applied now, the data from most surveys in most countries would be suppressed! How should the threshold be set in a meaningful way? I do not know how to answer that question. Furthermore, I think that the likely effectiveness of the nonresponse compensation procedures needs to be factored into a decision about suppression. In the case of item nonresponse, for example, if highly predictive auxiliary variables are used in imputing for an item with a high nonresponse rate, then the case for suppression is much reduced.

Having given my arguments against suppression on the grounds of high nonresponse, let me consider those for suppression. Anders suggests that one is that the threat of suppression makes the data collectors aware of the importance of high response rates, and hence keeps them on their toes. I can see that suppression may be effective for this purpose in surveys repeated at short intervals, but this would not seem to readily generalize. Also, I feel that there should be other methods for achieving this objective that do not involve jettisoning data.

A second argument for suppression relates mainly to surveys conducted by national statistical offices (NSOs). The concern here is that an NSO should not provide estimates of low accuracy because such estimates will tarnish the reputation of all the estimates the NSO provides. Given the great importance of maintaining the credibility of the information that an NSO provides (see Fellegi, 2001), this is an important point. However, note that it rests on an assumption that users fail to distinguish between estimates of low and high accuracy. If they were educated to do so, that should overcome this concern. Many users may not be able to make the distinction at this time, but the aim for the future should be to educate them to do so.

I should note that the above comments relate to suppression of estimates of low accuracy only. Suppression is also used to avoid data disclosure. That is a separate topic.

Finally, I should like to add my support to the position taken by Anders and Eric Rancourt that the best solution to missing data is to avoid the problem. The naïve view that nonresponse weighting adjustments and imputation, however well performed, can remedy the problem is fallacious and needs to be firmly refuted. In general, we should strive for accuracy along all dimensions. In addition, we should devote resources to measuring accuracy and to reporting it fully to users. Then we need to work towards educating users to understand and to take account of the accuracy measures we provide. If users fully absorb the importance of accuracy, they will also appreciate the value of designing high-quality surveys and will accept the often increased costs needed to produce a quality product.

SOURCE: The Survey Statistician, July 2002