Summary

Establishment surveys differ from household surveys. This fact is reflected in the different culture of questionnaire development, evaluation and testing. First of all, the response process is more complex than in household surveys. The extensive adaptation of cognitive methods in social questionnaire development and testing based on Tourangeau's (1984) response process model, in the case of establishments, had to be enhanced by including new dimensions. The four step model, consisting of comprehension, retrieval, judgment and communication focuses on the individual. In the context of establishment surveys, however, the respondent is an informant selected within an organisation. Besides cognition, testing must also take into account the institutional frame, the need for cooperation, and the fact that required data items may be stored in business records. The non-existence of data in business records has to be taken into account as well.

Another consideration in the field of questionnaire testing is the specific nature of business surveys with their technical and intricate concepts and definitions; therefore, comprehension depends considerably on instructions. Closely connected with that and having its own consequences is the predominately self-administered data collection mode. There are other distinguishing features of the establishment population which pose a challenge for testing procedures. Among those are the longitudinal character of business surveys and the subsequent use of resulting data as inputs for other surveys. Yet another problem is a negative attitude to any changes in the questionnaires.

Nonetheless, the need for testing is beyond contention. This requirement is stated clearly in the Eurostat Code of Practice: "In the case of statistical surveys, questionnaires are systematically tested prior to the data collection". Ongoing data collection instruments are also under scrutiny. The goals to achieve include the improvement of the quality of statistical output, the reduction of costs to the surveying agency and to respondents, a decrease in the scope of output variables, an increase in the use of administrative data (Giesen, 2005). The adoption of Computer Assisted Interviewing has prompted redesign efforts to explore new prospects in data collection. This has added a new level of complexity to questionnaire testing. One new dimension is usability testing, which is intended to assess if the testing tool is user friendly and whether the interaction with the computer is intuitive and simple for the respondent. The optimal approach to efficient testing requires the involvement of end users. This leads to a paradox: in an effort to improve the collection instrument and ease the response burden, another burden is imposed on respondents (Willimack, 2005). Adding more burden during the response process, which in itself is burdensome, can hardly meet with the respondent's approval. It is, therefore, important that respondents should be aware of the goals of the testing procedure, which is intended to simplify and ease the response. When the aims of the procedure are clear, additional efforts can be received with a higher degree of approval. On the other hand, the iterative and longitudinal character of business surveys makes it possible to work out a systematic approach to improving the data collection instrument in a step-by-step procedure, which involves incorporating the testing and developing research into ongoing and repeated surveys. Instead of the usual practice of relying on post-collection activities to correct errors, a new paradigm is proposed, encouraging research on improving questionnaire design that leads to "error prevention rather than error correction" (Willimack et al., 2004).

A broad spectrum of recommended practices for developing and testing statistic questionnaires can be found in the Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System (2006). The Handbook adds valuable enhancement to the general subject of developing and testing questionnaires in statistical surveys. A detailed discussion of testing and evaluation questionnaires for establishment surveys can be found in Willimack (2013).

To read the entire document, please access the pdf file (link under "Related Documents" on the right-hand-side of this page).

Your feedback is appreciated. Please send your remarks, suggestions for improvement, etc. to memobust@cbs.nl.