Dillman, 2007, Effective Survey Design

Dillman’s (2007) presentation slides and exercise related to effective survey design make an excellent point about the importance of survey design and wording of questions for gathering data and conducting useful research. The presentation offers some very tangible illustrations of what can go wrong, and how results can vary as a result of survey question design/wording issues. For example, in two slides on page 4, Dillman demonstrates the difference of opinions expressed about divorce when the question is presented in an open-ended structure and then when it is presented in a close-ended structure (both ordered and “out of order”). The rates of opinion vary so considerably that they greatly call into question the validity of the results.

The good news is that the presentation slides provide some points of consideration to help researchers avoid such pitfalls. Dillman suggests that researchers consider a number of factors or questions when designing a survey. Overall, one is advised to develop survey questions that are most likely to: a) help the researcher get at what they would like to know, b) motivate respondents to answer with honesty and accuracy, and c) reduce unintentional response or measurement error. Different question structures (open-ended, close-ended [with ordered & unordered response categories] and partially close-ended) are then reviewed with the strengths and (especially) the drawbacks of each design’s capacity to contribute to these goals reviewed. Emphasis is also placed on wording questions effectively. The take-home message is that it is a good idea to vary question structures in order to achieve the goals previously discussed.

While Dillman’s notes do not, and could not possibly, provide an exact roadmap to effective survey design, the point about the importance of effective survey design is well-stated and the tips offered are useful. The exercise/illustration at the end of the document, which shows the numerous ways in which a single question can be posed, provides an interesting piece for discussion while also demonstrating how a question can tell us more than simply what we think we might be asking. Which of these question designs would you choose? Why? What could one question structure helps us to learn that another design might not be able to accomplish?

In support of Dillman’s emphasis on the importance of test design and wording, Albright and Thyer’s article about test validity and the ASWB clinical social worker exam offers a strong complement. Albright and Thyer’s experiment, which called into question the genuine challenge presented by (and therefore validity of) the social worker examination, reinforces Dillman’s assertion about the importance of wording and design. Their work also emphasizes what I see as an underlying message in both pieces, which is that test/survey questions express a lot more than the simple question we may think we are asking.

Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.