Laying the Analytics Foundation II: Designing a Questionnaire

Designing a successful questionnaire often involves balancing two dueling objectives: (1) getting all the information you need, and (2) persuading your survey participant to provide the information. Ideally, your questionnaire should not have too many questions or be too difficult to fill out; least your participant becomes frustrated and quits, or worse, provides bogus information. At the same time, if you do not get the information you need, the entire exercise becomes a waste of time and resources. Prior to designing the questionnaire and carrying out the survey, it is assumed that you have tried to obtain the information from secondary sources and failed. Gathering information by primary sources, such as a survey, is almost always more expensive than obtaining it through secondary sources.

Before creating the questionnaire, determine exactly what information you need for your analyses. Use short and simple questions to query the information. Avoid using difficult or ambiguous language. The rule of thumb (what I've been told) is that an 8th grade student should be able to read and completely understand the questions. Make a good faith effort to limit the number of questions to what you absolutely need. Provide enough space for respondents to be able to write out the answers.

The next step is arranging the questions according to some logical sequence, to not confuse the participant. If you look at the example below (Figure 2.1), the questionnaire was designed to obtain information on USPS packages transported by rail vans. We decided gathering information on the packages was not enough for our analyses; we also needed information on the rail vans and plants. Accordingly, we divided the questionnaire into three parts. The first part pertained to the rail plant, because that's what the data collector would first encounter. Once the data collector had entered the plant and filled out the necessary data, the next step was finding a rail van. Hence, the second part of the questionnaire involved gathering data on the rail van. Finally, after locating the rail van, the data collector would be able to find the mail packages and fill out the third and final part of the questionnaire.

Carefully decide the type of question to include in your questionnaire. Your questionnaire can consist of Structured questions or Unstructured questions, or both. For structured questions, you essentially know the answers of the questions and force the participant to provide a specific answer. It may be multiple choice, binary (i.e., Yes/No), or inquire for a specific type of information (i.e., Mail Code). The common predicament with structured questions is that you have to know the potential answers in advance. Rarely, you get unknown information with a purely structured questionnaire. Unstructured questions, on the other hand, provides the participant with a free form to volunteer information. Unstructured questionnaires allow you to uncover new details about your test subject. However, you might not get the information you need for your analyses. Although structured questions are great for analytics purposes, use some unstructured questions to provide some flexibility in your questionnaire (See Question #24 in the sample questionnaire).

In our sample questionnaire, you'll see that I tried to fit everything on two sides of a single page. The front page contained the questions, while the back page contained the instructions for answering the questions. This was intentionally done to simplify the job of printing and distributing the questionnaires. I provided a brief purpose so that the data collectors had a broad overview of why we were gathering the information. Each question number in the front page had a corresponding number in the back page that provided the instructions on how to collect the data. Exceptions were highlighted in bold or underlined to draw attention. I also provided hints on where the data collector could find the necessary information. If a data collector - after reading the instructions - had any questions about the survey or procedures, I provided the name and phone number of a contact person to help him/her out. For contingencies where data collectors had to record a lot more data than expected, I provided supplemental questionnaires.

On the bottom right corner of the front page, you'll see a space for processing code. The processing code is used to tag the completed questionnaire after you receive it. It is good practice to save the original paper copies. During later stages of data processing and analyses - if you ever stumble on data that makes no sense - you can use the processing code or tag number to pull up the original questionnaire and see how it was filled out.

Finally, test your questionnaire once it is completed. Give copies to people you know and ask to fill them out. This will allow you to identify and fix any wrinkles you may have overlooked.

Figure 2.1: Both sides of a sample questionnaire