Whether it’s market research or academic research, administering surveys can get time-consuming and expensive. If you want to get higher quality from your desired respondents then it becomes not only a question of proper screening protocols but also how that survey is written. Even though you want to get the data fast, it’s not prudent to just put the official survey out there without having a pre-test survey first. Here’s what you need to know about pre-testing a qualitative research survey.
Start With a Focus Group
Find 5-10 people from the targeted groups you’d like to survey. You may need to be a little flexible and find people who are close to the target group as possible if you’re short on time and and willing participants are available. Once a focus group has been formed, have a productive discussion led by a moderator that will lend additional insights on how the survey should be structured. This would include the goal and concepts in the survey, respondent burden, and how sensitive the topic is to them.
Ask Participants to “Think out Loud”
Everyone in the focus group should complete the survey alone and not watching others complete it. They should take this pre-test survey just like you’d expect them to do so for the final version, such as having a trained interviewer ask live questions or an online format. Most importantly, the participants need to think out loud: ask them to tell you what comes to mind every time they read or hear one of the questions. Note their reactions and feedback and observe how they complete the rest of the survey. Do they take a long time with certain questions or hesitate to answer? Are any questions frequently skipped?
Add an “Unsure” Option to Identify Confusing Questions
Depending on your target group and how comfortable they are answering certain questions, they simply may not want to answer certain questions or hesitate too much. While you should note all of the different gut reactions the focus group elicits, you need to have an “I don’t know” option on your pre-test survey questions. It can indicate a genuine lack of knowledge on a topic or you may have posed the question in a vague or confusing manner. Putting in an option for “I don’t know” or “Unsure” will improve the quality of your responses and show you what should be refined on the final version of the survey.
Test Different Question Types on the Pre-Test Survey With Different Respondents
The pre-test survey questions should deploy different question styles rather than different questions altogether. This variation can be random across the entire focus group or they can be concentrated on certain demographics, attributes, and parameters that set the participants apart. The gist of these questions should be exactly the same but the style of question should be different to see if people respond to it differently.
For example, you can use a Net Promoter Score question to know how much they like your product. An example of that is asking: “How likely are you to recommend this product to a friend?” or you can use an open-ended question; for example, “Tell us what we should improve about this product”. Analyze the data and use the question that gives you more insights.
By taking these measures to create your pre-test survey then getting feedback from your focus group, you can make tractable improvements that will increase both participation rates and outcome quality. When survey pilots run into problems, often it’s a question of data collection logistics, survey distribution, or poor staff training in addition to the survey itself. When you properly pre-test your survey, it becomes easier to identify these implementation issues and address them since the survey design is handled independently prior to the pilot.