SAVE | EMAIL | PRINT | MOST POPULAR |  RSS | | REPRINT
|
Asking the Right Questions: Best Practices from Evidence-Based Design
August 28, 2009
By Dan Hawthorne
In today's economy, training programs increasingly come into the cross-hairs when management looks at ways to cut costs. Do you know which training programs really work and add value to the company?
You need data—such as from course surveys. Good, relevant post-training surveys allow you to check that your training programs align with your company's goals.
Good survey design requires more than just asking questions. In general, you must achieve two goals:
• Ask relevant questions your audience understands.
• Collect the information you need to make informed decisions.
When psychometricians design personality tests, they use a strict methodology to design and test each question. While you may not plan to publish your survey results in an academic journal, the information you collect will form the basis for important business decisions. Here are tips you can use to improve your surveys by following design principles used by researchers. That way, you will collect the best data.
Reflect before you write questions
Psychometricians take time to think about the topics. They don't rush to write questions. For example, they might conduct a comprehensive literature review. Fortunately, you can take a simpler road. You should spend time thinking about the topic you're training your learners on. In addition, you should research and then think about the audience(s) that will be completing the survey. For instance, if the training program is going to be delivered to managers and first-line employees, do you need to have different surveys for each group? Managers and employees may have different perspectives.
Writing the questions
When you are ready to write the survey questions, think about the following points carefully.
Pyschometricians use these principals to create good surveys:
• Write more items than you need—when test designers create survey items, they write many more questions than they will actually need. That way, when they pilot test the items, they will find the ones to eliminate and end up with their best ones.
• Avoid double-barreled items—double-barreled questions pack two separate questions into one. For example: "The workbook was complete and helpful." A trainee might feel the manual was complete but not helpful (or the reverse). Tip: break those items up into two items, and then decide if it is necessary to ask both.
• Beware of social desirability—when test designers ask about sensitive, personal topics, they expect respondents may be less than truthful with their answers. For instance, if the trainer is well-liked or in a position of power (supervisor), then any questions about the trainer's professionalism or effectiveness may be contaminated by those feelings.
• The shorter, the simpler, the better—make your items as short as possible. Each added word increases the level of complexity and creates more room for personal interpretation.
Ideally, you want every single person to interpret your items in the same way. Complex and vague questions will only increase the odds that your data will be murky and difficult to use. So, shoot for items that are fewer than 20 words, written in the most basic language possible, and only contain one complete thought.
• Search for different opinions—when everyone answers the question the same way, you will gain little information. While you might think it is ideal to have everyone give high answers to the questions you ask, response patterns like this provide few insights. With variability, you can get some idea why people responded the way they did.
Without variability, you're in the dark about why someone did or didn't learn.
• Be careful using "no"—when a person reads a negative modifier, such as "don't," the brain has to process the word in a two-step process. The reader's brain has to process the positive word and then apply the negative modifier to understand it. Processing a positive word is only a one-step process. Instead of using negatively worded items, try to reword them into positively worded items.
• Pilot test your items—if you have run this training program before, pick a few people you think will give you honest answers, and ask them to answer all of the candidate items. Also ask them for their feedback about questions they wished had been asked.
When you get their feedback, go through and look for which items seem to be the best ones to collect the information you need.
Once you have a bank of good survey items targeted specifically for the training program you are putting together, you are ready to launch the post-training survey:
• Present the surveys to the audience properly.
• Give them sufficient time to answer the questions.
• Make sure trainees know their opinions are important to you.
• Let them know their responses will be used to make the training better for the next group of people.
When you follow all of these steps, you should collect data you can use to improve your training programs and ultimately, your organization as a whole.
Dan Hawthorne is director of research for Intulogy LLC., an outsource training provider specializing in design and delivery of training programs built around their clients' proprietary subject matter. For more information on Intulogy, visit www.intulogy.com
|
SAVE | EMAIL | PRINT | MOST POPULAR | RSS |
|
|
| Back to Training Index |
|
|
