SAVE | EMAIL | PRINT | MOST POPULAR |  RSS | | REPRINT
|
E-Learning Evaluation
May 20, 2009
By Saul Carliner
Although most organizations recognize the importance of evaluating e-learning programs, confusion exists around the specifics for doing so. This article will answer six frequently asked questions and clarify some misconceptions.
1. How does evaluation for e-learning differ from evaluation for classroom learning?
Fundamentally, it does not. The purposes of evaluation, the different types of evaluation, and the means of designing them are essentially the same. What differs are some techniques, because being online creates some specific opportunities for evaluation and some unique challenges.
2. What's the difference between formative and summative evaluation?
<!-- begin ad //-->
<SCRIPT language="JavaScript">CM8ShowAd("Middle");</SCRIPT>
<!-- end generated ad //-->
Formative evaluation refers to activities conducted while the learning program is under development (or in formation). It aims to determine whether the draft learning program is likely to meet its intended objectives and, if not, what specific learning activities are off track so course designers can fix them before the course is widely released. Formative evaluation activities include running a pilot of a course (with e-learning, these often are conducted by usability experts), technical reviews by subject matter experts, and editorial reviews by professional editors or other instructional designers.
Because designers of e-learning programs lack the direct, immediate contact with learners, they cannot easily clear up confusion after a course goes into wide use. That's why it's important for course designers to conduct formative evaluations—to avoid problems later.
In contrast, summative evaluation assesses whether a course has achieved its objectives. The most common framework for conducting summative evaluation of workplace learning programs is the Kirkpatrick framework. Common types of summative evaluations include post-class satisfaction surveys, tests, post-course observations on the job, and analysis of the learning program's impact.
3. Why do summative evaluations have to be validated?
Because survey questions on post-class satisfaction surveys and questions on post-class quizzes and tests might not be clear (and, therefore, might not be assessing what the designer intended), summative evaluations should undergo formative evaluations. In fact, only summative evaluations that have been through a formal formative evaluation are considered to be validated—that is, appropriate and fair instruments.
And one may only assess the effectiveness of a course based on validated instruments. In practical terms, this means results of summative evaluations from pilot classes are not appropriate sources of information about the effectiveness of courses, as the pilot test is intended to assess whether those summative evaluations are doing their jobs.
4. What's an appropriate test question?
The foundation of most workplace learning—especially technical and sales training—is learning objectives. These objectives serve key purposes throughout the course. At the beginning, they state what learners should be able to do—in observable and measurable terms. In the middle, they comprise the content of the course. That is, the course content addresses the learning objectives.
At the end of the course, learning objectives serve as the source of test questions. For example, if the objective states, "Given the characteristics of prospective customers, match those customers with a model of the XYZ copier that meets their needs," the test question should provide students with profiles of prospective customers, models of the copier, and ask them to match the models with the customers. In fact, the behavior word in the objective should be the word used in the test question.
Furthermore, the only fair test questions are ones that are directly generated from objectives. If the objectives state what's important to learn, the test questions provide a means of assessing whether learners mastered those objectives.
5. Why do participation rates in Level 1 evaluations differ substantially online and in the classroom?
Because classroom instructors can prod learners to submit forms in ways online instructors can't. Under the watchful eye of instructors, many classroom courses boast between 60 and 100 percent response rates. With e-learning, response rates can range as low as 1 to 10 percent. To increase those rates online, consider these suggestions:
• Rather than including the evaluation form at the end of the course, track the completion and separately send the form to the learner.
• Offer an incentive for completing the evaluation form.
• Send follow-up reminders to ensure a high response rate.
• Send surveys to learners who started courses but did not finish them.
Also, note that online surveying offers one significant advantage: It analyzes results automatically and can easily prepare basic reports.
6. How can technology assist with online quizzes and tests?
• Not only do most e-learning authoring tools and learning management software let you ask questions and track learners' responses, but many provide templates for writing the most types of objective questions (that is, questions that can be answered with a letter or word—not to be confused with a learning objective).
The software also lets you provide users with feedback to their responses. For exercises and questions, effective feedback can provide one of the most meaningful learning experiences. The challenge, of course, is first writing meaningful distracters—that is, incorrect responses to questions. The most appropriate distracters are ones that can be easily confused for the correct answer. For example, the incorrect response to an arithmetic problem that involves multiplication might be a choice in which the same numbers were divided or added. The reason for the error ("you divided, but the correct response involved multiplication" can be easily pinpointed and provided). Writing good distracters and feedbacks are design tasks not best handled online.
• Many programs that let you simulate other software also let you simulate tasks in the software and provide learners with feedback. So for software training and software-based tasks, you can evaluate learners' ability to perform actual tasks, rather than guessing at their abilities by asking multiple choice questions and hoping they transfer their knowledge to the real world.
• Other programs let designers create drag-and-drop exercises that mimic real-world tasks, such as configuring hardware and designing a room.
• Because of learning standards such as SCORM, you can use information about learner performance in one course to appropriately place them in another—all handled automatically and with limited intervention by the learner.
• With the assistance of Webcams and keystroke tracking, some programs let you visually observe learners performing tasks on other computers, record what occurs, and evaluate that to provide effective feedback.
In addition, many people believe essay questions cannot be included in online tests and quizzes. Not true. But because analysis tools are still at the experimental stage, you'll need to grade those manually (and have a plan for doing so).
One issue many overlook is the necessity of validating identities of people taking tests. Learners can exchange passwords, so in some instances, passwords alone do not provide sufficient protection. To avoid this problem, some organizations only offer online tests with an in-person proctor, who checks a photo ID before assigning learners to testing stations.
One last thought on evaluation: More than merely providing numbers, make sure evaluation includes written feedback, so all recipients of evaluation information can interpret what it means.
Saul Carliner is an associate professor of educational technology at Concordia University in Montreal and has published five books on designing learning and e-learning. Click here for more information.
|
SAVE | EMAIL | PRINT | MOST POPULAR | RSS |
|
|
| Back to Training Index |
|
|
