Assessment Mistakes by E-Learning Developers

In a webinar this month (December 2006), I asked a group of about 100 e-learning professionals, what was the highest level of assessment they did (based on Kirkpatrick's Four Levels) on their most recent learning-intervention development project.

• 11% said they did NO evaluation
• 26% said they did Level 1 smile sheets
• 48% said they measured Level 2 learning
• 15% said they measured Level 3 on-the-job performance
• 0% said they measured Level 4 business results (or ROI).

Unfortunately, smile sheets are very poor predictors of meaningful learning outcomes, being correlated at less than an r of .2 with learning and performance. See Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-357.

Stunning: Even after all the hot air expelled, ink spilled, and electrons excited in the last 10 years regarding how we ought to be measuring business results, nobody is doing it !!!!!!!!!

------------------------------

When I asked them how they did their most recent assessment, in terms of WHEN they did the assessment---whether they did the assessment immediately at the end of learning or after a delay,

• 77% said they did the assessment, "At the end of training."
• 7% said they did the assessment, "After a delay."
• 14% said they did the assessment, "At end—and after a delay."
• 2% said, "Never done / Can’t remember."

Unfortunately, the 77% are biasing the results in a positive direction. They are measuring learning when it is top-of-mind, easily accessible from long-term memory. They are measuring the learning intervention's ability to create short-term learning effects. They are not measuring its ability to support long-term remembering, or its ability to specifically minimize forgetting.

In the graphic depiction below, the top of the left axis (the y axis) represents more remembering, the bottom is less remembering. Consider what happens if we assess learning at the end (or the top) of the first (leftmost) "Learning" curve. If the learners utilize what they've learned on the job, such an assessment has a negative bias. However, what typically happens over time is more like the forgetting curve (depicted at the lower right). Unless learners regularly use what they've learned, any assessment at the end of the first learning curve is likely to be a poor predictor of future remembering---and show a definite positive bias.

------------------------------

When I asked them how they did their most recent assessment, in terms of WHERE the learners were when they completed the assessment,

• 70% said they did the assessment, "In the training room/context."
• 26% said they did the assessment, "In a different room/context."
• 5% said, "Never done / Can’t remember."

Unfortunately, the 70% are biasing the results of their assessments in a positive direction. When learners are in the same context during retrieval as during learning, they tend to recall more because the background context stimulates improved retrieval. So, providing our training assessments in the training room (or using the same background stimuli in an e-learning course), is not a fair way for us to get feedback on our performance as instructional designers.

For one example of this research paradigm, see the following example. Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

The Bottom Line

First, I'm not blaming these particular folks. This is a common reality. I have regularly failed---and continue to fail often---in validly assessing my own instructional-development efforts. I'm much better when people pay me to evaluate their learning interventions (see LearningAudit.com).

Almost all of us---as far as I can tell---are just not getting valid feedback on our instructional-development efforts. Note that though 48% said they did Level 2 learning evaluation on their most recent project, probably most of those folks delivered the assessment in a way that biased those results. This leaves very few of us who are getting valid feedback on our designs.

We're in a dark fog about how we're doing, and so we have massively impoverished information to use to make improvements.

Basically, we live in a shameful, self-imposed fog.

Bring on the fog lights!!

Comments

[Cameron Nichol]

Fantastic. Simple, clear.

Many researchers roll their eyes at any results that come out of the field of education.

If you want to make a case for funding (and continuing employment) then you need data that stands up to scrutiny.

[Ben Curtis]

I think there are strong incentives to do smile sheets and level 2 evaluations in a setting that enhances passing. Students, teachers, administrators, managers all have an interest in positive outcomes. The incentives for unbiased levels 2, 3 and 4 seem to be less compelling. ROI results that are not positive are likely to displease powerful decision makers, yet there are many factors that predispose the results to be less than favorable. Maybe be we could place more of the burden on incentives than on evaluations.

[Phil Vitkus]

Hi, Will,

I am surprised you combined level 4 business results and ROI.

Level 4 analyses speak to the data of organizational improvement - reduction in errors, increase in customer satisfaction, increase in revenue.

When doing ROI (not frequently, but have done), I tend to compare the total money value of the level 4 to the total cost of designing, developing, and delivering training. This in a sense becomes the fifth level of evaluation, not 4th. Thoughts?

Level 4 business results (or ROI).

[Will Thalheimer]

Phil,

I collapsed Level 5 into Level 4 because Level 5 depends on Level 4 data. Level 5 equals Level 4 divided by the costs of learning) Since no one did Level 4, no one did Level 5, so for this group anyway, it matters not that the data was collapsed. It would certainly be interesting if lots of people were doing Level 4 to see how many were also doing Level 5.

[Mary Hayes]

"we ought to be measuring business results, nobody is doing it !"

From my own experience, I think you've got this wrong. It's not that we're not measuring it, but that we're not publishing it. Every project I've done, we have measured the business results. Corporations do not publish data that provides a competitive advantage, and when you find something that increases sales, reduces costs, improves retention... good luck as an instructional designer getting corporate sign off to share how you did it. But watch and listen-- are more corporations investing? There's your measure.