Assessment Mistakes by E-Learning Developers

In a webinar this month (December 2006), I asked a group of about 100 e-learning professionals, what was the highest level of assessment they did (based on Kirkpatrick's Four Levels) on their most recent learning-intervention development project.

• 11% said they did NO evaluation
• 26% said they did Level 1 smile sheets
• 48% said they measured Level 2 learning
• 15% said they measured Level 3 on-the-job performance
• 0% said they measured Level 4 business results (or ROI).

Unfortunately, smile sheets are very poor predictors of meaningful learning outcomes, being correlated at less than an r of .2 with learning and performance. See Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-357.

Stunning: Even after all the hot air expelled, ink spilled, and electrons excited in the last 10 years regarding how we ought to be measuring business results, nobody is doing it !!!!!!!!!

------------------------------

When I asked them how they did their most recent assessment, in terms of WHEN they did the assessment---whether they did the assessment immediately at the end of learning or after a delay,

• 77% said they did the assessment, "At the end of training."
• 7% said they did the assessment, "After a delay."
• 14% said they did the assessment, "At end—and after a delay."
• 2% said, "Never done / Can’t remember."

Unfortunately, the 77% are biasing the results in a positive direction. They are measuring learning when it is top-of-mind, easily accessible from long-term memory. They are measuring the learning intervention's ability to create short-term learning effects. They are not measuring its ability to support long-term remembering, or its ability to specifically minimize forgetting.

In the graphic depiction below, the top of the left axis (the y axis) represents more remembering, the bottom is less remembering. Consider what happens if we assess learning at the end (or the top) of the first (leftmost) "Learning" curve. If the learners utilize what they've learned on the job, such an assessment has a negative bias. However, what typically happens over time is more like the forgetting curve (depicted at the lower right). Unless learners regularly use what they've learned, any assessment at the end of the first learning curve is likely to be a poor predictor of future remembering---and show a definite positive bias.

------------------------------

When I asked them how they did their most recent assessment, in terms of WHERE the learners were when they completed the assessment,

• 70% said they did the assessment, "In the training room/context."
• 26% said they did the assessment, "In a different room/context."
• 5% said, "Never done / Can’t remember."

Unfortunately, the 70% are biasing the results of their assessments in a positive direction. When learners are in the same context during retrieval as during learning, they tend to recall more because the background context stimulates improved retrieval. So, providing our training assessments in the training room (or using the same background stimuli in an e-learning course), is not a fair way for us to get feedback on our performance as instructional designers.

For one example of this research paradigm, see the following example. Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

The Bottom Line

First, I'm not blaming these particular folks. This is a common reality. I have regularly failed---and continue to fail often---in validly assessing my own instructional-development efforts. I'm much better when people pay me to evaluate their learning interventions (see LearningAudit.com).

Almost all of us---as far as I can tell---are just not getting valid feedback on our instructional-development efforts. Note that though 48% said they did Level 2 learning evaluation on their most recent project, probably most of those folks delivered the assessment in a way that biased those results. This leaves very few of us who are getting valid feedback on our designs.

We're in a dark fog about how we're doing, and so we have massively impoverished information to use to make improvements.

Basically, we live in a shameful, self-imposed fog.

Bring on the fog lights!!

Comments

[Cameron Nichol]

Fantastic. Simple, clear.

Many researchers roll their eyes at any results that come out of the field of education.

If you want to make a case for funding (and continuing employment) then you need data that stands up to scrutiny.

[Ben Curtis]

I think there are strong incentives to do smile sheets and level 2 evaluations in a setting that enhances passing. Students, teachers, administrators, managers all have an interest in positive outcomes. The incentives for unbiased levels 2, 3 and 4 seem to be less compelling. ROI results that are not positive are likely to displease powerful decision makers, yet there are many factors that predispose the results to be less than favorable. Maybe be we could place more of the burden on incentives than on evaluations.

[Phil Vitkus]

Hi, Will,

I am surprised you combined level 4 business results and ROI.

Level 4 analyses speak to the data of organizational improvement - reduction in errors, increase in customer satisfaction, increase in revenue.

When doing ROI (not frequently, but have done), I tend to compare the total money value of the level 4 to the total cost of designing, developing, and delivering training. This in a sense becomes the fifth level of evaluation, not 4th. Thoughts?

Level 4 business results (or ROI).

[Will Thalheimer]

Phil,

I collapsed Level 5 into Level 4 because Level 5 depends on Level 4 data. Level 5 equals Level 4 divided by the costs of learning) Since no one did Level 4, no one did Level 5, so for this group anyway, it matters not that the data was collapsed. It would certainly be interesting if lots of people were doing Level 4 to see how many were also doing Level 5.

[Mary Hayes]

"we ought to be measuring business results, nobody is doing it !"

From my own experience, I think you've got this wrong. It's not that we're not measuring it, but that we're not publishing it. Every project I've done, we have measured the business results. Corporations do not publish data that provides a competitive advantage, and when you find something that increases sales, reduces costs, improves retention... good luck as an instructional designer getting corporate sign off to share how you did it. But watch and listen-- are more corporations investing? There's your measure.

[SweetLakeisha27]

I took my first loan when I was 20 and it helped me a lot. Nevertheless, I require the credit loan once again.

[A learning]

HI,
How to download videos in internet of Adobe flash players type like some internet e-learning site provides?

[Gucci shoes wholesale]

This is a very sophisticated article, the argument is right with unique features, analysis, more clearly, this is a very good article, there will be many people like it, of course, including my own person. Yes, I have always liked this article. Are often concerned about this article. This theme I've always liked it. Thank you!

[cheap Vibram Five Fingers]

Very good web page, where did you come up while using the info in this piece? I’m pleased I discovered it although, ill be checking back soon to see what other posts you might have.

[discount jerseys]

My friendS told me that this blog is competitive. i will continue to read.
http://www.nfljerseysmalls.com

[life coach]

This is a good essay.
http://www.christianlouboutineonsale.com
http://www.coachoutlethandbagsonline.com

[クレジットカード現金化]

お元気ですか?

[air max 2009]

Thank you so very much for your timely tip, a wonderful gift to everyone.
Have a great and joyous Christmas.

[small business credit cards]

thanks man..
i liked this page.

[cheap nike free]

I am so happy here and enjoy this story. It brings me some special feelings. Hope you have a cheerful day.

[creditcard genkinka]

Very interesting topic and useful

[saga hair]

Thank you so very much for your timely tip, a wonderful gift to everyone.
Have a great and joyous Christmas.

[saga remy]

Thank you so very much for your timely tip, a wonderful gift to everyone.
Have a great and joyous Christmas.

[wholesale hats]

It has been a pleasure to read the different articles in it, I'm subscribing to your rrs feed right now!
http://www.wholesalehatsstore.com

[cheap coach purses]

it is seasonable for your idea. i agree it.

[gucci belts]

i like your posted meassage and enjoy reading it.

[cheap nike free shoes]

What a great post! Just one suggestion:If you add some pics, it would be easiler to follow!

[shopping genkinka]

Excellent site, keep up the good work my colleagues would love this.

[canada goose coat]

Your blog has truly inspired me, especially the current article. thank you for sharing your thoughts and the impact of them has been tremendous to me.Thank you more time and keep up with the good work.

[loss weight quick]

I am very much worried about my health but you provided me great information.
I feel really good to participate in this discussion. If you are worried about your health just visit my blog.
this is really a nice link to built relationship online in a fast way. i wana share some good links for health concious people.

THIS MAY SEEM RUDE BUT DO YOU OR ANYONE ELSE KNOW WHAT TRACK WAS USED AT THE END OF SEASON 1 EPISODE 2 WHEN JACKIES HUSBAND POURS HERA CUP OF COFFEE IN HIS BAR.

baba sleeping

thanks

[renlewei]

Gucci handbag is a world-renowned high-quality materials. They also have a timeless classic look.

[cheap canada goose]

I have paid attention to, thank you for sharing!

[cheap snapback hats]

I agree with your point of view of this article. This is a good article. Very timely given us so much useful information. Thank you!

[nike air max]

are very poor predictors of meaningful learning outcomes, being correlated at less than an r of .2 with learning and performance. See Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria

[chanel bags]

"Learning" curve. If the learners utilize what they've learned on the job, such an assessment has a negative bias. However, what typically happens over time is more like the forgetting curve (depicted at the lower right). Unless learners regularly use what they've learned, any assessment at the end of the first learning curve is likely to be a poor predictor of future remembering---and show a definite positive bias.

[true religion jeans]

remembering. Consider what happens if we assess learning at the end (or the top) of the first (leftmost) "Learning" curve. If the learners utilize what they've learned on the job, such an assessment has a negative bias. However, what typically happens over time

[nike shox]

It is interesting concept, i wonder if it will really increase sales tho, hopefully it does work.

[football jerseys]

Your blog has truly inspired me, especially the current article. thank you for sharing your thoughts and the impact of them has been tremendous to me.Thank you more time and keep up with the good work.

[North Face Outlet]

Yes, all the above all post are informative and provide great information so thanks to share it
http://www.buynorthfacediscount.org

[supra footwear]

Yes, all the above all post are informative and provide great information so thanks to share it