Will at Work Learning

Major Research Review on eLearning Effectiveness

Is elearning effective? As effective as classroom instruction -- more or less effective? What about blended learning -- when elearning and classroom learning are combined?

 

These critical questions have now been answered and are available in the research report, Does eLearning Work? What the Scientific Research Says!

In this research review, I looked at meta-analyses and individual research studies, and was able to derive clear conclusions. The report is available for free, it includes an executive summary, and research jargon is kept to a minimum.

 

Click here to download the report...

 

 

 

Note that the August 10, 2017 version of this report incorrectly cited the Rowland (2014) study in a footnote and omitted it from the list of research citations. These issues were fixed on August 11, 2017. Special thanks to Elizabeth Dalton who notified me of the issues.

 

Doing Research On Our Learning Products

The learning profession has been blessed in recent years with a steady stream of scientific research that points to practical recommendations for designers of learning. If you or your organization are NOT hooked into the learning research, find yourself a research translator to help you! Call me, for example!

That's the good news, but I have bad news for you too. In the old days, it wasn't hard to create a competitive advantage for your company by staying abreast of the research and using it to design your learning products and services. Pretty soon, that won't be enough. As the research becomes more widely known, you'll have to do more to get a competitive advantage. Vendors especially will have to differentiate their products -- NOT just by basing them on the research -- but also by conducting research (A-B testing at a minimum) on your own products.

I know of at least a few companies right now who are conducting research on their own products. They aren't advertising their research, because they want to get a jumpstart on the competition. But eventually, they'll begin sharing what they've done.

Do you need an example of a company who's had their product tested? Check out this page. Scroll down to the bottom and look at the 20 or so research studies that have been done using the product. Looks pretty impressive right?

To summarize, there are at least five benefits to doing research on your own products:

1. Gain a competitive advantage by learning to make your product better.
2. Gain a competitive advantage by supporting a high-quality brand image.
3. Gain a competitive advantage by enabling the creation of unique and potent content marketing.
4. Gain a competitive advantage by supporting creativity and innovation within your team.
5. Gain a competitive advantage by creating an engaging and learning-oriented team environment.

Using Effect Sizes and Considering Meta-Analyses

This post is for research geeks, and it's really just an introduction -- maybe a gentle warning -- as I don't have time or the statistical expertise to explore this deeply.

 

The Basics

When scientific experiments get done, researchers typically compare one experimental treatment to second one (or to no-treatment at all). So for example, we might compare two versions of the same elearning program, one that utilizes spaced repetitions and a second that uses unspaced repetitions. When we do such comparisons we need to know two things before we can draw conclusions:

1. Statistical Significance:
   How likely is it that the experimental results might be caused by random chance. Social scientists aim for results that are more than 95% likely to result from the experimental factors being studied. In other words, if we did the same experiment 100 times, we should expect the same outcome at least 95% of the time.
2. Effect Size:
   How different are the actual results. Are they sufficiently large to be meaningful?

If we don't take effect sizes into account, we can have an experiment that is statistically significant but not practically significant. That is, we can have statistical significance, but not effect-size significance. Without looking at effect-size calculations, we can be fooled into thinking that an experimental result is meaningful when it actually shows no substantial advantage for one learning method compared with another.

So for example, suppose that a new mobile-learning app improves learning by less than one-half of one percent, but cost $10,000 per learner...

Meta-analyses are statistical studies that compile many scientific studies, looking at the whole of the results. Meta-analyses have been a potent source of wisdom because they take complicated and complex results over a range of studies and combine them in a way that helps us make sense of the overall trends. Meta-analyses rely on effect sizes to calculate the overall importance of the factors being studied.

 

Some Subtleties

As with all things in science, over time scientists make improvement and refinements in their work. Effect sizes are no different. Recently, researchers have found that meta-analyses have to be interpreted with wisdom, otherwise the results may not be what they seem. Of specific concern is the finding that published studies tend to report higher effect sizes than unpublished studies. Quasi-experimental designs reported higher effect sizes than randomized control studies. Et cetera...

Here are some recommendations for researchers from Cheung and Slavin (2016), who are focused on educational research, but whose recommendations are widely applicable:

• In doing a meta-analysis, don't just look at published studies. Moreover, work diligently to gather all studies that have been done.
• Researchers, in general, should utilize randomized trials whenever possible. Those doing meta-analyses should look at these separately because they are likely to have the least-biased data.
• Policy makers and educators (and I, Will Thalheimer, would add all workplace learning professionals) should "insist on large, randomized evaluations to validate promising programs."

 

Some Research Articles of Relevance

Cheung, A. C. K. & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283-292.

 

Ueno, T., Fastrich, G. M., & Murayama, K. (2016). Meta-analysis to integrate effect sizes within an article: Possible misuse and Type I error inflation. Journal of Experimental Psychology: General, 145(5), 643-654. http://dx.doi.org/10.1037/xge0000159

 

van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20(3), 293-309. http://dx.doi.org/10.1037/met0000025

Research on Smile Sheets -- New Video

I created a video to help organizations fully understand the meaning of their smile sheets.

 

You can also view this directly on YouTube: https://youtu.be/QucqCxM2qW4

Psychological Scientists Now Reviewing Movies -- To Bring Clarity

Mary V. Spiers, professor of psychology (and neuropsychologist) at one of my Alma Maters, Drexel University, has created a brilliant website to clarify the psychological science depicted in the movies.

You must check this out:

• http://www.neuropsyfi.com/

I love the review of Finding Nemo. Do you remember Dory, the amnesic sidekick? http://www.neuropsyfi.com/reviews/finding-nemo

For another movie I really enjoyed, Memento, the reviewers point out both the ways in which the movie is accurate in reporting on anterograde amnesia, and inaccurate. http://www.neuropsyfi.com/reviews/memento

The website also has a page devoted to brain science, with information you can actually trust -- instead of some of the hyped stuff you might have seen from vendors in the learning field.

 

Spacing Learning Over Time -- Research Report

The spacing effect is one of the most potent learning factors there is--because it helps minimize forgetting.

Here's a research-to-practice report on the subject, backed by over 100 research studies from scientific refereed journals, plus examples. Originally published in 2006, the recommendations are still valid today.

Click to download the research-to-practice report on spacing.   It's a classic!

 

And here's some more recent research and exploration.

Poor New York Times piece on Education

Here is the comment I sent to the NY Times in response to their focus on a supposed research study that purported to show that gifted kids are being underserved.

I'm a little over the top in my comments, but still I think this is worth printing because it demonstrates the need for good research savvy and it shows that even the most respected news organizations can make really poor research-assessment mistakes.

Egads!! Why is the New York Times giving so much "above-the-fold" visibility to a poorly-conceived research study funded by a conservative think tank with obvious biases?

Why isn't at least one of your contributors a research-savvy person who could comment on the soundness of the research? Instead, your contributors assume the research is sound.

Did you notice that the references in the original research report were not from top-tier refereed scientific journals?

In the original article from the Thomas Fordham Institute (a conservative-funded enterprise), the authors try to wash away criticisms about regression-to-the-mean and test-variability, but this bone against the obvious--and most damaging, and most valid--criticisms is not good enough.

If you took the top 10% of players in the baseball draft, the football draft, any company's onboarding class, any randomly selected group of maple trees, a large percentage of the top performers would not be top performers a year or two later. Damn, ask any baseball scout whether picking out the best prospects is a sure thing. It's not!

And, in the cases I mentioned above, the measures are more objective than an educational test, which has much higher variability--which would make more top performers leak out of the top ranks.

NY Times--you should be embarrassed to have published these responses to this non-study. Seriously, don't you have any research-savvy people left on your staff?

We have scientific journals because the research is vetted by experts.

New Journal for Applied E-Learning Research

A new journal is forming to support applied e-learning research. Check it out. Get involved.

In Memory of H.M., cognitive science's most unforgettable experimental subject.

H.M. died on Tuesday. He was a severe amnesiac. At the age of 27 he underwent surgery to correct severe and debilitating seizures. When he awoke, he was unable to remember much of anything ever again--at least not anything in the declarative memory system.

He lived life as the most famous experimental subject in the history of cognitive psychology and neuroscience. I remember reading about him when I was a graduate student in the late 1980's and 1990's. What researchers learned by studying him was that there was more than one memory system. This information led to a revolution in our understanding of human cognition and learning.

After years being known only as H.M., to protect his identity, in death we learn that his name was Henry Gustav Molaison, and he lived his life in Connecticut, on the east coast of the United States.

The New York Times tells his story better than I can. It is well worth the read.

And NPR has a previous story, that you can hear. It is well worth the listen.

And here's H.M.'s wikipedia entry.

FREE Research Report on Feedback

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops.

And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right?

You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times.

Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback.

Here are some of the insights from the two-part 88-page research report:

1. The most important thing to remember about feedback is that it is generally beneficial for learners.
2. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the “whys” and “wherefores.”
3. The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning.
4. Feedback works by correcting errors, whether those errors are detected or hidden.
5. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval.
6. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities.
7. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback.
8. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval.
9. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice.
10. Delayed feedback creates a beneficial spacing effect.
11. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback.
12. Feedback should usually be provided before learners get another chance to retrieve incorrectly again.
13. Provide feedback on correct responses when:
    a. Learners experience difficulty in responding to questions or decisions.
    b. Learners respond correctly with less-than-high confidence.
    c. All the information learned is of critical importance.
    d. Learners are relatively new to the subject material.
    e. The concepts are very complex.
14. Provide feedback on incorrect responses:
    a. Almost always.
    b. Except:
    i. When feedback would disrupt the learning event.
    ii. When it would be better to wait to provide feedback.
15. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself.
16. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results.
17. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay.
18. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback.
19. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback.
20. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts.
21. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts.

Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below.

The link to download the feedback report