FREE Research Report on Feedback

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops.

And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right?

You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times.

Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback.

Here are some of the insights from the two-part 88-page research report:

1. The most important thing to remember about feedback is that it is generally beneficial for learners.
2. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the “whys” and “wherefores.”
3. The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning.
4. Feedback works by correcting errors, whether those errors are detected or hidden.
5. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval.
6. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities.
7. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback.
8. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval.
9. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice.
10. Delayed feedback creates a beneficial spacing effect.
11. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback.
12. Feedback should usually be provided before learners get another chance to retrieve incorrectly again.
13. Provide feedback on correct responses when:
    a. Learners experience difficulty in responding to questions or decisions.
    b. Learners respond correctly with less-than-high confidence.
    c. All the information learned is of critical importance.
    d. Learners are relatively new to the subject material.
    e. The concepts are very complex.
14. Provide feedback on incorrect responses:
    a. Almost always.
    b. Except:
    i. When feedback would disrupt the learning event.
    ii. When it would be better to wait to provide feedback.
15. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself.
16. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results.
17. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay.
18. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback.
19. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback.
20. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts.
21. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts.

Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below.

The link to download the feedback report

Comments

[Sharon McGann]

Thanks for your generosity Will and for the high quality information. I will enjoy using it to review our programs.

[Thiagi]

Great stuff, as always!

Thank you for choosing the free approach.

Your experiment has succeeded. I have sent the URL to 11 people. I will mention the report in the next issue of my newsletter (that goes to 12000+ online readers).

As someone who believes that evidence-based principles should inform our profession, I thank you for the wonderful service you are providing. May the Force be with you.

(Hmm ... this is not corrective feedback. Does it mean this is useless feedback?)

[Loretta Donovan]

This is such a fine piece of research and very, very relevant. I'm delighted that you have joined the open source movement. Thanks for your leap of faith.

[familydoc]

what you have done is just wonderful!
I will be reading through your report I really appreciate it.
Please keep up the good work!

[kevin kissack]

Hi Will
thanks for the free info. Always top notch and useful.

Have you thought about other ways of generating compensation (revenue) for your research? Such as annual membership? This is popular for some sources of info. You could conduct a poll online to see what is a preferred way? I would pay.

[kevin kissack]

Hi Will
thanks for the free info. Always top notch and useful.

Have you thought about other ways of generating compensation (revenue) for your research? Such as annual membership? This is popular for some sources of info. You could conduct a poll online to see what is a preferred way? I would pay.

[Mark Frank]

I would like to echo all the thanks above. I loved this report. It was clear, useful and represented a massive amount of hard work and expertise. So I hope you will take it in the right spirit if I become devil's advocate for a moment and pursue a favourite theme.

What concerns me is how to integrate this research based approach to learning with practitioners’ own experience and logic. I don’t doubt there is tremendous value. But we should not let our respect for academic research and a scientific approach cause us to dismiss other sources of expertise and knowledge. Almost all citizens of first world countries have spent decades at school and college being learners, and therefore have enormous in-depth experience of what works and what doesn’t. In addition, most members of the learning profession have spent many years trying to help people learn, observing the results and sometimes acting on the observations. This represents an extraordinary resource and right now if I came across a piece of research that conflicted with my personal experience – it is the research I would doubt.

How much is there in the report that an experienced, intelligent instructor would not already know to be true? There are some real new insights. But many of the conclusions border on the obvious. For example, a strong conclusion from the report is that feedback is effective, and is most effective in correcting wrong answers. Did it really need several man-years of research effort to conclude that learners benefit from being told when they have got it wrong? A much more difficult question is when to give feedback. But it is precisely at this point that the research becomes inconclusive and the professional has to fall back on their own experience and logic. It is interesting to note what happened when the research appeared to show that feedback hurt learning (part 2 page 13). Here the research results conflicted with most people’s experience. The research eventually proved to be faulty.

Having said this, there is no question that an awful lot of training and education is poor – to the point of being a waste of time – and does not implement even the most straightforward recommendations in this report. But is it because we don’t know better? Do we need academic research to tell us that learners benefit from correcting their mistakes? Or do we just need reminding? Or even just the resources to do it?

I suggest that the big value of Will’s work is to remind us what we knew all along, motivate us to take it more seriously, and provide evidence to make the case for more resources to do things properly.

[Will Thalheimer]

Mark,

Thanks for your thoughtful comments.

Let me first agree that research from refereed journals cannot be our sole source of information. Even if there was a complete database of research, we as practitioners would have to test out the generalized recommendations to see how they worked in our specific situtations.

Let me disagree with you too. Our experience as learners and as learning professionals are useful, but are not always helpful in making good learning design decisions. For example, most people in the learning-and-performance field believe erroneously that immediate feedback is always better than delayed feedback. If I remember correctly, more than 80% of people assumed that immediate feedback was better on a scenario-based question I asked a few years ago. The bottom line is that learners aren't always good at understanding their own learning. We know from research that learners are overly optimistic about their ability to remember, they think poor memory strategies are more effective than they are, and they lean on using inadequate learning methods (e.g., cramming) even when their own experience demonstrates that their long-term retention suffers. To see the extent of people's inadequate knowledge, see my blog post of a few years ago -- www.willatworklearning.com/2007/02/learning_resear.html.

Learning professionals are in a worse predicament. The feedback we get on our own learning programs is grossly inadequate. Most of us rely solely on smile sheets (i.e., learner reactions) which have been show to be almost completely unpredictive of learning results. Even if we rely on Level 2 tests of learning, these tests are biased by being given immediately after learning or in the presence of biasing contextual hints.

Research is not perfect, and if you've ever attended any of my speaking gigs or workshop you'll hear me reiterate this time and again, but research is our first best step in understanding how to do learning right.

Throwing out the research baby with the research bathwater is simply not a productive strategy in our age of darkness.

Here's my recommendation. Use the research as a first approximation and as a way to really understand learning at a deep level. Then test out these research-based prescriptions (as if they were hypotheses) in our real-world learning interventions. See if they work, tweak them, etc. Of course, this requires valid measurement techniques, which is another long story...

You are also wrong in the specifics of the research report on feedback. Many many people do not understand that feedback corrects errors. They assume falsely that feedback is just as important for correct answers as it is for incorrect answers. Maybe you didn't have this false impression or maybe the way the research report was written provided just the right metaphorical supports to provide you with the correct mental models of how feedback works, but I've been talking about feedback for several years and many people in our field don't understand this basic concept.

Yes, we need the researchers to continue their good work. I wish they were doing even better work in making the research more real-world, more complete, more generalizable; but they are providing a valuable service.

We also need more than one or two people doing translational research, spending years with our heads in the research, spending years talking to learning professionals, trying against hope and exhaustion to figure out what illustrations, metaphors, and examples resonate with those on the front lines of learning. There are so few of us because it is so damned hard to do, and perhaps because we have to swim against the tide of the research bathwater as it is thrown in our faces.

Mark, thank you again for your kind words and your thoughtful analysis. Keep up your advocacy. We need more people who fully understand the challenges we face in improving the current state of learning.

[Mark Frank]

Will
Thanks for such a prompt reply. Unless you do a gig in the UK I am unlikely to ever hear you live, so this forum will have to do.

I most emphatically do not want to throw out the research baby with the bathwater. But I also am nervous of making this baby head of the family. Just like our own experience, research can be limited to a specific context and lead to wrong conclusions. My question is how to combine the two.

Your model seems to be that research will throw up the deep hypotheses that can then be tested out in the reality of experience. You say “research is the first step”. But I would argue that as often as not it is the other way round. Reflecting on our experience gives us deep insights into what is going on when we learn; insights which we can test out against research results as well as our own experience. I don't see many fundamental additional and reliable insights coming out of research. There are exceptions e.g. the limits on working memory and value of spacing out learning. But its big value (so far) is to act as a check on the assumptions and beliefs that arise from our reflections on our own experience.

The relative value of feedback on correct or wrong answers is an excellent example (unless I have totally misunderstood what you wrote). You show that research has concluded that it is usually more valuable to give feedback on wrong answers than correct answers, but there is some value in giving feedback on correct answers if the subject is not certain they are correct. As I understand it, in almost all these cases the value of feedback is determined by giving a test, then some feedback, then testing again. In this context it is surely obvious that

(a) If I got it right first time and I am sure I got it right, then I will almost certainly get it right the second time with or without feedback i.e. feedback makes no difference.

(b) If I got it right first time but I am not sure I got right, then I will very likely get it right again – but feedback reduces my chance of changing my mind.

(c) If I got it wrong and I am told why I got it wrong, then I am very likely to change my mind and get it right next time i.e. feedback makes a big difference.

(d) If I got it wrong and no one tells me I got it wrong, I am very likely to get it wrong again!

It seems to me that this logic makes these the results utterly predictable and explains why. (As an experiment I asked my son who is currently studying for his A levels whether it is more valuable to be told about the answers he got right in practice tests or the ones he got wrong. He gave me the pained look teenagers reserve for their ageing parents, and replied “what’s the point in being told about the stuff you already know?”).

You say that many learning professionals believe feedback on correct answers is more important than feedback on wrong answers. I cannot believe they would not accept the logic above and suspect they have a different reason for valuing feedback on correct answers. If your teacher only dwells on your failures and fails to acknowledge your successes, then it becomes demotivating in the long run. I am sure that is true - but I base it on my experience. I think it would be hard to test academically :-)

[ErnestO Stolpe]

My comment is in reference to item 21 in the research paper "Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts".

In Formal On Job Training (FOJT) the best feedback is given in real world problem solving activites. The learner is completely engaged and in failing to recall the solution is at that moment ready to thrust the trainers/mentors feedback (solution) into long term memory.

[chhabi Ghimire]

I'studying LLM in Nepal and it is me necessary to much
Thanks

[IFKRISTA]

I think that there is not a pretty good idea to accomplish the definition essay by your own efforts! I think, that would be better to order the good essay at the custom sociology essays service, just because that will save your time.

[workout dvd]

It sounds well for this paper.

[coach handbags wholesale]

Good check list, I am running through it with my sites now .

[coach handbags wholesale]

Very interesting article. It’s funny how history can be twisted in so many different ways. These photos certainly give us clues, but I guess we’ll never know the true story. . . .

[cheap fashion clothes]

I have never read such a wonderful article and I am coming back tomorrow to continue reading.
http://www.belowbulk.com

[home loans]

Set your life more simple take the home loans and all you require.

[north face backpack sale]

It sounds well for this paper.

[Mac Cosmetics]

Colour plus texture for the lips. Stands out on the runway. Simmers on the street! What made M·A·C famous.

[ralph lauren outlet]

Good day! Would you mind if I share your blog with my myspace group? There’s a lot of folks that I think would really enjoy your content. Please let me know. Cheers… http://www.hipoloshirts.com

[cheap louie vutton underwear]

Your article swept me away with its vast information and great writing. Thank you for sharing your views with such passion. I like your views.

[cheap hats]

Thank landlord impartiality. Come up with such a good article to share with us. This is the official information from articles I and my friends need. Thanks again!

[cheap clothes online]

It makes me feel so surprise.

[retro jordans]

It's so great Get a move on now!

[retro jordans]

Get a move on now!

[oil press]

I'm a trainee teacher and I've lost track of how many of my lecturers have presented us with those figures. I shall post this up on our noticeboard.

[New Era Hats]

No one can succeed without any hard work. Karl Max was successful, because he spent more than 30 years writing the book "Communist Manifesto"; Tomas Edison succeeded, because he had experimented thousands of times to find the best material for lights. Every success calls for hard work. If you want to suc-ceed, work hard first.

[Cheap New Era Hats]

Great post.Thank you for sharing!

[Wholesale New Era Hats]

Great post.he article is informative and witty at the same time.

[Pellet Mill]

Whatever is the explanation very open and very clearly the issues.

[hermes birkins]

it seems to be insteresting for your paper here.

[onthebroadway]

This is central, obviously, during landing approach but is also a security edge if the crew were ever to become unfocused and allow the airspeed to decay in any phase of flight.

[Dried Fruits]

If not, then all we've done is questioned published results, without an alternative hypothesis.

[Moncler Jackets]

Very well said, your blog says it all about that particular topic.
http://www.cheapmonclermarket.com

[Red bull caps]

This article is exactly what I needed, thank you for sharing. I would recommend friends over to read. I hope you can keep writing more and more good articles. This is really useful to me.

[onthebroadway]

Everyone is happy with that inning, whether you like homers, endurance, hard singles with men on foot, hits with runners in scoring location, you name it.