« Purchasing eLearning Tools? Consider Adobe! | Main | Thiagi Instructional Design Guru on Video »

Thursday, 08 May 2008


TrackBack URL for this entry:

Listed below are links to weblogs that reference FREE Research Report on Feedback:


Sharon McGann

Thanks for your generosity Will and for the high quality information. I will enjoy using it to review our programs.


Great stuff, as always!

Thank you for choosing the free approach.

Your experiment has succeeded. I have sent the URL to 11 people. I will mention the report in the next issue of my newsletter (that goes to 12000+ online readers).

As someone who believes that evidence-based principles should inform our profession, I thank you for the wonderful service you are providing. May the Force be with you.

(Hmm ... this is not corrective feedback. Does it mean this is useless feedback?)

Loretta Donovan

This is such a fine piece of research and very, very relevant. I'm delighted that you have joined the open source movement. Thanks for your leap of faith.


what you have done is just wonderful!
I will be reading through your report I really appreciate it.
Please keep up the good work!

kevin kissack

Hi Will
thanks for the free info. Always top notch and useful.

Have you thought about other ways of generating compensation (revenue) for your research? Such as annual membership? This is popular for some sources of info. You could conduct a poll online to see what is a preferred way? I would pay.

kevin kissack

Hi Will
thanks for the free info. Always top notch and useful.

Have you thought about other ways of generating compensation (revenue) for your research? Such as annual membership? This is popular for some sources of info. You could conduct a poll online to see what is a preferred way? I would pay.

Mark Frank

I would like to echo all the thanks above. I loved this report. It was clear, useful and represented a massive amount of hard work and expertise. So I hope you will take it in the right spirit if I become devil's advocate for a moment and pursue a favourite theme.

What concerns me is how to integrate this research based approach to learning with practitioners’ own experience and logic. I don’t doubt there is tremendous value. But we should not let our respect for academic research and a scientific approach cause us to dismiss other sources of expertise and knowledge. Almost all citizens of first world countries have spent decades at school and college being learners, and therefore have enormous in-depth experience of what works and what doesn’t. In addition, most members of the learning profession have spent many years trying to help people learn, observing the results and sometimes acting on the observations. This represents an extraordinary resource and right now if I came across a piece of research that conflicted with my personal experience – it is the research I would doubt.

How much is there in the report that an experienced, intelligent instructor would not already know to be true? There are some real new insights. But many of the conclusions border on the obvious. For example, a strong conclusion from the report is that feedback is effective, and is most effective in correcting wrong answers. Did it really need several man-years of research effort to conclude that learners benefit from being told when they have got it wrong? A much more difficult question is when to give feedback. But it is precisely at this point that the research becomes inconclusive and the professional has to fall back on their own experience and logic. It is interesting to note what happened when the research appeared to show that feedback hurt learning (part 2 page 13). Here the research results conflicted with most people’s experience. The research eventually proved to be faulty.

Having said this, there is no question that an awful lot of training and education is poor – to the point of being a waste of time – and does not implement even the most straightforward recommendations in this report. But is it because we don’t know better? Do we need academic research to tell us that learners benefit from correcting their mistakes? Or do we just need reminding? Or even just the resources to do it?

I suggest that the big value of Will’s work is to remind us what we knew all along, motivate us to take it more seriously, and provide evidence to make the case for more resources to do things properly.

Will Thalheimer


Thanks for your thoughtful comments.

Let me first agree that research from refereed journals cannot be our sole source of information. Even if there was a complete database of research, we as practitioners would have to test out the generalized recommendations to see how they worked in our specific situtations.

Let me disagree with you too. Our experience as learners and as learning professionals are useful, but are not always helpful in making good learning design decisions. For example, most people in the learning-and-performance field believe erroneously that immediate feedback is always better than delayed feedback. If I remember correctly, more than 80% of people assumed that immediate feedback was better on a scenario-based question I asked a few years ago. The bottom line is that learners aren't always good at understanding their own learning. We know from research that learners are overly optimistic about their ability to remember, they think poor memory strategies are more effective than they are, and they lean on using inadequate learning methods (e.g., cramming) even when their own experience demonstrates that their long-term retention suffers. To see the extent of people's inadequate knowledge, see my blog post of a few years ago -- www.willatworklearning.com/2007/02/learning_resear.html.

Learning professionals are in a worse predicament. The feedback we get on our own learning programs is grossly inadequate. Most of us rely solely on smile sheets (i.e., learner reactions) which have been show to be almost completely unpredictive of learning results. Even if we rely on Level 2 tests of learning, these tests are biased by being given immediately after learning or in the presence of biasing contextual hints.

Research is not perfect, and if you've ever attended any of my speaking gigs or workshop you'll hear me reiterate this time and again, but research is our first best step in understanding how to do learning right.

Throwing out the research baby with the research bathwater is simply not a productive strategy in our age of darkness.

Here's my recommendation. Use the research as a first approximation and as a way to really understand learning at a deep level. Then test out these research-based prescriptions (as if they were hypotheses) in our real-world learning interventions. See if they work, tweak them, etc. Of course, this requires valid measurement techniques, which is another long story...

You are also wrong in the specifics of the research report on feedback. Many many people do not understand that feedback corrects errors. They assume falsely that feedback is just as important for correct answers as it is for incorrect answers. Maybe you didn't have this false impression or maybe the way the research report was written provided just the right metaphorical supports to provide you with the correct mental models of how feedback works, but I've been talking about feedback for several years and many people in our field don't understand this basic concept.

Yes, we need the researchers to continue their good work. I wish they were doing even better work in making the research more real-world, more complete, more generalizable; but they are providing a valuable service.

We also need more than one or two people doing translational research, spending years with our heads in the research, spending years talking to learning professionals, trying against hope and exhaustion to figure out what illustrations, metaphors, and examples resonate with those on the front lines of learning. There are so few of us because it is so damned hard to do, and perhaps because we have to swim against the tide of the research bathwater as it is thrown in our faces.

Mark, thank you again for your kind words and your thoughtful analysis. Keep up your advocacy. We need more people who fully understand the challenges we face in improving the current state of learning.

Mark Frank

Thanks for such a prompt reply. Unless you do a gig in the UK I am unlikely to ever hear you live, so this forum will have to do.

I most emphatically do not want to throw out the research baby with the bathwater. But I also am nervous of making this baby head of the family. Just like our own experience, research can be limited to a specific context and lead to wrong conclusions. My question is how to combine the two.

Your model seems to be that research will throw up the deep hypotheses that can then be tested out in the reality of experience. You say “research is the first step”. But I would argue that as often as not it is the other way round. Reflecting on our experience gives us deep insights into what is going on when we learn; insights which we can test out against research results as well as our own experience. I don't see many fundamental additional and reliable insights coming out of research. There are exceptions e.g. the limits on working memory and value of spacing out learning. But its big value (so far) is to act as a check on the assumptions and beliefs that arise from our reflections on our own experience.

The relative value of feedback on correct or wrong answers is an excellent example (unless I have totally misunderstood what you wrote). You show that research has concluded that it is usually more valuable to give feedback on wrong answers than correct answers, but there is some value in giving feedback on correct answers if the subject is not certain they are correct. As I understand it, in almost all these cases the value of feedback is determined by giving a test, then some feedback, then testing again. In this context it is surely obvious that

(a) If I got it right first time and I am sure I got it right, then I will almost certainly get it right the second time with or without feedback i.e. feedback makes no difference.

(b) If I got it right first time but I am not sure I got right, then I will very likely get it right again – but feedback reduces my chance of changing my mind.

(c) If I got it wrong and I am told why I got it wrong, then I am very likely to change my mind and get it right next time i.e. feedback makes a big difference.

(d) If I got it wrong and no one tells me I got it wrong, I am very likely to get it wrong again!

It seems to me that this logic makes these the results utterly predictable and explains why. (As an experiment I asked my son who is currently studying for his A levels whether it is more valuable to be told about the answers he got right in practice tests or the ones he got wrong. He gave me the pained look teenagers reserve for their ageing parents, and replied “what’s the point in being told about the stuff you already know?”).

You say that many learning professionals believe feedback on correct answers is more important than feedback on wrong answers. I cannot believe they would not accept the logic above and suspect they have a different reason for valuing feedback on correct answers. If your teacher only dwells on your failures and fails to acknowledge your successes, then it becomes demotivating in the long run. I am sure that is true - but I base it on my experience. I think it would be hard to test academically :-)

ErnestO Stolpe

My comment is in reference to item 21 in the research paper "Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts".

In Formal On Job Training (FOJT) the best feedback is given in real world problem solving activites. The learner is completely engaged and in failing to recall the solution is at that moment ready to thrust the trainers/mentors feedback (solution) into long term memory.

chhabi Ghimire

I'studying LLM in Nepal and it is me necessary to much


I think that there is not a pretty good idea to accomplish the definition essay by your own efforts! I think, that would be better to order the good essay at the custom sociology essays service, just because that will save your time.

workout dvd

It sounds well for this paper.

coach handbags wholesale

Good check list, I am running through it with my sites now .

coach handbags wholesale

Very interesting article. It’s funny how history can be twisted in so many different ways. These photos certainly give us clues, but I guess we’ll never know the true story. . . .

cheap fashion clothes

I have never read such a wonderful article and I am coming back tomorrow to continue reading.

home loans

Set your life more simple take the home loans and all you require.

north face backpack sale

It sounds well for this paper.

Mac Cosmetics

Colour plus texture for the lips. Stands out on the runway. Simmers on the street! What made M·A·C famous.

ralph lauren outlet

Good day! Would you mind if I share your blog with my myspace group? There’s a lot of folks that I think would really enjoy your content. Please let me know. Cheers… http://www.hipoloshirts.com

cheap louie vutton underwear

Your article swept me away with its vast information and great writing. Thank you for sharing your views with such passion. I like your views.

cheap hats

Thank landlord impartiality. Come up with such a good article to share with us. This is the official information from articles I and my friends need. Thanks again!

cheap clothes online

It makes me feel so surprise.

retro jordans

It's so great Get a move on now!

retro jordans

Get a move on now!

oil press

I'm a trainee teacher and I've lost track of how many of my lecturers have presented us with those figures. I shall post this up on our noticeboard.

New Era Hats

No one can succeed without any hard work. Karl Max was successful, because he spent more than 30 years writing the book "Communist Manifesto"; Tomas Edison succeeded, because he had experimented thousands of times to find the best material for lights. Every success calls for hard work. If you want to suc-ceed, work hard first.

Cheap New Era Hats

Great post.Thank you for sharing!

Wholesale New Era Hats

Great post.he article is informative and witty at the same time.

Pellet Mill

Whatever is the explanation very open and very clearly the issues.

hermes birkins

it seems to be insteresting for your paper here.


This is central, obviously, during landing approach but is also a security edge if the crew were ever to become unfocused and allow the airspeed to decay in any phase of flight.

Dried Fruits

If not, then all we've done is questioned published results, without an alternative hypothesis.

Moncler Jackets

Very well said, your blog says it all about that particular topic.

Red  bull caps

This article is exactly what I needed, thank you for sharing. I would recommend friends over to read. I hope you can keep writing more and more good articles. This is really useful to me.


Everyone is happy with that inning, whether you like homers, endurance, hard singles with men on foot, hits with runners in scoring location, you name it.

The comments to this entry are closed.

Search Will's Blog

  • Google
    This Blog Web


Notable Books

Sponsoring Ads (vT1)

Sponsoring Ads (vG2)

Sponsoring Ads (vL3)