Will at Work Learning

FREE Research Report on Feedback

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops.

And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right?

You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times.

Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback.

Here are some of the insights from the two-part 88-page research report:

1. The most important thing to remember about feedback is that it is generally beneficial for learners.
2. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the “whys” and “wherefores.”
3. The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning.
4. Feedback works by correcting errors, whether those errors are detected or hidden.
5. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval.
6. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities.
7. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback.
8. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval.
9. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice.
10. Delayed feedback creates a beneficial spacing effect.
11. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback.
12. Feedback should usually be provided before learners get another chance to retrieve incorrectly again.
13. Provide feedback on correct responses when:
    a. Learners experience difficulty in responding to questions or decisions.
    b. Learners respond correctly with less-than-high confidence.
    c. All the information learned is of critical importance.
    d. Learners are relatively new to the subject material.
    e. The concepts are very complex.
14. Provide feedback on incorrect responses:
    a. Almost always.
    b. Except:
    i. When feedback would disrupt the learning event.
    ii. When it would be better to wait to provide feedback.
15. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself.
16. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results.
17. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay.
18. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback.
19. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback.
20. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts.
21. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts.

Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below.

The link to download the feedback report

What Prevents the Use of Research

What prevents people in the learning-and-performance field from utilizing proven instructional-design knowledge?

This is an update to an old newsletter post I wrote about in 2002. Most of it is still relevant, but I've learned a thing or two in the last few years.

Back in 2002, I spoke with several very experienced learning-and-performance consultants who have each---in their own way---asked the question above. In our discussions, we've considered several options, which I've flippantly labeled as follows:

1. They don't know it. (They don't know what works to improve instruction.)
2. They know it, but the market doesn't care.
3. They know it, but they'd rather play.
4. They know it, but don't have the resources to do it.
5. They know it, but don't think it's important.

Argument 1.
They don't know it. (They don't know what works to improve instruction.)
Let me make this concrete. Do people in our field know that meaningful repetitions are probably our most powerful learning mechanism? Do they know that delayed feedback is usually better than immediate feedback? That spacing learning over time facilitates retention. That it's important to increase learning and decrease forgetting? That interactivity can either be good or bad, depending on what we're asking learners to retrieve from memory? One of my discussants suggested that "everyone knows this stuff and has known it since Gagne talked about it in the 1970's."

Argument 2.
They know it, but the market doesn't care.
The argument: Instructional designers, trainers, performance consultants and others know this stuff, but because the marketplace doesn't demand it, they don't implement what they know will really work. This argument has two variants: The learners don't want it or the clients don't want it.

Argument 3.
They know it, but they'd rather play.
The argument: Designers and developers know this stuff, but they're so focused on utilizing the latest technology or creating the snazziest interface, that they forget to implement what they know.

Argument 4.
They know it, but don't have the resources to use it.
The argument: Everybody knows this stuff, but they don't have the resources to implement it correctly. Either their clients won't pay for it or their organizations don't provide enough resources to do it right.

Argument 5.
They know it, but don't think it's important.
The argument: Everybody knows this stuff, but instructional-design knowledge isn't that important. Organizational, management, and cultural variables are much more important. We can instruct people all we want, but if managers don't reward the learned behaviors, the instruction doesn't matter.

My Thoughts In Brief

First, some data. On the Work-Learning Research website we provide a 15-item quiz that presents people with authentic instructional-design decisions. People in the field should be able to answer these questions with at least some level of proficiency. We might expect them to get at least 60 or 70% correct. Although web-based data-gathering is loaded with pitfalls (we don't really know who is answering the questions, for example), here's what we've found so far: On average, correct responses are running at about 30%. Random guessing would produce 20 to 25% correct. Yes, you've read that correctly---people are doing a little bit better than chance. The verdict: People don't seem to know what works and what doesn't in the way of instructional design.

Some additional data. Our research on learning and performance has revealed that learning can be improved through instruction by up to 220% by utilizing appropriate instructional-design methods. Many of the programs out there do not utilize these methods.

Should we now ignore the other arguments presented above? No, there is truth in them. Our learners and clients don't always know what will work best for them. Developers will always push the envelope and gravitate to new and provocative technologies. Our organizations and our clients will always try to keep costs down. Instruction will never be the only answer. It will never work without organizational supports.

What should we do?

We need to continue our own development and bolster our knowledge of instructional-design. We need to gently educate our learners, clients, and organizations about the benefits of good instructional design and good organizational practices. We need to remind technology's early adopters to remember our learning-and-performance goals. We need to understand instructional-design tradeoffs so that we can make them intelligently. We need to consider organizational realities in determining whether instruction is the most appropriate intervention. We need to develop instruction that will work where it is implemented. We need to build our profession so that we can have a greater impact. We need to keep an open mind and continue to learn from our learners, colleagues, and clients, and from the research on learning and performance.

New Thoughts in 2006

All the above suggestions are worthy, but I have two new answers as well. First, people like me need to do a much better job (me included) communicating research-based ideas. We need to figure out where the current state of knowledge stands and work the new information into that tapestry in a way that makes sense to our audiences. We also have to avoid heavy-handedness in sharing research-based insights, as we must realize that research is not the only means of moving us toward more effective learning interventions.

Secondly, I have come to believe that sharing research-based information like this is not enough. If the field doesn't get better feedback loops into our instructional-design-and-development systems, then nothing much will improve over time, even with the best information presented in the most effective ways.

New Educational Research Organization Forming

A new educational research organization is forming as a counterpart to AERA (the American Educational Research Association). I get the impression that the impetus for this is that too much of the current research on education has the flaw that it doesn't really care about cause and effect.

Since cause-and-effect are critical---because it's the only way to really find out what the critical factors are, I'm hoping for great things from this organization.

It's name is: Society for Research on Educational Effectiveness

Wrong About Cell Phones and Driving

Well, it looks like one of my previous brainstorms was wrong. Check out this link from the American Psychological Association on cell-phone use while driving. Initial research on cell phones while driving seems to suggest that cell phones Do hurt driving. Still not sure if drivers can learn to use cell phones more effectively while driving.

Cell Phones While Driving

There have been several published studies (and even more newspaper articles) that show cell-phone use while driving is correlated with accidents. The suggestion from these studies is that cell phones CAUSE accidents. The implication is that we should ban cell phones while driving.

This may be true. I was scared to death last week while my taxi driver was looking at his cell phone to dial numbers. He clearly did not have his eyes on the road. If anything unusual occurred (like the van in the next lane entering our lane right in front of us---watch out please watch out!), his reaction time would have been considerably slowed and we would have been much more likely to have an accident.

On the other hand, I wonder how much of the current problems are caused by a learning deficit. After all, for most of us cell phones are rather new. More importantly, driving while using a cell phone is also new. This kind of multitasking can be learned. There are research studies that show that experience doing multitasking can increase performance on the tasks being done. With enough practice, less working-memory capacity is needed, freeing up capacity to engage in the various tasks.

One hypothesis suggested by this is that cell-phone-related accidents will decrease with time as drivers get more practice using their cell phones while driving. Judging from the number of people I see driving and phoning, not many people are heeding the warnings, so lots of people are gaining more experience. Cell-phone accident rates will also decline as new technologies are utilized, namely voice-dialing and hands-free cell-phones.

On the other hand, a second hypothesis is that anything that prompts drivers to take their eyes off the road will produce similar deficits to cell-phone driving. Here’s a short list:

1. People who read maps while driving.
2. People who look at the radio to tune to a particular station.
3. People who glance at the person sitting next to them while in conversation.
4. People who look at their food before stuffing it in their mouths.
5. People who admire the scenery.
6. People who rubberneck at accident scenes.

People who look at their cell phones to dial a number are just asking for trouble. It probably helps to have two hands on the wheel, as well.

I’d be willing to bet that for most people fewer accidents will occur when using a hands-free, voice-dialing cell phone than when talking with someone sitting beside them in the front seat, assuming equal levels of experience doing both. The natural human tendency to want to look someone in the eyes while talking to them will prompt most of us to try and steal a glance at our conversational partners, increasing slightly the danger from unforeseen events.

Like most things in life, learning plays a central role in our cell-phone-while-driving performance. Like most things for us humans, our cognitive machinery sets the boundaries for this performance.

New Information from the Research (An Update on My Thinking)

Although I still wonder about our ability to learn how to utilize cell phones while driving, recent research suggests that right now, we are not too good at it. Check out my updated post on this.