Will at Work Learning

FREE Research Report on Feedback

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops.

And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right?

You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times.

Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback.

Here are some of the insights from the two-part 88-page research report:

1. The most important thing to remember about feedback is that it is generally beneficial for learners.
2. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the “whys” and “wherefores.”
3. The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning.
4. Feedback works by correcting errors, whether those errors are detected or hidden.
5. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval.
6. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities.
7. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback.
8. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval.
9. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice.
10. Delayed feedback creates a beneficial spacing effect.
11. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback.
12. Feedback should usually be provided before learners get another chance to retrieve incorrectly again.
13. Provide feedback on correct responses when:
    a. Learners experience difficulty in responding to questions or decisions.
    b. Learners respond correctly with less-than-high confidence.
    c. All the information learned is of critical importance.
    d. Learners are relatively new to the subject material.
    e. The concepts are very complex.
14. Provide feedback on incorrect responses:
    a. Almost always.
    b. Except:
    i. When feedback would disrupt the learning event.
    ii. When it would be better to wait to provide feedback.
15. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself.
16. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results.
17. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay.
18. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback.
19. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback.
20. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts.
21. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts.

Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below.

The link to download the feedback report

Human Cognition is More Reactive Than You Think

In today's New York Times there is a great article, Who's Minding the Mind?, by BENEDICT CAREY that sums up a large number of research studies on human cognition that show that human beings are more reactive than we might think. We tend to believe that we, as human beings, are very proactive and consciously in control of our thoughts and actions; but these studies show that much of what we do and think is due to hard-wired, often unconscious processes.

For example, the article cites how sitting near a briefcase (as opposed to a backpack) can make people more competitive. Or as Carey writes:

New studies have found that people tidy up more thoroughly when there’s a faint tang of cleaning liquid in the air; they become more competitive if there’s a briefcase in sight, or more cooperative if they glimpse words like “dependable” and “support” — all without being aware of the change, or what prompted it.

This basic fact about human behavior is relevant to the learning and performance field, of course. One of the things I've talked about for years is the notion of "spontaneous remembering." If we create learning right, we're more likely to help our learners—when they're on the job at a later time—by helping them spontaneously trigger memories of what they've learned. We can do this best by requiring our learners to utilize realistic cues in the learning context in making real-world decisions and taking real-world actions. This is why simulations are so effective (if they are well designed).

When learners process learning objectives or prequestions before encountering learning material, the learners are primed to pay attention to relevant learning material. It's not necessarily a conscious process, but it works.

There are many examples available, but here's another point: The learner-centric movement of the 1990's and 2000's has relied too heavily on the notion that the learners always know best, and that they are in conscious control of their learning and we just need to let them make the best decisions.

When we realize that our learners are more deterministically driven than the we want to believe (its about free will a little, isn't it?), we have more work to do if we really want to drive maximum performance. Even when our clients consciously want to do something, we may be able to help them reach their goals by setting up learning and performance situations that unconsciously trigger the behavior they want to achieve.

The Magic Question for Learning and Instructional Design

The most important question that instructional designers can ask is:

“What do learners need to be able to do, and in what situations do they need to do those things?”

While we might discount such a simple question as insignificant, the question brilliantly forces us to focus on our ultimate goals and helps us to align our learning interventions with the human learning system.

Too many of us design with a focus on topics, content, knowledge. This tendency pushes us, almost unconsciously, to create learning that is too boring, filled with too much information, and bereft of practice in realistic situations.

The Magic Question requires us to be relevant. For workplace learning, it focuses our thinking toward learners' future job situations. For education learning, it focuses our thinking toward real-world relevance of our academic topics.

The Magic Question in Practice

In practice, the Magic Question forces us to begin our instructional-design efforts by not only creating a list of instructional objectives, but also by creating a list of performance situations. For example, if we're creating leadership training, we not only need to compile objectives like, "For most decisions, it can be helpful to bring your direct reports into decision-making, so as to increase the likelihood that they will bring energy and passion in implementing decisions." We also need to compile a list of situations were this objective is relevant, for example in weekly staff meetings, project meetings, in one-on-one face-to-face conversations, in phone conversations, etc. Also, for general decision making, but not in situations where time is urgent, where safety is an issue, where legal ramifications are evident, etc.

By framing our instructional-design projects in this way, we get to think about our learning designs in ways that are much more action-oriented, relevant, and practical. The framing makes it more likely that we will align our learning and performance contexts, making it more likely that our learners, in their future situations, will spontaneously remember what we've taught them. The framing makes it more likely that we will focus on practice instead of overloading our learners with information. The framing also makes it more likely that we will utilize relevant scenarios that more fully engage our learners. Finally, using the Magic Question forces our SME's (subject-matter experts) to reformulate their expertise into potent practical packages of relevant material. It's not always easy to bend SME's to this discipline, but after the pain, they'll thank you profusely as together you push their content to a much higher level.

Obviously, there is more to be said about how the Magic Question can be integrated into learning-design efforts. On the other hand, as my clients have reported, the Magic Question has within it a simple power to (1) change the way we think about instructional design, and (2) transform the learning interventions we build.

Measuring Learning Results: The New Research-to-Practice White Paper

I've just completed a new research-to-practice white paper. As far as I can tell, it is the first work on learning measurement (assessment and evaluation) that actually takes human learning into consideration. I'd like to thank Questionmark for agreeing to support this work.

Words from the paper's introduction:

In writing this report on using fundamental learning research to inform assessment  design, I am combining two of my passions—learning and the measurement of learning. As an experienced learner and learning designer, I have come to the belief that those of us responsible for designing, developing, and delivering learning interventions are often left in the dark about our own successes and failures. The measurement techniques we use simply do not provide us with valid feedback about our own performances.

The traditional model of assessment utilizes end-of-learning assessments provided to learners in the context in which they learned. This model is seriously flawed, especially in failing to give us an idea of how well our learning interventions are doing in preparing our learners to retrieve information in future situations—the ultimate goal of training and education. By failing to measure our performance in this regard, we are missing opportunities to provide ourselves with valid feedback. We are also likely failing our institutions and our learners because we are not able to create a practice of continuous improvement to maximize our learning outcomes.

This report is designed to help you improve your assessments in this regard. I certainly won’t claim to have all the answers, nor do I think it is easy to create the perfect assessment, but I do believe very strongly that all of us can improve our assessments substantially, and by so doing improve the practice of education and training.

How to buy the paper:

• Click to go to my catalog
• Click to buy the paper directly

Take the Original Learning Research Quiz

For the last three or four years, I have provided a quiz on the Work-Learning Research website so that you can test your knowledge of research-based instructional-design fundamentals. The 15-item quiz presents you with realistic instructional-design scenarios and asks you to make decisions about which design will produce the best learning. The quiz also provides detailed feedback on each question, so that by taking the quiz, you're really getting a great learning opportunity for yourself.

When I compiled the results several years ago, the results were astonishing. I won't tell you how they were astonishing---because I don't want to bias your input---but they were astonishing. I will leave the quiz up for some time, but I'd like to make a big push in the first two weeks of February of 2007 so that I can publish the results on this blog. So, please take the quiz, and encourage everyone you know who is a learning professional (training, learning, instructional design, e-learning, performance improvement, education) to take the quiz now.

Take the Quiz by Clicking Here

Please take the Quiz by February 15th, 2007 so that I can do a large compilation by the end of the month.

Tell All Your Fellow Learning Professionals to take the Quiz. Send them this Link:

www.willatworklearning.com/2007/01/take_the_origin.html

(Hmmm. It's a little long, so you may have to click on it and then copy it from your address bar.)

And feel free to comment on the quiz below.

Using Celebrity to Increase Learning

Unless you've been a hermit since birth, you should have noticed that we human beings are drawn to celebrity. From peasants to the so-called intellectual elite, we are vastly more likely to pay attention to celebrities (and their activities) than to anyone else.

The evidence is everywhere.

• Movie producers hire big name actors to draw people to theaters.
• Yahoo's home page almost always leads with a celebrity.
• Google gets more searches for Johnny Depp than for any of your neighbors.
• Advertisers pay big bucks for celebrity endorsements.
• Celebrities get the largest book advances (and sales).
• Conferences pay big bucks to hire celebrities, even if they have little to say.
• Much of our idle chatter involves talk of celebrities.
• Tournament organizers groan when the big names get ousted or withdrawal due to injury.

I have to admit it myself. While browsing the web, I often find myself clicking through to a story about a famous person. It's almost an unconscious response. I seem to be drawn to want to know what's going on with them.

The New York Times has a nice piece about the power of celebrities to draw interest on the internet.

How Can We Use Celebrity to Increase Learning?

First, we have to distinguish between drawing attention and creating distraction. Using celebrity is a good method for getting more of our learners to initially access the learning message. It might also be useful to maintain attention longer than normal. But we have to watch out for the distraction factor and make sure that the learning message is supreme.

I'd actually like to see a study that tests my little theory.

• Hypothesis 1: Celebrity helps garner attention, AND increases long-term retrieval of the main learning points.
• Hypothesis 2: Celebrity helps garner attention, AND decreases long-term retrieval of the main learning points (but increases long-term retrieval of information about the celebrity or the learning event itself).
• Hypothesis 3: Celebrity does nothing. We only think it does.

The research would have to be very careful to avoid experimenting with captured learners. Why? Because the hypothesized benefit is due to more people attending to the learning event in the first place. If everyone, regardless of whether we use a celebrity or not, is captured, there are less likely to be learning benefits.

In a natural, authentic training environment, I'm betting on Hypothesis #1 above.

If anybody has learners and funds to test this theory, give me a call. Let's get to work.

Training Implementations

Some of us use celebrity already. I've seen e-learning programs from Type A Learning Agency that used in-company celebrities to reinforce messages within e-learning programs.

Celebrity doesn't have to reside only in individuals either. Executives love being sent off the Harvard Business School events, for example.

Elliott Masie--the training industy's P. T. Barnum--uses his celebrity to draw people to his conferences, which are pretty good learning events after all.

Local ASTD, ISPI, SHRM (etc.) chapters use celebrity speakers to draw a crowd at their sessions.

More and more, executives (in-company celebrities) are teaching leadership courses to company managers.

The Bottom Line

It's worth considering how we might use the concept of celebrity to draw learners to our learning initiatives and increase their attention on our learning messages.

Bloom's Taxonomy Problems

The Bloom is Off the Vine

I just came across this nifty little piece on Bloom's Taxonomy, written by Brenda Sugrue for ISPI's Performance Express.

It's a nice critique on the validity and usefulness of Bloom's Taxonomy for Instructional Design.

Read it here.

I tend to agree with Brenda's Critique. For a long time I've been suspicious of Blooms.

New Taxonomy for Learning Objectives

Let me propose a new taxonomy for learning objectives.

This taxonomy is needed to clear up the massive confusion we all have about the uses and benefits of learning objectives. I have tried to clarify this in the past in some of my conference presentations—but I have not been successful. When I get evaluation-sheet comments like, "Get real you idiot!" from more than a few people, I know I've missed the mark. SMILE

Because I don't give up easily—and because learning objectives are so vitally important—I'm going to give this another try. Your feedback is welcome.

The premise I'm working from is simple. Instructional professionals use learning objectives for different purposes—even for different audiences. Learning objectives are used to guide the attention of the learner toward critical learning messages. Learning objectives are used to tell the learner what's in the course. They are used by instructional designers to guide the design of the learning. They are used by evaluation designers to develop metrics and assessments.

Each use requires its own form of learning objective. Doesn't it seem silly to use the exact same wording regardless of the use or intended audience? Do we provide doctors and patients with the exact same information about a particular prescription drug? Do designers of computer software require the same set of goal statements as users of that software? Do creators of films need to have the same set of objectives as movie goers?

Until recently I have argued that we ought to delineate between objectives for learners and objectives for designers. This was a good idea in principle, but it still left people confused because it didn't cover all the uses of objectives. For example, learners can be presented with objectives to help guide their attention or to simply give them a sense of the on-the-job performance they'll be expected to perform. Instructional designers can utilize objectives to guide the design process or to develop evaluations.

The New Taxonomy

1. Focusing Objective
   A statement presented to learners before they encounter learning material—provided to help guide learner attention the most important aspects of that learning material.
2. Performance Objective
   A statement presented to learners before they encounter learning material—provided to help learners get a quick understanding of the competencies they will be expected to learn.
3. Instructional-Design Objective
   A statement developed by and for instructional designers to guide the design and development of learning and instruction.
4. Instructional-Evaluation Objective
   A statement developed by and for program evaluators (or instructional designers) to guide the evaluation of instruction.

I made a conscious decision not to include a "table-of-contents objective" despite the widespread use of this method for presenting learners with objectives. I can't decide whether this should be included. There's no direct research on this (that I've encountered), but there may be some benefit for learners in having an outline of the coming learning material. Your comments welcome. I'm leaning toward including this notion into the taxonomy because it is a stategy that I've seen in use. Maybe I'll call them "Content-Outlining Objectives" or "Outlining Objectives."

One of the clear benefits of this taxonomy is that it separates Focusing Objectives from the other objectives. These objectives—those presented to learners to help focus their attention—have been researched with the greatest vigor. And the results of that research are clear:

1. Focusing objectives guide learner attention to the information in subsequent learning material that has been targeted by objectives, but they also take attention away from the information not targeted by objectives.
2. Similarly, focusing objectives improve learning for the targeted information and hurt learning for the information not targeted.
3. Prequestions are as powerful in creating this focusing effect as learning objectives, and they may be more powerful.
4. The wording of the focusing objective or prequestion must specifically mirror the wording in the learning material. General or abstract wording doesn't cut it.
5. Adding extra words, particularly words that specify the criteria of performance (ala Mager) will actually distract learners and hurt learning.

Aligning the Learning and Performance Contexts

Research from the world's preeminent refereed journals on learning and instruction shows that by aligning the learning and performance contexts, learning results can be improved by substantial amounts. In fact, it is this alignment that makes simulations effective, that creates the power behind hands-on training, and that enables action learning to produce its most profound effects.

The research suggests the following points related to instructional design:

1. When humans learn, we absorb both the instructional message and background stimuli and integrate them into memory so that they become interconnected.

2. Humans in their performance situations are reactive beings. Our thoughts and actions are influenced by stimuli in our surrounding environment. If cues in our environment remind us of what we previously learned, we'll remember more.

3. These first two principles can combine to aid remembering, and hence performance in powerful ways. If during the learning situation we can connect the key learning points to background stimuli that will be observed in the learner's on-the-job performance situation, than these stimuli will remind learners of what they previously learned!

4. The more the learning context mirrors the real-world performance context, the greater the potential for facilitating remembering. When the learning and performance contexts include similar stimuli, we can say they are "aligned."

5. The more learners pay attention to the background contextual stimuli, the higher the likelihood of obtaining context effects.

6. Context effects can take many forms. People who learn in one room will remember more in that room than in other rooms. People who learn a topic when they are sad, will remember more about that topic when they are sad. People who learn while listening to Mozart will retrieve more information from memory while listening to Mozart than listening to jazz. People who learn a fact while smelling peppermint will be better able to recall that fact while smelling peppermint than while smelling another fragrance. People who learn in the presence of their coworkers will remember more of what they learned in the presence of those coworkers.

7. Context can aid remembering and performance, but it can have negative effects when aspects of the learning context are not available in the on-the-job performance context.

8. Context effects can be augmented by prompting learners to focus on the background context. Context effects can be diminished by prompting learners to focus less on the background context.

9. The fewer the background contextual elements per learning point, the more powerful the context effects.

10. The easiest and most effective way to align the learning and performance contexts is to modify the learning context. But other options are available as well.

11. The performance context can be modified through management involvement, performance-support tools, and other reminding devices.

12. When the performance context cannot be determined in advance---or when the learned tasks will be performed in many contexts---multiple learning contexts can facilitate later memory retrieval and performance.

13. Learners in their performance situations can improve the recall of what they learned by visualizing the learning situation.

14. Cues can be added to the both the learning contexts and the performance context to aid remembering.

15. Context effects have their most profound impact when other retrieval cues are not available for use. For example, context effects typically do not occur on multiple-choice tests or for other performance situations where learners are provided with hints.

16. To fully align the learning and performance contexts, instructional practice should include opportunities for learners to face all four aspects of performance, (1) situation, (2) evaluation, (3) decision, and (4) action. To create the best results, learners must be faced with realistic situations, make sense of them, decide what to do, and then practice the chosen action.

To read more about this fundamental learning factor (or to see the research behind these suggestions), you can access an extensive report from the Work-Learning Research catalog.