Will at Work Learning

FREE Research Report on Feedback

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops.

And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right?

You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times.

Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback.

Here are some of the insights from the two-part 88-page research report:

1. The most important thing to remember about feedback is that it is generally beneficial for learners.
2. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the “whys” and “wherefores.”
3. The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning.
4. Feedback works by correcting errors, whether those errors are detected or hidden.
5. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval.
6. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities.
7. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback.
8. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval.
9. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice.
10. Delayed feedback creates a beneficial spacing effect.
11. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback.
12. Feedback should usually be provided before learners get another chance to retrieve incorrectly again.
13. Provide feedback on correct responses when:
    a. Learners experience difficulty in responding to questions or decisions.
    b. Learners respond correctly with less-than-high confidence.
    c. All the information learned is of critical importance.
    d. Learners are relatively new to the subject material.
    e. The concepts are very complex.
14. Provide feedback on incorrect responses:
    a. Almost always.
    b. Except:
    i. When feedback would disrupt the learning event.
    ii. When it would be better to wait to provide feedback.
15. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself.
16. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results.
17. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay.
18. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback.
19. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback.
20. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts.
21. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts.

Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below.

The link to download the feedback report

Purchasing eLearning Tools? Consider Adobe!

I'm writing you from the eLearning Guild's annual conference. I went to a session presented by Silke Fleischer and colleagues at Adobe and was blown away by the work Adobe is doing to create products that support learning and related efforts. I then asked a number of industry thought leaders who confirmed my interpretation: Adobe is now a 500-pound Gorilla, likely to continue out-investing their competitors and thus creating better and better products for folks like us to use.

If you're considering elearning tools, you owe it to your organization to consider Adobe products. I have no financial relationship with Adobe, by the way. This is not to say that other products aren't worthy and/or do some things better than Adobe products. My thinking is this: Companies who invest in their products are often more likely to be there for you in the years to come. I've seen many clients who started using a particular tool five-to-ten years ago, and they are basically stuck with it because of their large installed base of learning courses.

Here are a few of the things that made me wake up and take notice:

1. Adobe's update cycle on Captivate seems to be shrinking, as they are aggressively moving forward in the development of Captivate 4.
2. Captivate is being used for many purposes, including the development of Podcasts, Advertising, etc.
3. You can embed a working Captivate file into Adobe connect and then have webinar or online-learning participants each interact with Captivate objects.
4. PDF files can now include fully-functional interactive images. So documents are not static anymore!!
5. Adobe is working on a new platform called AIR, which will enable the compilation of many types of objects for display and interaction.

What Work-Learning Audit Reveals

I recently completed one of the most comprehensive work-learning audits I've ever been asked to do for a major U. S. retailer. The goal of the audit was to find out how their learning programs AND work-learning environment were supporting the stores in being successful. The audit involved (1) structured and unstructured interviewing with all levels of the organization, especially with store personnel, (2) focus groups held across the country with specific groups of store personnel (e.g., clerk, store managers, assistant managers, etc.), (3) task force meetings with senior line managers and representatives throughout the company, (4) learning audits of e-learning courses, (5) learning audits of a mission-critical classroom course, (6) review of company artifacts (CEO messages, publications, databases, intranet, etc.), (7) interviews with learning-and-performance professionals, (8) discussions of business strategy, (9) discussions regarding corporate information and data-gathering capabilities, (10) job shadowing, (11) store observations, etc.

Who/What Do Workers Learn From?

One of the most intriguing results came out of a relatively simple exercise I did with focus-group participants. The following is a rough approximation of those results.

What I did was ask focus-group participants who they learned from. I would hold up a large 6 x 8 index card with a position label on it, for example, "District Manager," "Clerks," or "Corporate." The group would shout out where they thought that card should go on a large diagram I had created on the wall. I would place it on the wall in a particular category based on the verbal responses and then we would negotiate as a group to determine it's final positioning. So for example, participants could say that they learned the following amounts from that person/position, and we often compromised using in-between placement:

• Learned Most
• Learned a Lot
• Learned Some
• Learned a Little
• Learned Least
• Had Little/No Contact with

See the diagram below for a rough example. This one is actually a composite based on several focus groups and more than one position. It gives a fair representation for how frontline retail clerks responded. Note that the orange boxes represent fellow employees, while the blue boxes represent other groups of people or things that they learned from.

There are several key insights from these results:

• People learn the most from those who they work closely with.
• People learn the most from their experience doing the job.
• People learn the most from their self-initiated efforts at learning.
• The more contact, the more learning (for the most part), however there are benefits from learning from experts (e.g., store managers, head clerks), though the worker has to have at least some signicant contact with them to create this benefit. You'll notice that district staff have only a little impact and regional and corporate staff have none.
• E-learning is seen as somewhat facilitative but not a place where workers learn the most. This result may be organization specific as different e-learning designs and implementations might easily move this result higher or lower.

Frontline clerks didn't get much from company magazines and the like, but managers (not represented in the results above) did find value in these. Store managers also reported that networking with other store managers was on of the "Learn Most" entries for them. For this company, this network was even more important than learning from their district managers (their direct bosses). This makes sense because their network is more accessible throughout the heat of the daily grind.

These results were eye-opening for my client, and they are still wrangling with the implications. For example, district managers and district training staff seemed to produce very little learning benefits. So, should their roles for learning be de-emphasized or re-emphasized?

These types of result have to be understood in the larger data-gathering effort of course. Analyzed alone, they suffer from the problem of de-contextualized self-report data. Combined with multiple other data sources, they paint a really robust picture of an organization's learning environment.

Informal Learning, Social Networks, etc.

Vendors are out and about in our field now selling the benefits of complicated and expensive analysis tools for looking at how people learn through so-called informal on-the-job mechanisms. The example above shows that if you don't have the big bucks, there are simpler ways to get good data as well. 

Elliott Masie's LMS Wish List

Elliott Masie came up with a great and very insightful wish list for LMS's. Click here to access it. He even added a few suggestions in the past few days, probably based on feedback from his loyal audience.

I really like the richness that Elliott's suggestions might create for a typical LMS. Most LMS implementations are just a list of course offerings.

On the other hand, I worry about overly complicating options for users. Most workers just don't have extra time to waste. Maybe the suggestion to let users rate the courses comes into play here.

I also worry about user-generated content. It can be great, could be better than what the training folks can create, could engender more engagement, could be bottom line more effective. But we should all recognize that it is a double-edge sword. User generated content could be incorrect, could be a huge waste of time, could cause the organization to leave itself vulnerable to legal liability.

Doesn't Fix the Biggest Problem with the LMS Mentality

The biggest problem with LMS's can't be fixed with Elliott's suggestions. The biggest problem is that the whole LMS face sends a powerful hidden message that "learning" is about taking courses or accessing other learning events. This "Learning Means Sitting" LMS mentality infiltrates whole organizations.

I've seen this recently with one of my clients, a huge retailer, where their LMS has encouraged store managers and other store leaders to focus learning time on taking courses, in lieu of coaching, learning from each other, trying things out and getting feedback, encouraging store employees to take responsibility for particular areas, etc. It's not that they completely ignore these other learning opportunities; it's that the LMS focuses everyones' time and attention on courses, creating a lot of wasted effort.

To get the most from an LMS, you ought to throw away your LMS and start over. People can learn something—develop competencies/skills—from courses or from other means. A competency-management system that offers multiple means to develop oneself is ideal, where courses/events are just one option. I still haven't seen a commercial system that does this though...Most are course first designs.

Maybe I'm too over-the-top recommending that we get rid of all LMS's. I make the statement to highlight the humongous problems that the LMS mentality is causing.

Starbucks to Shutdown for Training.

Starbucks is shutting all its stores on February 26th for three hours to train or retrain its employees.

You can read related articles at the following links:

Seattle Times

Starbuck Gossip (Blog)

Marketplace (Audio, story starts at 19:42 mins:secs)

Street Insider

The question will be, is the training well-designed?

Wednesday morning February 27th will give only a partial answer. March, April, and May will be more important. And of course, maybe this has nothing to do with training at all. Maybe it's a store management problem. Maybe the brand is too diluted, no longer special. Maybe competition from Dunkin' Donuts and McDonalds is creating issues, especially as consumers try to save money in these uncertain economic times.

Will the training work? Stay tuned.

Technology for Measuring Social Interactions. Can We Use This For Learning?

MIT researchers have developed a technology to track people's social interactions, for example, at a conference. Check out this link to learn more.

Can we use such a technology for learning?

Certainly, we could use the technology to help people learn about their current networking tendencies and to give learners feedback as they attempt to change those tendencies. But, what other applications can we brainstorm? Let me give this a try.

1. Leadership Simulations: Does the technology enable better in-basket simulations, or in-basket simulations that are more economical to deploy (because they don't require the same high numbers of observer/consultants to observe interactions and provide feedback? Note: By in-basket simulations, I mean simulations in which many learner/players each play a different role, each have different in-basket tasks to accomplish, and the way they act in the simulation is by talking with other learner/players.
2. On-the-job Leadership Activity Feedback: Imagine a retail store manager who is tasked (partially) with developing his or her people (those who work in the store). The system could track the number of interactions the store manager had with each employee, and the interactions the employees had with each other. This "intelligence" data could be used by a store manager to learn about the number of learning opportunities (i.e., coaching, providing feedback, observing, encouraging, sharing, etc.) that occur in a given period of time. Such data could be compared with "best-practice" store manager data, and store managers could use this information to change their behavior. Admittedly, quantity doesn't equate to quality, but by tracking such social contact, managers might get a start in thinking about increasing the number of "learning opportunities."
3. Organizational Learning. Organizations (or business units, teams, etc.) could track each other's social networks to find out who the most networked folks are. Such information could be utilized to select for job assignments, project roles, etc., or to actively change the observed dynamics (for example, encouraging some people to spend more time in individual productive work while encouraging others to limit their isolation).

Anyway, these are some initial thoughts. I expect some enlightened simulation companies to begin brainstorming ways to use the technology to differentiate their offerings from the competition. In the meantime, can you think of any other learning opportunities inherent in the technology?

Evaluating Learning 2.0

The eLearning Guild has asked me to lead a discussion at their upcoming conference (Guild Annual Gathering) on how we might think about evaluating Learning 2.0 interventions.

I'd welcome your examples and insights.

For those who don't know what "Learning 2.0" means, I'll forgo my cynical answer, and say that others describe Learning 2.0 as learning that enables learner-creation of information, comparing Learning 2.0 to the stereotypic traditional model where the teacher teaches and the learner absorbs the information (or the e-learning delivers content and the learner absorbs).

So, Learning 2.0 is said to include such things as Wikis, Blogs, Learner Portfolios, Media Development and Sharing by Learners, Informal Learning, etc.

Here's a few things I'm contemplating:

1. Traditional metrics are certainly appropriate because bottom-line we want to know if Learning 2.0 interventions produce learning, enable on-the-job performance, and produce desirable individual and organizational outcomes.
2. Comparisons to other methods of learning. Especially important to see if Learning 2.0 methods (on the positive side) create more elaborate mental models, produce more satisfaction, etc. and (on the negative side) waste time, create unproductive distractions, communicate incorrect or inappropriate information.
3. We need to measure not only what HAS been learned, but also on what MAY BE LEARNED IN THE FUTURE. It could be, for instance, that Learning 2.0 is inefficient for learning anything specifically, but enables faster future learning in the same area of inquiry.

Here's where I can use your help. Let me know if you know of any of the following:

• Rigorous research studies on Learning 2.0 interventions
• Anecdotal evidence on Learning 2.0 interventions

Better yet, join me at the eLearning Guild's Annual Conference -- Specifically the Learning Management Colloquim and discuss this in real time.