This blurb is reprised from an earlier Work-Learning Research Newsletter, circa 2002. These "classic" pieces are offered again to make them available permanently on the web. Also, they're just good fun. I've added an epilogue to the piece below.
"What prevents people in the learning-and-performance field from utilizing proven instructional-design knowledge?"
Recently, I've spoken with several very experienced learning-and-performance consultants who have each---in their own way---asked the question above. In our discussions, we've considered several options, which I've flippantly labeled as follows:
They don't know it. (They don't know what works to improve instruction.)
Let me make this concrete. Do people in our field know that meaningful repetitions are probably our most powerful learning mechanism? Do they know that delayed feedback is usually better than immediate feedback? That spacing learning over time facilitates retention. That it's important to increase learning and decrease forgetting? That interactivity can either be good or bad, depending on what we're asking learners to retrieve from memory? One of my discussants suggested that "everyone knows this stuff and has known it since Gagne talked about it in the 1970's."
They know it, but the market doesn't care.
The argument: Instructional designers, trainers, performance consultants and others know this stuff, but because the marketplace doesn't demand it, they don't implement what they know will really work. This argument has two variants: The learners don't want it or the clients don't want it.
They know it, but they'd rather play.
The argument: Designers and developers know this stuff, but they're so focused on utilizing the latest technology or creating the snazziest interface, that they forget to implement what they know.
They know it, but don't have the resources to use it.
The argument: Everybody knows this stuff, but they don't have the resources to implement it correctly. Either their clients won't pay for it or their organizations don't provide enough resources to do it right.
They know it, but don't think it's important.
The argument: Everybody knows this stuff, but instructional-design knowledge isn't that important. Organizational, management, and cultural variables are much more important. We can instruct people all we want, but if managers don't reward the learned behaviors, the instruction doesn't matter.
My Thoughts In Brief
First, some data. On the Work-Learning Research website we provide a 15-item quiz that presents people with authentic instructional-design decisions. People in the field should be able to answer these questions with at least some level of proficiency. We might expect them to get at least 60 or 70% correct. Although web-based data-gathering is loaded with pitfalls (we don't really know who is answering the questions, for example), here's what we've found so far: On average, correct responses are running at about 30%. Random guessing would produce 20 to 25% correct. Yes, you've read that correctly---people are doing a little bit better than chance. The verdict: People don't seem to know what works and what doesn't in the way of instructional design.
Some additional data. Our research on learning and performance has revealed that learning can be improved through instruction by up to 220% by utilizing appropriate instructional-design methods. Many of the programs out there do not utilize these methods.
Should we now ignore the other arguments presented above? No, there is truth in them. Our learners and clients don't always know what will work best for them. Developers will always push the envelope and gravitate to new and provocative technologies. Our organizations and our clients will always try to keep costs down. Instruction will never be the only answer. It will never work without organizational supports.
What should we do?
We need to continue our own development and bolster our knowledge of instructional-design. We need to gently educate our learners, clients, and organizations about the benefits of good instructional design and good organizational practices. We need to remind technology's early adopters to remember our learning-and-performance goals. We need to understand instructional-design tradeoffs so that we can make them intelligently. We need to consider organizational realities in determining whether instruction is the most appropriate intervention. We need to develop instruction that will work where it is implemented. We need to build our profession so that we can have a greater impact. We need to keep an open mind and continue to learn from our learners, colleagues, and clients, and from the research on learning and performance.
Will's New Thoughts (November 2005)
I started Work-Learning Research in 1998 because I saw a need in the field to bridge the gap between research and practice. In these past seven years, I've made an effort to compile research and disseminate it, and though partly successful, I often lament my limited reach. Like most entrepreneurs, I have learned things the hard way. That's part of the fun, the angst, and the learning.
In the past few years, the training and development field has gotten hungrier and hungrier for research. I've seen this in conferences where I speak. The research-based presentations are drawing the biggest crowds. I've seen this in the increasing number of vendors who are highlighting their research bonafides, whether they do good research or not. I've seen this recently in Elliott Masie's call for the field to do more research.
This hunger for research has little to do with my meager efforts at Work-Learning Research. Though sometimes in my daydreams I like to think I have influenced at least some in the field---maybe even some opinion leaders. As a data-first empiricist, the evidence is clear. I know that my efforts are often under the radar. Ultimately, this is unimportant. What is important is what gets done.
I'm optimistic. Our renewed taste for research-based practice provides an opportunity for all of us to keep learning and to keep sharing with one another. I've got some definite ideas about how to do this. I know many of you who read this do too. We may not---as a whole industry---use enough research-based practices, but there are certainly some individuals and organizations who are out there leading the way. They are the heroes, for it is they who are out there taking risks, asking for the organizational support most of us don't ask for, making a difference one mistake at at time.
One thing we need to spur this effort is a better feedback loop. If we don't go beyond the smile sheet, we're never going to improve our practices. We need feedback on whether our learning programs are really improving learning and long-term retrieval. Don't think that just because you are out there on the bleeding edge that you're championing the revolution. You need to ensure that your efforts are really making things better---that your devotion is really improving learning and long-term retention. If you're not measuring it, you don't really know.
Let me end by saying that research from refereed journals and research-based white papers should not be the only arbiter of what is good. Research is useful as a guide---especially when our feedback loop is so enfeebled and organizational funds for on-the-job learning measurement are so impoverished.
It would be better, of course, that we might all test our instructional designs in their real-world contexts. Let us more toward this, drawing from all sources of wisdom, dipping our ladles into the rich research-base and the experiences of those who measure their learning efforts.
Research from the world's preeminent refereed journals on learning and instruction shows that by aligning the learning and performance contexts, learning results can be improved by substantial amounts. In fact, it is this alignment that makes simulations effective, that creates the power behind hands-on training, and that enables action learning to produce its most profound effects.
The research suggests the following points related to instructional design:
1. When humans learn, we absorb both the instructional message and background stimuli and integrate them into memory so that they become interconnected.
2. Humans in their performance situations are reactive beings. Our thoughts and actions are influenced by stimuli in our surrounding environment. If cues in our environment remind us of what we previously learned, we'll remember more.
3. These first two principles can combine to aid remembering, and hence performance in powerful ways. If during the learning situation we can connect the key learning points to background stimuli that will be observed in the learner's on-the-job performance situation, than these stimuli will remind learners of what they previously learned!
4. The more the learning context mirrors the real-world performance context, the greater the potential for facilitating remembering. When the learning and performance contexts include similar stimuli, we can say they are "aligned."
5. The more learners pay attention to the background contextual stimuli, the higher the likelihood of obtaining context effects.
6. Context effects can take many forms. People who learn in one room will remember more in that room than in other rooms. People who learn a topic when they are sad, will remember more about that topic when they are sad. People who learn while listening to Mozart will retrieve more information from memory while listening to Mozart than listening to jazz. People who learn a fact while smelling peppermint will be better able to recall that fact while smelling peppermint than while smelling another fragrance. People who learn in the presence of their coworkers will remember more of what they learned in the presence of those coworkers.
7. Context can aid remembering and performance, but it can have negative effects when aspects of the learning context are not available in the on-the-job performance context.
8. Context effects can be augmented by prompting learners to focus on the background context. Context effects can be diminished by prompting learners to focus less on the background context.
9. The fewer the background contextual elements per learning point, the more powerful the context effects.
10. The easiest and most effective way to align the learning and performance contexts is to modify the learning context. But other options are available as well.
11. The performance context can be modified through management involvement, performance-support tools, and other reminding devices.
12. When the performance context cannot be determined in advance---or when the learned tasks will be performed in many contexts---multiple learning contexts can facilitate later memory retrieval and performance.
13. Learners in their performance situations can improve the recall of what they learned by visualizing the learning situation.
14. Cues can be added to the both the learning contexts and the performance context to aid remembering.
15. Context effects have their most profound impact when other retrieval cues are not available for use. For example, context effects typically do not occur on multiple-choice tests or for other performance situations where learners are provided with hints.
16. To fully align the learning and performance contexts, instructional practice should include opportunities for learners to face all four aspects of performance, (1) situation, (2) evaluation, (3) decision, and (4) action. To create the best results, learners must be faced with realistic situations, make sense of them, decide what to do, and then practice the chosen action.
To read more about this fundamental learning factor (or to see the research behind these suggestions), you can access an extensive report from the Work-Learning Research catalog.
Learning is a many-splendored thing. Want evidence? Consider the overabundance of theories of learning. Greg Kearsley has a nice list. To me, this overabundance is evidence that the human learning system has not yet been lassoed and cataloged with any great precision. Ironic that DNA is easier to map than learning.
Being a political junkie, I'm fascinated with how a population of citizens learns about their government and the societal institutions of power. Democracy is rooted in the idea that we the citizenry have learned the right information to make good decisions. In theory this makes sense, while in practice imperfect knowledge is the norm. This discussion may relate to learning in the workplace as well.
Take one example from recent events. On September 11th, 2001, the United States was attacked by terrorists. The question arose, who were these terrorists? Who sent them? Who helped them? One particular question was asked. "Was Saddam Hussein (dictator of Iraq) involved?" I use this question because there is now generally-accepted objective evidence that Saddam Hussein was not involved in the 9/11 attack in any way. Even President Bush has admitted this. On September 17th, 2003, Bush said, in answer to a question from a reporter, "No, we've had no evidence that Saddam Hussein was involved with September the 11th." Despite this direct piece of information, the Bush administration has repeatedly implied, before and after this statement, that the war in Iraq is a response to 9/11. We could discuss many specific instances of this---we could argue about this---but I don't want to belabor the point. What I want to get at is how U.S. citizens learned about the reality of the question.
Take a look at polling data, which I found at PollingReport.com. I've marked it up to draw your eyes toward two interesting realities. First, look at the "Trend" data. It shows that we the citizens have changed our answer to the question asked over time. In September of 2002, 51% of Americans incorrectly believed that Saddam was personally involved in September 11th. Last month in October or 2005, the number had dived to 33%. The flip side of this showed that 33% correctly denied any link between Saddam and 9/11 in October of 2002, while today the number is a more healthy 55% correct, but still a relatively low number. If we think in terms of school-like passing-grade cutoffs, our country gets a failing grade.
The second interesting reality is how different groups of people have "Different Realities" about what is true. You'll notice the difference in answering these questions between Republicans and Democrats.
These data encourage me to conclude or wonder about the following:
These things are too big for me to understand. But lots of the same issues are relevant to learning in organizations---both formal training and informal learning.
Most of what we call "training" is designed with the intention of improving people's performance on the job. While it is true that much of training does not do this very well, it is still true that on-the-job performance is the singular stated goal of training.
But something is missing from this model. What's missing is that a learning intervention can also prepare learners for future on-the-job learning. Let's think this through a bit.
People on the job---people in any situation---are faced with a swarm of stimuli that they have to make sense of. Their mental models of how the world works will determine what they perceive. I've noticed this myself when I walk in the woods with experienced bird watchers. I hear birds, but can't see them, no matter how hard I look. Experienced bird watchers see birds where I see nothing. The same stimuli have different outcomes because the expert birders have superior mental models about where birds might locate themselves.
The same is true for many things. As a better-than-average chess player, I will understand the patterns of the pieces better than a novice will. Experienced computer programmers see things that inexperienced programmers do not. Experienced lawyers will understand the nuances in someone's testimony more than a novice lawyer.
Experience enables distinctions to be drawn between otherwise ambiguous stimuli. It enables people to perceive things that others don't perceive. It helps people notice what others ignore.
Learning can be designed to provide amazing-grace moments, helping those who were once blind to see. If we're serious about on-the-job learning, we ought to begin to build models of how to design formal learning to facilitate informal on-the-job learning.
Dan Schwartz, PhD (a learning psychologist at Stanford) has written recently about a concept called Preparation for Future Learning or PFL. Schwartz argues that generally poor transfer results may be due to the common practice of assessing what was learned but failing to assess what learners are able to learn. This makes a lot of sense given how complex the real world is, how learners forget stuff so quickly, and how much they learn on the job.
Schwartz and his colleagues are working on ways to improve future learning by using "contrasting cases" that enable learners to see distinctions they hadn't previously noticed. This concept might be used in formal training courses to prepare learners to see things they hadn't seen before when they return to the job. For example, a manager being trained on supervisory skills may be taught that some decisions require group input, whereas other decisions require managers to decide on their own. Cases of both types could be provided in training so that relevant distinctions will be better noticed on the job.
A different way to prepare learners for future learning is to prime them with questions. In my dissertation research, I included one experiment that I asked college students questions about campus attractions. For example, I asked them what the statue "Alma Mater" was carrying. A week later, I suprised the students by asking them some of the same questions again. The results revealed that simply asking them questions (even when no feedback was provided) improved how much they paid attention to the items on which they were queried. Between the two sets of questions, learners apparently paid attention to the statue in ways they hadn't before. By being asked about an item, the learners were more likely to spend time learning about that item when they encountered those items in their day-to-day walking around.
There are likely to be other similar learning opportunities, but the point is that we need ways to design our learning interventions to intentionally create these types of learning responses. I'm going to be thinking about this for a while. My hope is that you will too.
Perhaps these meager paragraphs have prepared you for future learning. SMILE.
There have been several published studies (and even more newspaper articles) that show cell-phone use while driving is correlated with accidents. The suggestion from these studies is that cell phones CAUSE accidents. The implication is that we should ban cell phones while driving.
This may be true. I was scared to death last week while my taxi driver was looking at his cell phone to dial numbers. He clearly did not have his eyes on the road. If anything unusual occurred (like the van in the next lane entering our lane right in front of us---watch out please watch out!), his reaction time would have been considerably slowed and we would have been much more likely to have an accident.
On the other hand, I wonder how much of the current problems are caused by a learning deficit. After all, for most of us cell phones are rather new. More importantly, driving while using a cell phone is also new. This kind of multitasking can be learned. There are research studies that show that experience doing multitasking can increase performance on the tasks being done. With enough practice, less working-memory capacity is needed, freeing up capacity to engage in the various tasks.
One hypothesis suggested by this is that cell-phone-related accidents will decrease with time as drivers get more practice using their cell phones while driving. Judging from the number of people I see driving and phoning, not many people are heeding the warnings, so lots of people are gaining more experience. Cell-phone accident rates will also decline as new technologies are utilized, namely voice-dialing and hands-free cell-phones.
On the other hand, a second hypothesis is that anything that prompts drivers to take their eyes off the road will produce similar deficits to cell-phone driving. Here’s a short list:
People who look at their cell phones to dial a number are just asking for trouble. It probably helps to have two hands on the wheel, as well.
I’d be willing to bet that for most people fewer accidents will occur when using a hands-free, voice-dialing cell phone than when talking with someone sitting beside them in the front seat, assuming equal levels of experience doing both. The natural human tendency to want to look someone in the eyes while talking to them will prompt most of us to try and steal a glance at our conversational partners, increasing slightly the danger from unforeseen events.
Like most things in life, learning plays a central role in our cell-phone-while-driving performance. Like most things for us humans, our cognitive machinery sets the boundaries for this performance.
New Information from the Research (An Update on My Thinking)
Although I still wonder about our ability to learn how to utilize cell phones while driving, recent research suggests that right now, we are not too good at it. Check out my updated post on this.
Here's the title of the research article:
When you know that you know and when you think that you know but you don't.
Great title. Much more colorful than most academic research articles. The thing for us to realize is that the "you" in the title of this classic research study is OUR LEARNERS. Sometimes our learners can become overly optimistic about their ability to remember. Oh no. If we believe---even a little bit---the adult-learning-theory mantra that our learners always know best (an incorrect assumption by the way), then we might be setting our learners up for failure.
The research these folks did was designed to look at the spacing effect, the finding that widely-spaced repetitions are more effective than narrowly-spaced repetitions. As the authors say, the spacing effect "has been obtained in a wide variety of memory paradigms, suggesting that it reflects the operation of a fundamental property of the memory system."
Here's What They Did
They had people study lists of words some of which were repeated. They repeated some of the words immediately and some of them after 3 to 5 other words. Notice that the spacings we're talking about here are rather small. For some of the words, they asked the subjects to rate how likely they would be to remember the words if asked about them later.
Here's What They Found
This is the generalizability question. Although the research had learners study words in a free-recall experimental design, instead of using complex knowledge in a more ecologically valid cued-recall design (real-world memory retrieval is almost always cued recall), there is no reason to think that the research results aren't widely applicable. The spacing effect is one of the most replicated phenomenon in learning research. Widely-spaced repetitions minimize forgetting, whereas immediate repetitions cannot. And because the spacing effect is such a fundamental learning mechanism, the college-student-as-subject problem is not relevant.
Zechmeister, E. B., & Shaughnessy, J. J. (1980). When you know that you know and when you think that you know but you don't. Bulletin of the Psychonomic Society, 15(1), 41-44.
At Elliott Masie's Learning 2005 Conference there was a session on Learning R&D efforts within organizations. This seems like a great idea to me. The Masie Center has offered to host a discussion on the conference's wiki, at this web address. It's just getting started, but I seeded the discussion with the following goals one might have for an Internal Learning Practice R&D effort. I repeat that here. Please comment.
Goals for Internal Learning R&D Group:
Learning 2005 was a very good conference, but as is typical in our industry (training & development) this conference still lacked some fundamental elements. Some of the good and bad points of the conference are detailed below. They are provided to encourage changes in all future industry conferences.
Yesterday was the final day of the conference. In the final general session, one of the speakers, Mike somebody (sorry, but there were no overheads to announce the general session speakers), warned the audience not to oversell Extreme Learning, Elliott's term for pushing the technological boundaries and creating very short and quick learning episodes. Actually, now that I think about it, "Extreme Learning" is not really that clear. I guess it has to do with technical innovation. Anyway, the warning seems reasonable to me. Every time there is a new technology, vendors and consultants and everybody oversells that technology and then there is the inevitable backlash. The key is to keep things aligned with the human learning system.
Bob Pike was the most prominent general-session speaker, and he spoke very knowledgeably about how to deal with difficult participants, and a few other things as well. He then literally went crazy and diagnosed 600 people's personality with just one question. He asked us to choose one of six animals---the one that we most resonated with. He then asked us to get out of our chairs and join our other fellow beavers, dolphins, owls, care bears, and two other animals I can't remember. These six groups were supposed to predict our behavior. It was freakin' unbelievable, and yet most of the participants seemed to buy into it completely. Actually, I probably shouldn't be so bold in that statement because I left the room after he divided us into groups. I sort of had to pee and I knew I wasn't going to miss anything. I returned in five or ten minutes just before we were all allowed to return to our chairs.
In a defense of the participants, it was the last day of the conference, everybody was a bit burned out, many had been up late the night before at the MGM theme park that Elliott rented for participants (too many long falls from the Tower of Terror), and Pike used one of the oldest tricks in the persuasion handbook, introducing the topic as an area of rare scientific inquiry, in this case "The Science of Axiology," developed by Robert S. Hartman. The main idea of this brilliant science is that (and this is a direct quote I think) "something is good when it completely fulfills its characteristics."
What this means I'm not sure. In fact, I can't make sense of it. Let's see, I have many characteristics. What does it mean to fulfill a characteristic? I have brown eyes. How do I fulfill that characteristic? Since brown eyes are supposed to be more suited to climates with lots of sunlight (than blue eyes), can I only be good if I live near the equator? I have a tongue. Tongues can do many things. Must I do them all each day to be good? My nose can smell and help me breathe, but it can also run with boogers when I have a cold. Must it run each day if I am to be good? It would have to run with boogers if it was going to enable the tongue to fulfill all of its characteristics. I think I'll stop there with the body parts. Maybe Pike meant inanimate objects only. Let's take cars. Maybe cars can only be good if they fulfill the characteristic of reaching 120 miles per hour, which most are capable of. My car has never reached that speed, so it must be bad. Cars are large objects that go fast. These characteristics make them perfect for killing small animals and children. Can a car only be good if it fulfills its potential for killing (insects on the windshield count to)? Cars also have the characteristic of breaking down. Do cars have to be bad to be good?
I watched Elliott on the Jumbotron at the end of Pike's delusionary sermon. He looked to be in pain. But he saved the show remarkably by thanking Pike for pushing the boundaries of our thinking (or something like that). Nicely done Elliott.
Pike ended in a classy fashion, giving a rather heartfelt plug to Elliott, hailing him as "The Great Connector." It was a touching moment. And it was true. Throughout the conference, we learned again and again how many people Elliott knew. It was quite amazing, and impressive, really.
And then, the show was over, and everyone went home.
in a later post, I'm going to rate the Learning 2005 conference along several dimensions. Overall, it was one of the better conferences I attended, but it is not near to where our conferences need to be to really generate useful learning.