Will at Work Learning

Human Cognition is More Reactive Than You Think

In today's New York Times there is a great article, Who's Minding the Mind?, by BENEDICT CAREY that sums up a large number of research studies on human cognition that show that human beings are more reactive than we might think. We tend to believe that we, as human beings, are very proactive and consciously in control of our thoughts and actions; but these studies show that much of what we do and think is due to hard-wired, often unconscious processes.

For example, the article cites how sitting near a briefcase (as opposed to a backpack) can make people more competitive. Or as Carey writes:

New studies have found that people tidy up more thoroughly when there’s a faint tang of cleaning liquid in the air; they become more competitive if there’s a briefcase in sight, or more cooperative if they glimpse words like “dependable” and “support” — all without being aware of the change, or what prompted it.

This basic fact about human behavior is relevant to the learning and performance field, of course. One of the things I've talked about for years is the notion of "spontaneous remembering." If we create learning right, we're more likely to help our learners—when they're on the job at a later time—by helping them spontaneously trigger memories of what they've learned. We can do this best by requiring our learners to utilize realistic cues in the learning context in making real-world decisions and taking real-world actions. This is why simulations are so effective (if they are well designed).

When learners process learning objectives or prequestions before encountering learning material, the learners are primed to pay attention to relevant learning material. It's not necessarily a conscious process, but it works.

There are many examples available, but here's another point: The learner-centric movement of the 1990's and 2000's has relied too heavily on the notion that the learners always know best, and that they are in conscious control of their learning and we just need to let them make the best decisions.

When we realize that our learners are more deterministically driven than the we want to believe (its about free will a little, isn't it?), we have more work to do if we really want to drive maximum performance. Even when our clients consciously want to do something, we may be able to help them reach their goals by setting up learning and performance situations that unconsciously trigger the behavior they want to achieve.

Evidence-Based Learning

The concept of evidence-based practice is bubbling up all over. I started Work-Learning Research in 1998 to bridge the gap between research and practice. Ruth Clark has been talking about evidence-based recommendations for years. The medical profession has transformed itself to focus on evidence-based medicine. The No-Child-Left-Behind initiative was based on the concept of evidence-based education. Though No Child Left Behind has been poorly implemented, the core concept is sound. Kevin Kruse, CEO of the custom e-learning company Axiom Health, has trademarked the term, "Evidence-Based Training." Management gurus, Jeffrey Pfeffer and Bob Sutton have recently written a best selling book (Pfeffer and Sutton, 2006. Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management). ISPI (the International Society for Performance Improvement) has been promoting evidence-based practices in their CPT certification program since the 1990's.

Evidence-Based Practice is on an upward trajectory, but what does it mean for those of us in the learning-and-performance field? I'm not going to go into depth now, but I want to highlight a few critical points.

1. Evidence-Based Learning (EBL) DOES NOT simply mean that we follow research-based prescriptions.
2. Evidence-Based Learning (EBL) requires us also to measure our own performance. In other words, we ought to be routinely gathering good EVIDENCE about how well our learning interventions are working. Only by having feedback loops can we learn from our performance.
3. Evidence-Based Learning (EBL) requires us also to build continuous cycles of improvement into our practices. After gathering and analyzing the evidence, we need to act on it. Then we need to evaluate and analyze and act again.

I spend an inordinate amount of time every year culling practical learning wisdom from the world's preeminent refereed journals on learning, memory, and instruction. It appears that I am one of the most passionate advocates for learning research living today. Yet still, I believe that following research-based recommendations can provide us with less than half, probably less than one-third, of the power of a full practice of evidence-based learning.

In fact, I've come to believe that, as a field, we have only reached a small measure of our potential because we don't utilize evidence-based practices. We don't have adequate feedback loops. We act too often on superstition, on the ideas inherent in commercial messaging, and on our learner's lowest-common denominator comfort zones.

The bottom line recommendation is this: Only with true evidence-based practices, not warmed over attempts to follow a few research-based prescriptions, can you build a maximally effective learning program.

Sorry I can't provide more specifics in this short blog format. If you want to know more, feel free to get in touch with me.

The Magic Question for Learning and Instructional Design

The most important question that instructional designers can ask is:

“What do learners need to be able to do, and in what situations do they need to do those things?”

While we might discount such a simple question as insignificant, the question brilliantly forces us to focus on our ultimate goals and helps us to align our learning interventions with the human learning system.

Too many of us design with a focus on topics, content, knowledge. This tendency pushes us, almost unconsciously, to create learning that is too boring, filled with too much information, and bereft of practice in realistic situations.

The Magic Question requires us to be relevant. For workplace learning, it focuses our thinking toward learners' future job situations. For education learning, it focuses our thinking toward real-world relevance of our academic topics.

The Magic Question in Practice

In practice, the Magic Question forces us to begin our instructional-design efforts by not only creating a list of instructional objectives, but also by creating a list of performance situations. For example, if we're creating leadership training, we not only need to compile objectives like, "For most decisions, it can be helpful to bring your direct reports into decision-making, so as to increase the likelihood that they will bring energy and passion in implementing decisions." We also need to compile a list of situations were this objective is relevant, for example in weekly staff meetings, project meetings, in one-on-one face-to-face conversations, in phone conversations, etc. Also, for general decision making, but not in situations where time is urgent, where safety is an issue, where legal ramifications are evident, etc.

By framing our instructional-design projects in this way, we get to think about our learning designs in ways that are much more action-oriented, relevant, and practical. The framing makes it more likely that we will align our learning and performance contexts, making it more likely that our learners, in their future situations, will spontaneously remember what we've taught them. The framing makes it more likely that we will focus on practice instead of overloading our learners with information. The framing also makes it more likely that we will utilize relevant scenarios that more fully engage our learners. Finally, using the Magic Question forces our SME's (subject-matter experts) to reformulate their expertise into potent practical packages of relevant material. It's not always easy to bend SME's to this discipline, but after the pain, they'll thank you profusely as together you push their content to a much higher level.

Obviously, there is more to be said about how the Magic Question can be integrated into learning-design efforts. On the other hand, as my clients have reported, the Magic Question has within it a simple power to (1) change the way we think about instructional design, and (2) transform the learning interventions we build.

Training and Carbon Offsets

The Carbon Offset idea works like this. We all pollute, but when we do so we can help limit the damaging effects by either (1) offsetting our damage by doing good in other ways (for example if we have to drive a large car we can replace all our light bulbs with energy-saving flourescents), or (2) we can donate money to projects that help support renewable energy, energy efficiency, and reforestation. For example, check out the not-for-profit organizations CarbonFund.org and The Clean Air Conservancy.

Here's some ideas for those of us in the training and development field:

1. Encourage the use of e-learning, which limits the carbon footprint of travel. And, make sure you build e-learning that is effective and engaging, so more folks will want to use e-learning.
2. When calculating the "cost" of training, calculate carbon footprint costs as well. See for example, The Carbon Fund's calculators or The Clean Air Conservancy's calculators. Make these costs evident.
3. Encourage your company to buy carbon offsets when utilizing training. It's not just a good thing to do, but it may help your company attract business and recruit highly-educated employees.
4. In your e-learning courses, provide an option for learners to calculate how many tons of carbon dioxide they would have utilized had they had to travel from their location to headquarters.

What other ideas can you think of?

Measuring Learning Results: The New Research-to-Practice White Paper

I've just completed a new research-to-practice white paper. As far as I can tell, it is the first work on learning measurement (assessment and evaluation) that actually takes human learning into consideration. I'd like to thank Questionmark for agreeing to support this work.

Words from the paper's introduction:

In writing this report on using fundamental learning research to inform assessment  design, I am combining two of my passions—learning and the measurement of learning. As an experienced learner and learning designer, I have come to the belief that those of us responsible for designing, developing, and delivering learning interventions are often left in the dark about our own successes and failures. The measurement techniques we use simply do not provide us with valid feedback about our own performances.

The traditional model of assessment utilizes end-of-learning assessments provided to learners in the context in which they learned. This model is seriously flawed, especially in failing to give us an idea of how well our learning interventions are doing in preparing our learners to retrieve information in future situations—the ultimate goal of training and education. By failing to measure our performance in this regard, we are missing opportunities to provide ourselves with valid feedback. We are also likely failing our institutions and our learners because we are not able to create a practice of continuous improvement to maximize our learning outcomes.

This report is designed to help you improve your assessments in this regard. I certainly won’t claim to have all the answers, nor do I think it is easy to create the perfect assessment, but I do believe very strongly that all of us can improve our assessments substantially, and by so doing improve the practice of education and training.

How to buy the paper:

• Click to go to my catalog
• Click to buy the paper directly

Wiki Improvements

I have critiqued the wiki revolution (see posts in 2005 and in 2006) because I worried about the garbage-in garbage-out problem. That is, how can consumers of wikized information know whether the information is valid? Previously, the knowledge culture has relied on editors, peer-reviewers, and publishers to vet information prior to publication.

Wikipedia has come to recognize this problem and is taking steps to lessen it. See their WikiProject Fact and Reference Check webpage. As you browse Wikipedia today, you'll often see pages marked with the warning, "This [entry] does not adequately cite its references or sources. Please help improve this article by adding citations to reliable sources."

This is a great first step. Whether it is enough, we will see. It probably depends on the topic area discussed.

And to be balanced, we must recognize that any influence can tilt content toward one truth or another, whether that influence is an editor or an admonition to cite reliable sources. Therefore, my next recommendation is to formalize alternative interpretations within wiki technology and practice. The wiki interface sometimes formalizes and encourages a "one-truth" paradigm, so where one truth is inadequate, the technology needs to encourage a different sort of mental model, one that allows information consumers to feel the heat, sweat, and uncertainty of the truth-building process.

Learning Research Quiz Results 2002-2007

Since 2002, I have offered the Learning Research Quiz on the Work-Learning Research website. Over 1200 people have taken that quiz. 991 people answered all 15 questions. There are 15 questions on the quiz. A blind monkey randomly guessing the answers would score about 24% (there are three to seven answer choices per question).

The questions are scenario-based questions presenting relatively realistic learning-design situations. The answer choices are plausible---there are no throwaway answer choices. The questions are thus not easy to answer. They are specifically designed to measure people's understanding of fundamental learning principles like repetition, spacing, feedback, retrieval practice, instructional objectives, etc.

People who answered the questions are self-selected from visitors to the Work-Learning Research website so we can't really be sure who this sample represents. In general, it is my experience that most people who come to the website are experienced learning professionals who are passionate about learning, training, teaching, and/or instructional design.

Here are the results for the 991 people who answered all 15 questions:

If 24% is a completely random score, what should we expect from those who take the quiz? Would you expect them to do twice as well (around 50% correct?) or do better, say 60%, 70%, 80%, 90%, or 100%?

On average our 991 participants scored 32% correct, barely better than chance responding!!

Scoring 1% to 25% correct were 279 respondents.
Scoring 25% to 50% correct were 610 respondents.
Scoring 50% to 75% correct were 101 respondents.
Scoring 75% to 100% correct was 1 respondent, who scored 80% correct.

Association affilitation had little impact on average correct:

ISPI members scored 36% correct.
ASTD members scored 33% correct.
Other trade association members 28% correct.
Research association members 30% correct.
Non-affilitated respondents scored 31% correct.

Age of respondents had little impact on average correct:

Under 21 years of age scored 26%
21-25 scored 28%
26-35 scored 32%
36-45 scored 33%
46-55 scored 30%
56-65 scored 35%
66-75 scored 33%

Education degree had little impact on average correct:

"Ph.D. in psychology, learning, etc." scored 35% correct.
"Ph.D. in other discipline" scored 34% correct.
"Masters in psychology, learning, etc." scored 33% correct.
"Masters in other discipline" scored 32% correct.
"Bachelors" scored 31% correct.
"Other" scored 29% correct.

Job Title had little impact on average correct:

"Trainer" scored 32% correct.
"Instructional Designer" scored 33% correct.
"Performance Consultant" scored 33% correct.
"Human Performance Technologist" 38% correct.
"E-Learning Specialist" scored 29% correct.
"Learning Technology Developer" scored 33% correct.
"Teacher" scored 31% correct.
"Professor (with little research activity)" scored 33% correct.
"Learning Researcher (professors too)" scored 31% correct.
"Manager of Instructional Development" scored 34% correct.
"Manager of Training" scored 34% correct.
"Student" scored 28% correct.
"Other" scored 29% correct.

Date of Taking the Quiz had little impact on average correct:

2002 scored 30%
2003 scored 33%
2004 scored 31%
2005 scored 33%
2006 scored 30%
2007 scored 33%

Conclusion

Because the sample of respondents is difficult to define, any conclusions must remain speculative. Still, the results suggest a massive competence gap. The 32% average score---and the stubborn lack of improvement regardless of experience, education, and age---suggests that most people in the learning-and-performance field are unprepared for roles as designers of learning, at least as far as their ability to apply knowledge of learning research.

Links

• The Quiz
• The Quiz Questions with Feedback

Fair-Assessment Quick-Audit

I'm a strong advocate for assessment. If we don't assess our learning interventions, (a) we as instructional designers don't learn ourselves, (b) we don't have valid data to give ourselves feedback, and (c) we can't possibly improve our learning designs.

If we're going to validly assess our learning interventions, we have to understand human learning and beware of biasing our results. Some of the things we have to watch out for:

1. Testing our learners only when information is top-of-mind.
2. Testing our learners in the learning context.
3. Testing our learners unfairly with biasing pretests.
4. Testing our learners with stupid, irrelevant questions.
5. Using Level 1 smile-sheet data exclusively.
6. Measuring with post-hoc metrics.

To help folks in the field avoid some of these pitfalls, I've developed the Fair-Assessment Quick-Audit courtesy of LearningAudit.com and Work-Learning Research.

Download The_Fair-Assessment_Quick-Audit.pdf

Some of these criteria are critical, especially because we in the field tend to do the exact opposite of what is fair and valid.

See this (click here) recent post that describes the mistakes we're making in assessment.

Learning Styles Instructional-Design Challenge

I will give $1000 (US dollars) to the first person or group who can prove that taking learning styles into account in designing instruction can produce meaningful learning benefits.

I've been suspicious about the learning-styles bandwagon for many years. The learning-style argument has gone something like this: If instructional designers know the learning style of their learners, they can develop material specifically to help those learners, and such extra efforts are worth the trouble.

I have my doubts, but am open to being proven wrong.

Here's the criteria for my Learning-Styles Instructional-Design Challenge:

1. The learning program must diagnose learners' learning styles. It must then provide different learning materials/experiences to those who have different styles.
2. The learning program must be compared against a similar program that does not differentiate the material based on learning styles.
3. The programs must be of similar quality and provide similar information. The only thing that should vary is the learning-styles manipulation.
4. The comparison between the two versions (the learning-style version and the non-learning-style version) must be fair, valid, and reliable. At least 70 learners must be randomly assigned to the two groups (with at least 35 minimum in each group completing the experience). The two programs must have approximately the same running time. For example, the time required by the learning-style program to diagnose learning styles can be used by the non-learning-styles program to deliver learning. The median learning time for the programs must be no shorter than 25 minutes.
5. Learners must be adults involved in a formal workplace training program delivered through a computer program (e-learning or CBT) without a live instructor. This requirement is to ensure the reproducability of the effects, as instructor-led training cannot be precisely reproduced.
6. The learning-style program must be created in an instructional-development shop that is dedicated to creating learning programs for real-world use. Programs developed only for research purposes are excluded. My claim is that real-world instructional design is unlikely to be able to utilize learning styles to create learning gains.
7. The results must be assessed in a manner that is relatively authentic--at a minimum level learners should be asked to make scenario-based decisions or perform activities that simulate the real-world performance the program teaches them to accomplish. Assessments that only ask for information at the knowledge level (e.g., definitions, terminology, labels) are NOT acceptable. The final assessment must be delayed at least one week after the end of the training. The same final assessment must be used for both groups. It must fairly assess the whole learning experience.
8. The magnitude of the difference in results between the learning-style program and the non-learning-style program must be at least 10%. (In other words, the average of the learning-styles scores subtracted by the average of the non-learning-styles scores must be more than 10% of the non-learning-styles scores). So for example, if the non-learning-styles average is 50, then the learning-styles score must be equal to 55 or more. This magnitude is to ensure that the learning-styles program produces meaningful benefits. 10% is not too much to ask.
9. The results must be statistically significant at the p<.05 level. Appropriate statistical procedures must be used to gauge the reliability of the results. Cohen's d effect size should be equal to .4 or more (a small to medium effect size according to Cohen, 1992).
10. The learning-style program cannot cost more than twice as much as the non-learning-style program to develop, nor can it take more than twice as long to develop. I want to be generous here.
11. The results can be documented by unbiased parties.

To reiterate, the challenge is this:

Can an e-learning program that utilizes learning-style information outperform an e-learning program that doesn't utilize such information by 10% or more on a realistic test of learning, even it is allowed to cost up to twice as much to build?

$1,000 says it just doesn't happen in the real-world of instructional design. $1,000 says we ought to stop wasting millions trying to cater to this phantom curse.

New Taxonomy for Learning Objectives

Let me propose a new taxonomy for learning objectives.

This taxonomy is needed to clear up the massive confusion we all have about the uses and benefits of learning objectives. I have tried to clarify this in the past in some of my conference presentations—but I have not been successful. When I get evaluation-sheet comments like, "Get real you idiot!" from more than a few people, I know I've missed the mark. SMILE

Because I don't give up easily—and because learning objectives are so vitally important—I'm going to give this another try. Your feedback is welcome.

The premise I'm working from is simple. Instructional professionals use learning objectives for different purposes—even for different audiences. Learning objectives are used to guide the attention of the learner toward critical learning messages. Learning objectives are used to tell the learner what's in the course. They are used by instructional designers to guide the design of the learning. They are used by evaluation designers to develop metrics and assessments.

Each use requires its own form of learning objective. Doesn't it seem silly to use the exact same wording regardless of the use or intended audience? Do we provide doctors and patients with the exact same information about a particular prescription drug? Do designers of computer software require the same set of goal statements as users of that software? Do creators of films need to have the same set of objectives as movie goers?

Until recently I have argued that we ought to delineate between objectives for learners and objectives for designers. This was a good idea in principle, but it still left people confused because it didn't cover all the uses of objectives. For example, learners can be presented with objectives to help guide their attention or to simply give them a sense of the on-the-job performance they'll be expected to perform. Instructional designers can utilize objectives to guide the design process or to develop evaluations.

The New Taxonomy

1. Focusing Objective
   A statement presented to learners before they encounter learning material—provided to help guide learner attention the most important aspects of that learning material.
2. Performance Objective
   A statement presented to learners before they encounter learning material—provided to help learners get a quick understanding of the competencies they will be expected to learn.
3. Instructional-Design Objective
   A statement developed by and for instructional designers to guide the design and development of learning and instruction.
4. Instructional-Evaluation Objective
   A statement developed by and for program evaluators (or instructional designers) to guide the evaluation of instruction.

I made a conscious decision not to include a "table-of-contents objective" despite the widespread use of this method for presenting learners with objectives. I can't decide whether this should be included. There's no direct research on this (that I've encountered), but there may be some benefit for learners in having an outline of the coming learning material. Your comments welcome. I'm leaning toward including this notion into the taxonomy because it is a stategy that I've seen in use. Maybe I'll call them "Content-Outlining Objectives" or "Outlining Objectives."

One of the clear benefits of this taxonomy is that it separates Focusing Objectives from the other objectives. These objectives—those presented to learners to help focus their attention—have been researched with the greatest vigor. And the results of that research are clear:

1. Focusing objectives guide learner attention to the information in subsequent learning material that has been targeted by objectives, but they also take attention away from the information not targeted by objectives.
2. Similarly, focusing objectives improve learning for the targeted information and hurt learning for the information not targeted.
3. Prequestions are as powerful in creating this focusing effect as learning objectives, and they may be more powerful.
4. The wording of the focusing objective or prequestion must specifically mirror the wording in the learning material. General or abstract wording doesn't cut it.
5. Adding extra words, particularly words that specify the criteria of performance (ala Mager) will actually distract learners and hurt learning.