Will at Work Learning

Human Cognition is More Reactive Than You Think

In today's New York Times there is a great article, Who's Minding the Mind?, by BENEDICT CAREY that sums up a large number of research studies on human cognition that show that human beings are more reactive than we might think. We tend to believe that we, as human beings, are very proactive and consciously in control of our thoughts and actions; but these studies show that much of what we do and think is due to hard-wired, often unconscious processes.

For example, the article cites how sitting near a briefcase (as opposed to a backpack) can make people more competitive. Or as Carey writes:

New studies have found that people tidy up more thoroughly when there’s a faint tang of cleaning liquid in the air; they become more competitive if there’s a briefcase in sight, or more cooperative if they glimpse words like “dependable” and “support” — all without being aware of the change, or what prompted it.

This basic fact about human behavior is relevant to the learning and performance field, of course. One of the things I've talked about for years is the notion of "spontaneous remembering." If we create learning right, we're more likely to help our learners—when they're on the job at a later time—by helping them spontaneously trigger memories of what they've learned. We can do this best by requiring our learners to utilize realistic cues in the learning context in making real-world decisions and taking real-world actions. This is why simulations are so effective (if they are well designed).

When learners process learning objectives or prequestions before encountering learning material, the learners are primed to pay attention to relevant learning material. It's not necessarily a conscious process, but it works.

There are many examples available, but here's another point: The learner-centric movement of the 1990's and 2000's has relied too heavily on the notion that the learners always know best, and that they are in conscious control of their learning and we just need to let them make the best decisions.

When we realize that our learners are more deterministically driven than the we want to believe (its about free will a little, isn't it?), we have more work to do if we really want to drive maximum performance. Even when our clients consciously want to do something, we may be able to help them reach their goals by setting up learning and performance situations that unconsciously trigger the behavior they want to achieve.

Evidence-Based Learning

The concept of evidence-based practice is bubbling up all over. I started Work-Learning Research in 1998 to bridge the gap between research and practice. Ruth Clark has been talking about evidence-based recommendations for years. The medical profession has transformed itself to focus on evidence-based medicine. The No-Child-Left-Behind initiative was based on the concept of evidence-based education. Though No Child Left Behind has been poorly implemented, the core concept is sound. Kevin Kruse, CEO of the custom e-learning company Axiom Health, has trademarked the term, "Evidence-Based Training." Management gurus, Jeffrey Pfeffer and Bob Sutton have recently written a best selling book (Pfeffer and Sutton, 2006. Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management). ISPI (the International Society for Performance Improvement) has been promoting evidence-based practices in their CPT certification program since the 1990's.

Evidence-Based Practice is on an upward trajectory, but what does it mean for those of us in the learning-and-performance field? I'm not going to go into depth now, but I want to highlight a few critical points.

1. Evidence-Based Learning (EBL) DOES NOT simply mean that we follow research-based prescriptions.
2. Evidence-Based Learning (EBL) requires us also to measure our own performance. In other words, we ought to be routinely gathering good EVIDENCE about how well our learning interventions are working. Only by having feedback loops can we learn from our performance.
3. Evidence-Based Learning (EBL) requires us also to build continuous cycles of improvement into our practices. After gathering and analyzing the evidence, we need to act on it. Then we need to evaluate and analyze and act again.

I spend an inordinate amount of time every year culling practical learning wisdom from the world's preeminent refereed journals on learning, memory, and instruction. It appears that I am one of the most passionate advocates for learning research living today. Yet still, I believe that following research-based recommendations can provide us with less than half, probably less than one-third, of the power of a full practice of evidence-based learning.

In fact, I've come to believe that, as a field, we have only reached a small measure of our potential because we don't utilize evidence-based practices. We don't have adequate feedback loops. We act too often on superstition, on the ideas inherent in commercial messaging, and on our learner's lowest-common denominator comfort zones.

The bottom line recommendation is this: Only with true evidence-based practices, not warmed over attempts to follow a few research-based prescriptions, can you build a maximally effective learning program.

Sorry I can't provide more specifics in this short blog format. If you want to know more, feel free to get in touch with me.

Training and Carbon Offsets

The Carbon Offset idea works like this. We all pollute, but when we do so we can help limit the damaging effects by either (1) offsetting our damage by doing good in other ways (for example if we have to drive a large car we can replace all our light bulbs with energy-saving flourescents), or (2) we can donate money to projects that help support renewable energy, energy efficiency, and reforestation. For example, check out the not-for-profit organizations CarbonFund.org and The Clean Air Conservancy.

Here's some ideas for those of us in the training and development field:

1. Encourage the use of e-learning, which limits the carbon footprint of travel. And, make sure you build e-learning that is effective and engaging, so more folks will want to use e-learning.
2. When calculating the "cost" of training, calculate carbon footprint costs as well. See for example, The Carbon Fund's calculators or The Clean Air Conservancy's calculators. Make these costs evident.
3. Encourage your company to buy carbon offsets when utilizing training. It's not just a good thing to do, but it may help your company attract business and recruit highly-educated employees.
4. In your e-learning courses, provide an option for learners to calculate how many tons of carbon dioxide they would have utilized had they had to travel from their location to headquarters.

What other ideas can you think of?

American Health Falling "Short"

In today's New York Times, columnist and economist Paul Krugman details new data that shows that Americans are literally losing stature. Here's a quote from the article.

The data show that Americans, who in the words of a recent paper by the economic historian John Komlos and Benjamin Lauderdale in Social Science Quarterly, were “tallest in the world between colonial times and the middle of the 20th century,” have now “become shorter (and fatter) than Western and Northern Europeans. In fact, the U.S. population is currently at the bottom end of the height distribution in advanced industrial countries.”

This is not a trivial matter. As the paper says, “height is indicative of how well the human organism thrives in its socioeconomic environment.”

The link to the article is here, but you have to be a subscriber to read it.

How might this relate to those of us in the United State's learning-and-performance field? Well, mostly this is an interesting tidbit that we have little control over. On the other hand, it might give us pause. After all, if we create learning programs of equal effectiveness to our overseas competitors, but their learners are healthier than our learners, their learners will learn more and perform better in their work. Their companies will have a competitive advantage. We will all die penniless and alone. (Exaggeration).

Krugman reflects on the argument that American's unhealthy ways might be related the fact that we work too much, and thus don't have time to exercise and eat right. Is that a hook into our responsibility as learning professionals? Is there anything that we can do to lower the average time our workers are swimming in the ocean of work responsibilities?

Don't just think content here. Preaching and information are not likely to help that much.

Well, I'm brainstorming here (as I have no idea), we can encourage e-learning to be done on work time, maybe by utilizing more synchronous, and more social interactions. We can realize that learners will forget a large chunk of what we teach, and either cut the forgettable crap out of our courses or demand from our organization and vendors that it be put into performance support. Maybe we can design m-learning interactions that are especially appropriate to be used during exercise. I don't know how to do this, and it may not be doable, but maybe you can be the one to figure it out. Maybe we can provide truly healthy and delicious food for our training participants.

What else? I don't know. Do you?

Or do you think it's outside our influence?

Wiki Improvements

I have critiqued the wiki revolution (see posts in 2005 and in 2006) because I worried about the garbage-in garbage-out problem. That is, how can consumers of wikized information know whether the information is valid? Previously, the knowledge culture has relied on editors, peer-reviewers, and publishers to vet information prior to publication.

Wikipedia has come to recognize this problem and is taking steps to lessen it. See their WikiProject Fact and Reference Check webpage. As you browse Wikipedia today, you'll often see pages marked with the warning, "This [entry] does not adequately cite its references or sources. Please help improve this article by adding citations to reliable sources."

This is a great first step. Whether it is enough, we will see. It probably depends on the topic area discussed.

And to be balanced, we must recognize that any influence can tilt content toward one truth or another, whether that influence is an editor or an admonition to cite reliable sources. Therefore, my next recommendation is to formalize alternative interpretations within wiki technology and practice. The wiki interface sometimes formalizes and encourages a "one-truth" paradigm, so where one truth is inadequate, the technology needs to encourage a different sort of mental model, one that allows information consumers to feel the heat, sweat, and uncertainty of the truth-building process.

Learning Research Quiz Results 2002-2007

Since 2002, I have offered the Learning Research Quiz on the Work-Learning Research website. Over 1200 people have taken that quiz. 991 people answered all 15 questions. There are 15 questions on the quiz. A blind monkey randomly guessing the answers would score about 24% (there are three to seven answer choices per question).

The questions are scenario-based questions presenting relatively realistic learning-design situations. The answer choices are plausible---there are no throwaway answer choices. The questions are thus not easy to answer. They are specifically designed to measure people's understanding of fundamental learning principles like repetition, spacing, feedback, retrieval practice, instructional objectives, etc.

People who answered the questions are self-selected from visitors to the Work-Learning Research website so we can't really be sure who this sample represents. In general, it is my experience that most people who come to the website are experienced learning professionals who are passionate about learning, training, teaching, and/or instructional design.

Here are the results for the 991 people who answered all 15 questions:

If 24% is a completely random score, what should we expect from those who take the quiz? Would you expect them to do twice as well (around 50% correct?) or do better, say 60%, 70%, 80%, 90%, or 100%?

On average our 991 participants scored 32% correct, barely better than chance responding!!

Scoring 1% to 25% correct were 279 respondents.
Scoring 25% to 50% correct were 610 respondents.
Scoring 50% to 75% correct were 101 respondents.
Scoring 75% to 100% correct was 1 respondent, who scored 80% correct.

Association affilitation had little impact on average correct:

ISPI members scored 36% correct.
ASTD members scored 33% correct.
Other trade association members 28% correct.
Research association members 30% correct.
Non-affilitated respondents scored 31% correct.

Age of respondents had little impact on average correct:

Under 21 years of age scored 26%
21-25 scored 28%
26-35 scored 32%
36-45 scored 33%
46-55 scored 30%
56-65 scored 35%
66-75 scored 33%

Education degree had little impact on average correct:

"Ph.D. in psychology, learning, etc." scored 35% correct.
"Ph.D. in other discipline" scored 34% correct.
"Masters in psychology, learning, etc." scored 33% correct.
"Masters in other discipline" scored 32% correct.
"Bachelors" scored 31% correct.
"Other" scored 29% correct.

Job Title had little impact on average correct:

"Trainer" scored 32% correct.
"Instructional Designer" scored 33% correct.
"Performance Consultant" scored 33% correct.
"Human Performance Technologist" 38% correct.
"E-Learning Specialist" scored 29% correct.
"Learning Technology Developer" scored 33% correct.
"Teacher" scored 31% correct.
"Professor (with little research activity)" scored 33% correct.
"Learning Researcher (professors too)" scored 31% correct.
"Manager of Instructional Development" scored 34% correct.
"Manager of Training" scored 34% correct.
"Student" scored 28% correct.
"Other" scored 29% correct.

Date of Taking the Quiz had little impact on average correct:

2002 scored 30%
2003 scored 33%
2004 scored 31%
2005 scored 33%
2006 scored 30%
2007 scored 33%

Conclusion

Because the sample of respondents is difficult to define, any conclusions must remain speculative. Still, the results suggest a massive competence gap. The 32% average score---and the stubborn lack of improvement regardless of experience, education, and age---suggests that most people in the learning-and-performance field are unprepared for roles as designers of learning, at least as far as their ability to apply knowledge of learning research.

Links

• The Quiz
• The Quiz Questions with Feedback

m-Learning Device is Coming: Apple iPhone

Apple's new iPhone is really quite amazing. I'm not usually a gadget guy, but you ought to check this out. It really has some great potential as an m-learning device. It will have some of the same limitations of any m-learning device, but it does give us a good glimpse of the future. Check out Steve Job's introduction of the iPhone by clicking here. Job's keynote is not only interesting for the content. It's great theater as well. I clicked on it to find a quote, but I ended up watching almost the whole damn keynote. I found it inspiring. And I don't think it was inspiring because I was once the proud owner of the first line of the Apple Macintosh's---the one's with the 128K disk drives. SMILE.

Of course, there is always a downside isn't there. Check out this article from the NY Times on how Apple is restricting iPod and iTunes users in using music files. Is Apple in danger of losing it's Teflon-coated image as the good-tech angel?

You can hear an analysis of the iPhone and Apple's stock option troubles on the radio show OnPoint.

Lev Grossman wrote the following in Time Magazine.

"Everybody hates their phone," Jobs says, "and that's not a good thing. And there's an opportunity there." To Jobs's perfectionist eyes, phones are broken. Jobs likes things that are broken. It means he can make something that isn't and sell it to you for a premium price.

This brilliant business analysis ought to relate to our industry as well. What's out there in the training-and-development, e-learning, learning-and-performance field that is broken? What can we fix? What can we get people to actually pay for?

Stay tuned for my answers to these questions over the next couple years. And leave your comments about what is broken and what can be fixed.

Neon Elephant Award 2006 to Cal Wick

I'd like to announce the inaugural edition of the Neon Elephant Award, awarded for 2006 to Cal Wick of the Fort Hill Company for leading the development of the first commercially-viable training-follow-through e-learning software and his work as co-author of the book, The Six Disciplines of Breakthrough Learning: How to Turn Training and Development into Business Results.

The Neon Elephant Award

The Neon Elephant Award is awarded to a person, team, or organization exemplifying enlightenment, integrity, and innovation in the field of workplace learning and performance. Announced on the day of the winter solstice—the day of the year when the world turns away from darkness toward the light and hope of warmer days to come—the Neon Elephant Award honors those who have truly changed the way we think about the practice of learning and performance improvement. Award winners are selected for demonstrated success in pushing the field forward in significant paradigm-altering ways while maintaining the highest standards of ethics and professionalism.

Click here to see the full announcement...

Ethics in Internet Selling and Advertising

Newspapers and magazines have recognized for many years that there must be a clear distinction between content and advertising in the perceptions of readers. These entities not only have clear standards about these distinctions, but they also enforce a clear delineation between their sales departments and their news departments so that their sales efforts cannot influence (bias) their news reporting.

It is similar in the TV news business. Even as recently as last year, news organizations have been chastened with public outrage when Bush Administration video propoganda---created to look like real news broadcasts---was used by several news organizations as if it was an actual independent news clip. See the following article for details, for example (access costs money), THE MESSAGE MACHINE: How the Government Makes News; Under Bush, a New Age of Prepackaged News. By DAVID BARSTOW AND ROBIN STEIN; ANNE E. KORNBLUT CONTRIBUTED REPORTING FOR THIS ARTICLE. NY Times: March 13, 2005

News organizations don't follow these ethical practices only because they believe in their sanctity. They follow these practices because the general public becomes outraged when news is seen as biased by financial interests or the influence of powerful elites.

Unfortunately, on the web---where the general public is being sorted into smaller and smaller subgroups---it is easier for organizations to deceive these micro-publics into believing that commercial messages come from independent unbiased sources.

With this in mind, I offer the following minimum standards of ethics for internet sellers and advertisers:

1. Almost everyone who views your advertising should understand immediately that it is an advertisement.
2. It is more ethical to offer independent content than sponsored content.
3. It is more ethical to offer both independent and sponsored content than it is to offer sponsored content alone.
4. It is more ethical to offer sponsored content and independent content if the financial benefits of the sponsored content are required to enable the production of the independent content.
5. Readers/viewers must be able to immediately recognize which information is sponsored and which information is independent.
6. Almost everyone who views sponsored content (web pages, white papers, articles, awards, best-of lists, etc.) should understand immediately that the information is being presented to sell or persuade the readers or viewers.
7. The development of independent content should not be influenced by sponsoring entities.
8. Authors must clearly acknowledge any relationships that may bias their work.

What else am I missing?

What do I have wrong?

What evidence of this do you see in the Workplace Learning-and-Performance Field?

What else, specifically, do we have to worry about? Any of the following (I'm just brainstorming)?

• product placements in e-learning?
• trade organization behaviors?
• helping our learners distinguish truth from sales pitches?
• conferences that offer sponsorships?
• webinars that are sales pitches masquerading as information?
• gurus that have vendor money in their pockets?
• online communities based on sponsorship money?
• white papers?
• award programs?
• vendor research?

Assessment Mistakes by E-Learning Developers

In a webinar this month (December 2006), I asked a group of about 100 e-learning professionals, what was the highest level of assessment they did (based on Kirkpatrick's Four Levels) on their most recent learning-intervention development project.

• 11% said they did NO evaluation
• 26% said they did Level 1 smile sheets
• 48% said they measured Level 2 learning
• 15% said they measured Level 3 on-the-job performance
• 0% said they measured Level 4 business results (or ROI).

Unfortunately, smile sheets are very poor predictors of meaningful learning outcomes, being correlated at less than an r of .2 with learning and performance. See Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-357.

Stunning: Even after all the hot air expelled, ink spilled, and electrons excited in the last 10 years regarding how we ought to be measuring business results, nobody is doing it !!!!!!!!!

------------------------------

When I asked them how they did their most recent assessment, in terms of WHEN they did the assessment---whether they did the assessment immediately at the end of learning or after a delay,

• 77% said they did the assessment, "At the end of training."
• 7% said they did the assessment, "After a delay."
• 14% said they did the assessment, "At end—and after a delay."
• 2% said, "Never done / Can’t remember."

Unfortunately, the 77% are biasing the results in a positive direction. They are measuring learning when it is top-of-mind, easily accessible from long-term memory. They are measuring the learning intervention's ability to create short-term learning effects. They are not measuring its ability to support long-term remembering, or its ability to specifically minimize forgetting.

In the graphic depiction below, the top of the left axis (the y axis) represents more remembering, the bottom is less remembering. Consider what happens if we assess learning at the end (or the top) of the first (leftmost) "Learning" curve. If the learners utilize what they've learned on the job, such an assessment has a negative bias. However, what typically happens over time is more like the forgetting curve (depicted at the lower right). Unless learners regularly use what they've learned, any assessment at the end of the first learning curve is likely to be a poor predictor of future remembering---and show a definite positive bias.

------------------------------

When I asked them how they did their most recent assessment, in terms of WHERE the learners were when they completed the assessment,

• 70% said they did the assessment, "In the training room/context."
• 26% said they did the assessment, "In a different room/context."
• 5% said, "Never done / Can’t remember."

Unfortunately, the 70% are biasing the results of their assessments in a positive direction. When learners are in the same context during retrieval as during learning, they tend to recall more because the background context stimulates improved retrieval. So, providing our training assessments in the training room (or using the same background stimuli in an e-learning course), is not a fair way for us to get feedback on our performance as instructional designers.

For one example of this research paradigm, see the following example. Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

The Bottom Line

First, I'm not blaming these particular folks. This is a common reality. I have regularly failed---and continue to fail often---in validly assessing my own instructional-development efforts. I'm much better when people pay me to evaluate their learning interventions (see LearningAudit.com).

Almost all of us---as far as I can tell---are just not getting valid feedback on our instructional-development efforts. Note that though 48% said they did Level 2 learning evaluation on their most recent project, probably most of those folks delivered the assessment in a way that biased those results. This leaves very few of us who are getting valid feedback on our designs.

We're in a dark fog about how we're doing, and so we have massively impoverished information to use to make improvements.

Basically, we live in a shameful, self-imposed fog.

Bring on the fog lights!!