Will at Work Learning

Spacing Learning Events Over Time. Send me Your Examples.

In 2006, I reviewed the research on the spacing effect and published a research-to-practice report, Spacing Learning Over Time: What the Research Says...

Since then I have been buoyed by the enthusiastic response to that report and by the changes that it  engendered. More training and e-learning has been built using spacing and more and more learning software has been built that incorporates the spacing effect as an inherent part of its design. If I died today, I would at least know that I'd made a small difference in our field.

Examples Wanted

I am working on an updated version of the report to include the latest research and new examples.

If you know of any examples of the use of spacing effect, please let me know. Send me demo links or disks so that I can see for myself how the spacing effect has been used. Or, just write me an email.

Testimonials Wanted

Also, if you read the original version and want to write a short testimonial about how it changed the way you build learning, that would be awesome. Just write me an email.

One Product Example: A Cameo Appearance

Just to get your juices flowing, check out this YouTube Video produced by a company who built a product with the spacing effect in mind, Yukon Learning. Yukon has built a very nice tool to support learning using the spacing effect. The product name is Cameo and the link below will take you to the Cameo website.

You can check out Cameo at this website.

Aging. Can We Enhance People's Cognitive Outcomes?

If you work in the workplace learning-and-performance field, one of your jobs is to ensure that employees are maximizing their cognitive performance, their decision making, and their overall work output. If people's cognitive abilities decreased with age, that would be a problem. More importantly, if we can improve our employee's cognitive abilities, we have a responsibility to do just that. The benefits will accrue to our organizations and to our employees too (and probably then to their families and society at large).

This begs the following questions then:

• "Is there research in refereed scientific journals that provides evidence for cognitive decline as people age?"
• "Is there research in refereed scientific journals that provides evidence that we can help improve people's cognitive abilities as they age?"

I've created a short 4-item quiz for you to test your knowledge in this area. Take the quiz. When you are done it will return you directly to this blog post (is that cool or what)?

Take the Quiz. Test your Knowledge of Aging's Effect on Cognitive Ability.

Click here to take the quiz

The quiz is based on an article by Christopher Hertzog, Arthur F. Kramer, Robert S. Wilson, and Ulman Lindenberger.

HEY, what are you doing? Go take the quiz first. There's research to show that the sort of questions I ask in the quiz will actually help you remember this topic. Doh!

The article by Hertzog, Kramer, Wilson, and Lindenberger is in Volume 9—Number 1 in the refereed scientific journal Psychological Science in the Public Interest that was just published in 2009. The title of the article is: Enrichment Effects on Adult Cognitive Development Can the Functional Capacity of Older Adults Be Preserved and Enhanced?

HEY, really. Go take the quiz first!!

Both of my parents (75+) are doing everything right according to the article.


Findings:

Cognitive ability does tend to decline with age. See graph from the article:



But notice that though AVERAGE cognitive ability declines there are wide ranges. And since I'm 51 years old as I write this, I'd like you to note that maximum cognitive performance seems highest near 50 years of age.

Can Cognitive Ability be Improved?

Yes, these researchers conclude that it can. Although they admit that more research is needed.

What Can Improve Cognitive Ability?

Well, they didn't look at everything that might impact cognitive ability, so we don't have a clear picture yet.

They highlighted the strongest findings in their conclusion:

"The literature is far from definitive, which is no surprise given the inherent difficulties in empirically testing the enrichment hypothesis. However, we believe there is a strong and sound empirical basis for arguing that a variety of factors, including engaging in intellectually and mentally stimulating activities, both (a) slow rates of cognitive aging and (b) enhance levels of cognitive functioning in later life." p. 41

"What is most impressive to us is the evidence demonstrating benefits of aerobic physical exercise on cognitive functioning in older adults. Such a conclusion would have been controversial in the not-too-distant past, but the evidence that has accumulated since 2000 from both human and animal studies argues overwhelmingly that aerobic exercise enhances cognitive function in older adults. The hypothesis of exercise-induced cognitive-enrichment effects is supported by longitudinal studies of predictors of cognitive decline and incidence of dementia, but also by short-term intervention studies in human and animal populations. The exercise-intervention work suggest relatively general cognitive benefits of aerobic exercise but indicates that cognitive tasks that require executive functioning, working memory, and attentional control are most likely to benefit." p. 41

They also noted some other more-tentative findings:

"...these data support the idea that a higher level of social engagement is related to a reduced risk of cognitive decline and dementia in old age. The basis of the association is not well understood, however." p. 33

"...these data suggest that chronic psychological distress may contribute to late-life loss of cognition by causing neurodeteriorative changes in portions of the limbic system that help regulate affect and cognition, changes that do not leave a pathologic footprint (e.g., dendritic atrophy) or whose pathology is not recognizable with currently available methods. These changes, when extreme, might actually be sufficient to cause dementia, but it is more likely that they contribute to cognitive impairment and thereby increase the likelihood that other common age-related neuropathologies are clinically expressed as dementia" p. 36

"...in observational studies that examine more than one lifestyle factor, cognitive activities appear to be the strongest predictor of cognitive change. However, this could be the result of several factors, including the following: (a) Rarely are physical activities characterized in terms of intensity, frequency, and duration; (b) the period across which activities are assessed has been different for cognitive and physical activities; (c) with one exception, activities have been treated as unidimensional in nature. Clearly, these issues require additional consideration in future studies." p. 39

They also offer a word of caution about software programs that are marketed as ways to improve cognitive ability:

"The majority of software programs marketed as enhancing cognition or brain function lack supporting empirical evidence for training and transfer effects. Clearly, there is a need to introduce standards of good practice in this area. Software developers should be urged to report the reliability and validity of the trained tasks, the magnitude of training effects, the scope and maintenance of transfer to untrained tasks, and the population to which effects are likely to generalize. Arriving at thisinformation requires experiments with random assignment to treatment and control groups, and an adequate sample description. Just as the pharmaceutical industry is required to show benefit and provide evidence regarding potential side effects, companies marketing cognitive-enhancement products should be required to provide empirical evidence of product effectiveness." p.48

So, to answer the quiz questions:

1. What happens to most people's cognitive abilities as they age from 50 years onward? Answer: Declines with age.

2. Is there valid research evidence from scientific refereed journals that suggests that people can improve their cognitive outcomes by engaging in certain activities? Answer: Solid evidence, but still some controversy.

3. Which of the following have been shown to improve cognitive ability as people age. Answer: The article didn't cover all the territory, but the strongest evidence is for (1) mentally and intellectually challenging activities and (2) aerobic physical activity.

4. Imagine that you work for a company that consists of a substantial number of workers over the age of 50. If you had a set budget to spend to improve their cognitive functioning, which of the following investments would garner the greatest results? Answer: Well, the research review does NOT compare the differences between (1) mentally challenging activities, (2) aerobic exercise, and (3) social engagement. However, see their overall conclusion below, which suggests that intellectual engagement and physically activity are key.

Their overall conclusion:

"We conclude that, on balance, the available evidence favors the hypothesis that maintaining an intellectually engaged and physically active lifestyle promotes successful cognitive aging." p.1

More research on benefits of exercise:

"Unlike the literature on an active lifestyle, there is already an impressive array of work with humans and animal populations showing that exercise interventions have substantial benefits
for cognitive function, particularly for aspects of fluid intelligence and executive function. Recent neuroscience research on this topic indicates that exercise has substantial effects on brain morphology and function, representing a plausible brain substrate for the observed effects of aerobic exercise and other activities on cognition." p. 1

They cite the potential for training interventions:

"...cognitive-training studies have demonstrated that older adults can improve cognitive functioning when provided with intensive training in strategies that promote thinking and remembering. The early training literature suggested little transfer of function from specifically trained skills to new cognitive tasks; learning was highly specific to the cognitive processes targeted by training. Recently, however, a new generation of studies suggests that providing structured experience in situations demanding executive coordination of skills—such as complex video games, task-switching paradigms, and divided attention tasks—train strategic control over cognition that does show transfer to different task environments. These studies suggest that there is considerable reserve potential in older adults’ cognition that can be enhanced through training." p. 1

But they offer a warning against one-shot interventions:

"There is no magic pill or no one-shot vaccine that inoculates the individual against the possibility of cognitive decline in old age. As noted earlier, participation in intervention programs is unlikely to affect long-term outcomes unless the relevant behaviors are continued over time." p. 47

What do we have to do?

Well, if we take our job seriously, we ought to heed the research. We can improve our fellow employees cognitive abilities as they age, so we ought to figure out how we might support that.

I certainly haven't got this nailed but if your company is interested, I think it would be fascinating to see what we might do.

I will join Annie Laures in talking about Learning Audits

FINAL DAYS TO REGISTER FOR MAY 27th SKILLCAST! Featured Presenters: Anne Marie Laures, CPT and Will Thalheimer, PhD "Is Your Learning Organization Healthy?"   Wednesday, May 27, 2009 1:00 pm ET/10:00 am PT (60 Minute Session) --  Is Your Learning Organization Healthy? How to Audit Your Learning Function and Create a Plan for Improvement Anne Marie Laures, CPT, Director, Learning Services, Walgreens Company and Will Thalheimer, PhD, President, Work-Learning Research This session will describe how a large, geographically-dispersed organization conducted a learning audit to determine the state of their learning function, the recommendations based on the audit, and the plan for changes. As a result of this session, participants will: Gain ideas and tips for conducting a learning audit Gain new ideas for using field expertise to support learning and career development Learn to blend formal learning strategies to informal learning opportunities Anne Marie Laures, CPT, is Director of Learning Services, which has been a repeated winner of ISPI's outstanding instructional and non-instructional awards. She has been responsible for supporting Walgreens' learning function within the company's corporate and operating divisions for over 30 years. She was a member of the group that designed the Certified Performance Technology (CPT) certification. She has presented at the annual conference 3 times and was invited to do an Encore Presentation at ISPI's 2005 conference in Vancouver.  Will Thalheimer, PhD is a learning-and-performance consultant and researcher who specializes in helping clients build world-class learning interventions. Dr. Thalheimer has worked in the field since 1985 as an instructional designer, simulation architect, project manager, trainer, and consultant. He founded Work-Learning Research in 1998 to provide research-based consulting services, workshops, and learning audits. Will speaks regularly, often receiving "best session of the conference" evaluations. His research-to-practice papers lead the industry with information backed by research and vetted with practical wisdom. Will Thalheimer has been invited twice to do Encore Presentations at ISPI's annual conference. Register Today by clicking here!! Designed to enhance the skills and knowledge of the Performance Improvement Professional, each month ISPI will feature the latest thinking from the experts you rely on for your continued professional development. In just one hour, you'll come away with new ideas, perspectives, and tools that you can put to work immediately. Put your focus on your own results, for a change, and join a SkillCast! ISPI Members:  $29 per SkillCast Non-Members:  $69 per SkillCast

Congrats to Hilary Wilder on her Fulbright Scholarship

Back in the early 1990's I was involved in a project which we ended up calling the Classroom, Inc. project. It was a joint venture between Teachers College, Columbia University (where I was a doctoral student); Morgan Stanley, the Mariposa Foundation led by Morgan Stanley's COO Lewis Bernard, and the New York City Public Schools. The project was designed to help at-risk kids in Brooklyn learn about business and to empower them to think like business leaders. It was a great project and has blossomed into a full not-for-profit organization that continues to do great work---taking it way beyond what we were able to do.

I was the project leader on the learning-development simulation-development side of the project. We built two computer-based simulations and accompanying learning materials. The programming wizard and educational technology guru on the project was Hilary Wilder, who played many other roles as well. It has been a pleasure to see her career unfold over the years since then.

Recently Hilary won a Fulbright scholarship to do educational technology work in Namibia, where she has been involved for years.

WP Perspective, a publication of William Paterson University, where Dr. Wilder is an Associate Professor, published the following article about Hilary's Fulbright. I couldn't be more proud. Congratulations Hilary!!

Hilary Wilder, associate professor in the educational leadership and professional studies department, has been named a Fulbright Scholar for the spring 2010 semester. She will be lecturing and conducting research at the University of Namibia (UNAM). The university, with an enrollment of 10,000 students, is the sparsely populated, African country’s only comprehensive four-year institution of higher education.

Wilder will also be establishing Namibia’s first and only master’s program in educational technology, her area of expertise.

“The people of Namibia are very open to new ideas,” Wilder says. “I’m really excited about the opportunity to develop an educational technology master’s degree program, in a country which so desperately needs it. I’m hoping this program will help create a solid cadre of local expertise in educational technology.”

Her work dovetails with a national development plan established by the Namibian government called Vision 2030, which calls for a shift to a knowledge-based economy by 2030. “Before independence in 1990, Namibia’s education system was based on apartheid practices and the majority of its children were taught through low-level rote-learning, often in rural bush schools with no resources. Today they are eager to move forward and ensure that their children will be successful and productive twenty-first century global citizens. The government, private sector, non-governmental organizations and international organizations such as the World Bank are all committed to making this happen“ she says.

Wilder will be working with teachers who will go on to become educational technology experts in their schools— helping their peers integrate technology literacy into the curriculum and their teaching. “There will be a trickle-down effect,” Wilder says. “My students will learn the technology integration skills, and take that back to their fellow teachers. The idea is that by 2030 the students who are then taught by those teachers will have acquired the technological literacy skills necessary to be a part of the knowledge-based economy.”

  
WP Perspectives, Vol.7 No. 1 April 2009, p. 7

One of the things that sets Hilary apart is that she is able to bring together a background in learning and educational theory and research (as a graduate of both Teachers College Columbia AND Harvard schools of education) WHILE at the same time bringing practical wisdom to her work (with her real-world experience as programmer, instructional designer, and educational technology specialist). The ability to bridge the theory/research side and the practice side is fundamental to making on-the-ground improvement in the learning field. We can all aspire to follow Hilary's path.

Interview with Roy Pollock. New Book Helps both Learners and Their Managers Make Training Work

It has been my pleasure and privilege to co-teach several learning measurement workshops with Dr. Roy Pollock, and to follow the important work that he and his colleagues have done at The Fort Hill Company over the years. I acknowledged their work by awarding Cal Wick, Fort Hill's Founding Father, the Neon Elephant Award back in 2006. I've also reviewed their ground-breaking book, The Six Disciplines of Breakthrough Learning, and have recently reviewed their new book, Getting Your Money's Worth from Training and Development.

Now, I have captured Roy in a video interview, that I think you'll enjoy and learn from.

You can purchase the book by clicking on the Amazon.com link below:

Again, I highly recommend the book. Read my book review to see how much.

Purchasing Performance Support

Here are some purchasing support tools you can use:

Buying wine imported to the US? Slate Magazine Guide
Read accompanying article: Slate Magazine Article

Buying cosmetics and want to avoid toxins: Environmental Working Group Cosmetics

Buying vegetables and want to avoid pesticides: Environmental Working Group Pesticides
Also download an iPhone App: Environmental Working Group Pesticide iPhone

Anybody know of others?

I was at a conference recently and somebody gave me a job aid the size of an employee security card. Too detailed, but potentially a great idea.

What other workplace performance job aids and performance supports have you seen?

Another tool in our toolbox.

John Karlin and the field of Human Factors

Just came across this presentation by John Karlin, who worked at Bell Labs back in the mid 1900's, and found it fascinating.

This was the guy who created the "Karlin dot" on rotary phones and helped people dial the phone. The dot gave dialers a spot to aim their finger at, increasing dialing speed (which if you remember rotary phones, was very important, especially if the number had lots of 7's, 8's, and 9's).

Interestingly, now the behavioral economists with their etched flies in urinals think they invented this idea.

Check out John Karlin's speech from 2003. On paper he's an engaging fellow.

Evaluation Objectives: A Brief Description

I got an email today from someone asking me about a term I created called "Evaluation Objectives." I realize that I have not actually written anything for public consumption on this, SO this blog post will suffice until my book on workplace learning is released. Apologies if the following is not completely clear.

The basic idea is that we ought to have evaluation objectives rather than learning objectives in the traditional sense.

Specifically, we need to decouple our learning objectives from our evaluation objectives so that what we evaluate is directly relevant. Of course our evaluation objectives and learning objectives have to be linked, but not necessarily with a one-on-one correspondence.

 

AN EXAMPLE

Suppose you want to train managers to be better at championing change efforts.

Traditionally, we might have objectives like:

The learner will be able to describe how people tend to resist change.

Or, put in a more performance-oriented fashion, a traditional objective might read:

The learner will engage in activities that lessen colleagues' resistance to change.

Examples of evaluation objectives might be as follows:

1. The learner will initiate a change effort within one month after the training ends and be successful in getting 75% of his/her colleagues to sign a public statement of support for the effort.

OR, if real-world compliance cannot be assessed, an evaluation objective might be something like:

2. In the "Change-Management Simulation" the learner will score 65 points out of a total possible of 90.

OR, if a simulated performance can't be created, an evaluation objective might focus on ratings by employees.

3. Two months after the training ends, the learners' colleagues will rate them on average at least 4.5 (of 6 levels) on the multi-rater 360-degree change-management scale on each of the 5 indices.

OR, if this can't be done, an evaluation objective might focus on a series of scenario-based questions.

4. On the 20-question scenario-based quiz on change management given two weeks after the course ended, the learner will get at least 17 correct.

NOTE: More than one evaluation objective can be used for any learning intervention.

THE POINT:

Evaluation objectives are NOT tied to individual learning points that have to be learned, though of course they are linked because both should be relevant to the overarching goals of the learning program.

THE BIG BENEFIT:

When objectives focus on the big picture, as compared to when there is a one-to-one correspondence between learning objectives and evaluation items, (1) they are more relevant, (2) the learners are more likely to see them as valuable and worth achieving, (3) organization stakeholders are more likely to see the evaluation results as having face validity, (4) the evaluation results will give us additional pertinent information on how to improve our learning interventions.

 

Anyway, I'm going to cover this in more detail in the book I'm working on, but for now, that's what evaluation objectives are all about.

Geary Rummler Tribute at ISPI. Will's Notes and Reflections.

Tonight at the ISPI conference in Orlando, I attended a tribute to Geary Rummler, who recently died after a long and distinguished career in the “Performance-Improvement Field.”

I didn’t know Geary, so I didn’t know how I would react or how long I would stay. I brought my laptop to do emails while I listened. I sat in the back of the cavernous ballroom.

I became transfixed as speaker after speaker who had worked closely with Geary talked about his work and the contributions he made to the field.

The following is my stream of consciousness note-taking with some later annotations. Not worthy of a tribute, but perhaps enough to help me remember some of Geary’s work—and perhaps enough to encourage YOU to take a look at his books and writings.

Notes and Annotations

Entrepreneurial experimentation. Science and sweat. Learning through trial and error.
Creating one-week program on programmed instruction. Didn’t have time to tell, to present objectives, etc. Showed them, had them practice. Curiosity, interested in what did NOT work, not just what DID work. Pre-Testing then Programmed Instruction then Maintenance of Behavior. Annotation: Even way back in the 1960’s and 1970’s someone was thinking about maintaining performance after training, and we are still struggling to get most of the field to do this.

After the initial workshop was developed and deployed, then Management of Behavior Change Workshop. Then General Systems theory workshop. Started small and specific—built up to systems.
During 1960’s Geary and colleagues wouldn’t do training without a thorough front-end needs analysis. 

Annotation: Hmmm. With today’s pressures, many are eschewing FEA.

Geary rebuffed a man who insulted one of his woman colleagues by telling him, “Shut up, she knows more than you do.”

Praxis (a Geary company) had a mission (we didn’t earn that much money speaker said)—to make the world a better place by improving the place where people worked. Annotation:

One of Geary’s former colleagues sang a song in tribute (a tear-inducing moment).

Geary was an engineer. Geary worked with Tom Gilbert. Helped plan Motorola University. Did coaching of functional managers of manufacturing curriculum. One of nice things about Geary is that as a consultant as he learned—he would even tell about the mistakes he made.

Quote from Geary (paraphrased): “Beware of false prophets, the HR people, who would rank and rate you, but don’t really understand the organization.”

Article: “You want performance, not just training.”

Did Situation analysis. Asked these questions: (1) What is happening now? (2) What should be happening?, THEN Define desired outputs. “As-is” “To-Be”

This all become six-sigma, etc. Annotation: Several people said Geary’s work became basis for the Six-Sigma movement, TQM, etc.

Geary said: You’ve got a lot of white space on the org chart you need to manage.

Annotation: People like Geary and his colleagues have been doing very valuable stuff for years (For example, they got cycle time from 17 weeks to 5 days), but why hasn’t this spread? Why hasn’t this performance-based approach gradually knocked-out the dominant training-based approach?

Rummler stuff got repackaged into 6-sigma stuff. Motorola bought license from Geary Rummler’s stuff into 6-sigma and TQM.

Geary’s true legacy: Changed the lens we all look through, moving from training to performance improvement.

Provided a common language:
White space, disconnects, organization as a system, cross-functional processes.

Geary Rummler: Managing the whitespace:

Performance Design Lab (Came out of retirement and started/joined this company).
Geary retired, but had to come back and knew the world didn’t get it and had to get back in the game, forming Performance Design Lab in 2000, where they were focusing on performance management systems (maintaining  improvement after the performance improvement).

Serious Performance Consulting. A Geary Book.

Geary always had to have a vehicle for packaging  his insights into his workshops and his books in ways that would make sense for people.

A speaker said, “He believed that it could be possible to create a prosperous society by construction a good system that will hold successful organizations and generate superior results.”

Two places that Geary touched, who seem to want to use Geary’s work to improve the world. Performance Improvement Institute in Cd. Obregon. Sonora Inst of Tehcnol and norwest of Mexico.

Geary Quote: “We cannot continue working the way we are expecting to get different results.”
Speaker quote (paraphrased): “Geary would share his materials more than anyone I know, I think because he was a learner, he wanted us all to know enough so that he could  discuss with us and we could learn together.”

Drinking alcohol and developing relationships with good discussions was a recurring theme. Many stories about drinks being drunk.

“If you put a good performer in a bad system the system wins every time.” Quote from Geary Rummler.

One speaker, quieted by tears, haltingly spoke about imagining Geary becoming a star in the sky…

Annotation: I really didn’t know that much about Geary’s work, but now I am motivated to learn more. Also, I’m glad I went because it gave me nice perspective on the field, even knowing that this “history” was filtered through the lens of tribute protocol.

Bottom line: I was touched and I’m motivated to learn more. Thanks to ISPI for providing this, for all the speakers (who I apologize for failing to capture their names), and for all the people who came to Orlando especially for the tribute.

Here are two of Geary's most popular books:

New Research Report on Using Culturally, Linguistically, and Situationally Relevant Scenarios

For years I've been compiling research from preeminent refereed journals that shows, time and time again, that aligning the learning and on-the-job performance context is key in supporting long-term remembering.

Now, I continue by focusing on cultural and linguistic context.

 

Read the research report.

Here are the major recommendations:

1. Utilize decision-making scenarios. Consider using them not just in a minor role—for example at the end of a section—but integrated into the main narrative of your learning design.
2. Figure out what the salient cues will be in the workplace situations that your learners will face in utilizing the content you are conveying. As much as possible, simulate those cues in your decision scenarios. Consider using multimedia to augment this effect, relying on excellent acting, directing, and set design to enable the context effects that will trigger remembering.
3. In simulating workplace cues, consider the range of cues that your learners will pay attention to in their work, including background objects, people and their facial expressions, language cues, and cultural referents.
4. Determine the most important points you want to get across AND the most important situations in which these points are critical. Then, provide extra repetitions spaced over time on these key points and situations.
5. Utilize culturally-appropriate objects, backgrounds, actors, and narrators in creating your scenarios. Consider not just ethnicity, but the many aspects of culture, including such things as socio-economics, education, international experience, immersion in popular culture, age, etc.
6. Pilot test new designs using valid evaluation methods to determine the most effective designs for your learners, your workplace situations, and your learning points.