Will at Work Learning

Fort Hill Gang Writes Again. Great New Book!!!

One of the biggest gaps in the learning-and-performance field occurs after the training is done. Learners fail to apply what they’ve learned and their managers fail to support training implementation. Fortunately, the Fort Hill gang writes again. Where their blockbuster book, The Six Disciplines of Breakthrough Learning, laid out a comprehensive process for getting training results, their new book (Getting Your Money’s Worth from Training and Development) provides a call-to-action for training’s most important players. Using the brilliantly diabolical approach of dividing the book in half—one half for learners, the other for managers—Jefferson, Pollock, and Wick provide an energizing action-plan to help organizations maximize training’s impact on job performance.

I’m so impressed with the Fort Hill guys. It seems that they (1) have looked deeply at the training-and-development trade, (2) found an area where time and time again we fail to do what’s right, and (3) written the perfect book to ensure that training maximizes business results. Too often in today’s organizations, training is seen as magic pill that works without alignment and support. In this double-dose of a book, Jefferson, Pollock, and Wick explode that myth, helping both learners and their managers bring potency to the training effort.

The design of the book tells the story itself. Managers read from one cover while learners read from the other cover. The book’s title stays the same—Getting Your Money’s Worth from Training and Development—but the subtitles change for the two audiences (i. e., A Guide to Breakthrough Learning for Managers; A Guide to Breakthrough Learning for Participants.). Only in partnership is training truly effective. The symbolism speaks loudly, but so too does the content, showing how both learners and their managers can work together to ensure that training transfers to on-the-job performance improvement.

The book is written in a conversational style. It speaks directly to the audiences in terms that will resonate. No motherhood and apple pie in the Fort Hill world. It’s all about results, wiifm’s, and tools. The example worksheets in the back of each book (remember it’s two books in one) are worth the cover price.

I recommend this book with the greatest enthusiasm. Companies ought to buy two copies for every training participant. One for the participant and one for his/her manager.

You can click the link below to learn more about the book (and go directly to Amazon to decide whether to purchase it).

Measuring Learning 8 – Book Review: Shrock & Coscarelli’s Book, Criterion-Referenced Test Development

Sharon Shrock and Bill Coscarelli recently completed the third edition of their important book, Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training. If this book isn't in your collection already, I'll give you a link below to buy from Amazon.com.

In this third edition, Sharon and Bill have updated the book from the second edition (published in 2000) in some critical ways. One of those ways is truly transformational for the workplace learning and performance field. I'll get to that in a minute.

Also updated is the excellent chapter at the end of the book by Patricia S. Eyres, a lawyer with employment-law credentials. Her chapter covers the legal ramifications and guidelines in dealing with employee testing, especially as that testing affects employee selection, advancement, and retention. She has updated her chapter with new case law and legal precedent from that in the second edition. Most people in the training field have very little knowledge of the legal ramifications of testing, and I'd recommend the book for this chapter alone—it's a great wakeup call that will spur a new appreciation of the legal aspects of testing.

In the second edition, Shrock and Coscarelli put forward what they call the "Certification Suite." In criterion-referenced testing, the goal is to decide whether a test taker has met a criterion or not. When they have met the criterion, they are said to be "certified" as competent in the area on which they were tested. The Certification Suite has six levels, some which offer full Certification and some which offer Quasi-Certification:

Certification

1. Real World
2. High-Fidelity Simulation
3. Scenarios

Quasi-Certification

1. Memorization
2. Attendance
3. Affiliation

As the authors say in the book (p. 111), "Level C represents the last level of certification that can be considered to assess an ability to perform on the job."

The truly transformational thing offered by Shrock and Coscarelli is that Level D Memorization, in the second edition of the book, was considered to offer Certification. NO MORE!! That's right. Two of our leading thinkers on testing say that memorization questions are no longer good enough!!!!!!!!!!!!!! Disclosure: In speaking with Bill Coscarelli in 2006, I gently encouraged this change. This is mentioned in the book, so it's not like I'm bragging. SMILE.

I love this, of course, because it follows what we know about human learning. For tests to be predictive of real-world performance, they have to offer similar cues to those that learners will face in the real world. If they offer different cues—like almost all memorization questions do—they are just not relevant. And, from a learning standpoint (as opposed to a certification standpoint) memorization questions won't spur spontaneous remembering through the triggering mechanism of the real-world cues.

This literal and figurative raising of the bar—to move it beyond memorization—should shake us to our core (especially since this is one of the few books on assessment that covers legal stuff—so it may have some evidentiary heft in court). If the compliance tests at the end of your e-learning programs are based on memorization questions, you are so in trouble. If your credentialing is based on completion (and 85% of our respondents in the eLearning Guild research report said they utilized completion as a learning measure), you are in even worse trouble. And, of course, if you ever thought your memorization-level questions supported learning, well, sorry. They don't! At least not as strongly as they might.

Have you bought the book yet? You should. You ought to at least have it around to show management (or your clients) why it's important (absolutely freakin' critical) to use high-value assessment items.

I've got some quibbles with the book as well. They list 6 reasons for testing. I've recently come up with 18, so it appears they're missing some, or I'm drinking too much. I also don't like the use of Bloom's Taxonomy to index some of the recommendations. In short, Bloom's has issues. I don't like the way they talk about learning objectives. They use the methodology of relying on a single objective to guide the process of both instructional design and evaluation. I am now advocating to free instructional-design objectives from the crazy constraint of being super-glued to the evaluation objectives. They need to be linked of course, but not hog-tied. I wish they emphasized more strongly the distinction between testing to assess and testing to support learning. They are different animals and most of us are confused about this.

Overall, it's a great and thoughtful book. I bought it. You should too.

Here's a link that will let you click here to buy.

The Learning Measurement Series will continue in January...

(But watch to see who wins this year's Neon Elephant Award, which I'll announce on Saturday (December 22nd 2007). The winner(s) is/are all about learning measurement.)

Research-to-Practice White Paper, Questioning Strategies for Audience Response Systems

Questioning Strategies for
Audience Response Systems:
How to Use Questions to Maximize
Learning, Engagement, and Satisfaction

by Dr. Will Thalheimer

Introduction

The buzz in the learning industry is focused on e-learning, m-learning, wikis, and blogs; but one of the most powerful learning technologies is being overlooked, probably because it's an in-the-classroom technology—audience response systems. In this research-to-practice white paper I offer a blueprint for how to use audience response systems to maximize higher-order learning in the classroom and beyond.

What One Reader Wrote to Me

Dr. Thalheimer,

Just wanted to drop you a little note this morning to express my gratitude for your paper "Questioning Strategies for Audience Response Systems: How to Use Questions to Maximize Learning, Engagement, and Satisfaction."

A friend recommended that I read it to prepare for a Higher Order Questioning staff development class that she and I are teaching together (in conjunction with some CPS [audience response] training we're offering). To tell you the truth, I really wasn't looking forward to reading it because I expected it to be dry and full of boring I'm-trying-to-sound-snobbily-intellectual writing, but I LOVED it. :)

I enjoyed your approachable style and dry sense of humor so much I read all the way through (including the endnotes!) and had many a good laugh along the way. In addition to being a blast to read, the paper challenged and inspired me to find new ways to push my questioning skills to a higher level for the next school year.

Thanks again, for the inspiration and for the great read. I'll be checking out your website later today and hope to find that equally enjoyable.

Sincerely,

Liz Walhof

Spanish Teacher
Colorado

From the Paper's Introduction

"Audience response systems have enormous potential for transforming lectures from dry recitals into rich jam sessions of deeply resonant learning. The technology is widely available, but the key to success is not in the technology; it’s in the instruction. To maximize meaningful learning, instructors must become adept in using questioning and discussion techniques. Unfortunately, some of us may come to believe that we can simply sprinkle our lectures with a few multiple-choice questions. This approach is emphatically inadequate, and is simply not worthy of our profession.

This report provides a near-exhaustive list of questioning strategies, and a comprehensive guide on using questions to facilitate classroom learning. No other resource exists that is research-based and comprehensive, while also being practical and useful. It has been designed specifically to provide practical guidance for trainers, teachers, and professors so that their learners—whether they are eight, forty-eight, or eighty years old—can experience deep and meaningful learning."

Special thanks to eInstruction for agreeing to license the paper for distribution to their clients. Such underwriting helps move the audience-response field forward and demonstrates an enlightened commitment to effective learning in classrooms of all types throughout the world. Other underwriting opportunities are available for research on audience-response learning. Contact Dr. Thalheimer with inquiries.

Additional Information

• Number of Pages: 124
• Number of Research Citations: 54
• Publication Date: March 2007
• Available to you Immediately as downloadable Electronic file (PDF).
• Purchasing utilizes industry-leading security.
• 100% Satisfaction Guaranteed.
• Cost: $40.00 (US)

Click here to purchase...

Best Books from 2006

My list of Best Books of 2006 for the Workplace Learning-and-Performance Field are as follows:

You can purchase these books directly at the Will's Blog Amazon.com Store

Wick, Pollock, Jefferson, and Flanagan (2006). The Six Disciplines of Breakthrough Learning: How to Turn Training and Development Into Business Results.

This book is nothing short of revolutionary, providing a comprehensive analysis of how to create learning interventions that have performance impact. I previously reviewed this book and awarded the lead author the 2006 Neon Elephant Award.

Pfeffer and Sutton (2006). Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management.

This book provides an evidence-based critique of many of today's most common management fads. The information in the book is critical to learning-and-performance professionals responsible for leadership and management development efforts (BECAUSE CONTENT MATTERS), but also as a shining example for our field (BECAUSE IF WE'RE NOT USING EVIDENCE-BASED PRACTICES, WE'RE NOT GETTING OPTIMAL RESULTS).

Clark, Nguyen, AND Sweller (2005). Efficiency in Learning: Evidence-Based Guidelines to Manage Cognitive Load

Another in Ruth Clark's excellent series of research-to-practice books where she partners with leading learning researchers. Here she joins with Frank Nguyen and partners with John Sweller, developer of cognitive load theory. The book tells us how to create learning interactions that avoid overloading our learners' limited working memory and perceptual-channel capacities.

Rossett and Schafer (2006). Job Aids and Performance Support: Moving From Knowledge in the Classroom to Knowledge Everywhere

Allison Rossett and Lisa Schafer have created a great book on performance-support systems. With an introduction from Gloria Gery, this book nicely outlines the power and potential of performance support to replace and/or augment traditional training interventions. The book is written using a sensible, research-based approach. It describes when performance support is valuable, and when it's not.

Israelite (2006). Lies About Learning: Leading Executives Separate Truth from Fiction In a $100 Billion Industry

Larry Israelite edits this book as the various authors take on many of the most important myths in our field. This kind of book is critical in our typically lost-in-denial field.

Allen (2006). Michael Allen's E-Learning Library: Creating Successful E-Learning: A Rapid System For Getting It Right First Time, Every Time

Michael Allen knows instructional design from the ground floor up. For four decades he's led the field. Now he shares his hard-earned knowledge with the rest of us. This book provides a model for real-world instructional designers to create effective e-learning. This is the first in Michael Allen's ongoing Pfeiffer e-learning series.

You can purchase these books directly at the Will's Blog Amazon.com Store

Book Review -- Wick, Pollock, Jefferson, & Flanagan (2006).

The book, The Six Disciplines of Breakthrough Learning: How to Turn Training and Development into Business Results; by Calhoun Wick, Roy Pollock, Andrew Jefferson, and Richard Flanagan; is one of the most important books published in the training and development industry in a very long time.

There are three ways to access my book review:

1. Listen online (Click on the play button)
2. Listen on your computer or MP3 play (Click on the MP3 link)
3. Read the review below.

The audio is about 20 minutes.

MP3 File

Book Review by Will Thalheimer
President of Work-Learning Research, Inc.

Book: The Six Disciplines of Breakthrough Learning: How to Turn Training and Development into Business Results

Authors: Calhoun Wick, Roy Pollock, Andrew Jefferson, and Richard Flanagan

Publisher: Pfeiffer

Publication Date: April 2006

Introduction

The learning-and-performance field—of which I am a devoted member—hasn’t had a really big idea since the performance-improvement crusade began gathering momentum in the 1980’s. But now, thanks to the work of Cal Wick, Roy Pollock, Andrew Jefferson, Richard Flanagan and their colleagues at the Fort Hill Company, we finally have a new innovation—a systematic method for training follow-through.

It’s not a surprise that training can only be effective if learners put what they learn into practice. What Wick and company have done is demonstrate the feasibility of driving training transfer into the flow of work. Their book is really a culmination of years of exploration as they bravely embraced the exhausting and dangerous work of pioneers.

They’ve taken an evidence-based approach to learning design—grappling with real-world clients, making careful observations, gathering data, utilizing research findings, and fine-tuning their practices. Perhaps most importantly, they’ve created a breakthrough technology that enables training-and-development leaders to push learning results into the actual workplace.

E-learning pundits haven’t recognized it yet, but Fort Hill’s Friday5s training-follow-through software (along with competitive products like ZengerFolkman’s ActionPlan Mapper) may be the most disruptive e-learning technology yet devised. While web-meeting platforms, LMS’s, rapid authoring tools, and even Google may seem potent, they don’t change training effectiveness as much as a good training follow-through system.

Wick, Pollock, Jefferson, and Flanagan may enjoy promoting Fort Hill products, but they go out of their way to craft a broader message in their brilliant new book, The Six Disciplines of Breakthrough Learning: How to Turn Training and Development into Business Results. The authors lay out a devastating analysis of the current state of training practice—not by being negative—but by illustrating with cases, examples, and research how to do training right.

The book is nothing short of revolutionary. Unfortunately, in our dysfunctional field not everyone will take up arms against their own ineffective practices, but the book provides solid guidance to the enlightened soldiers in our midst. If you want to improve on-the-job performance and business results, this book is a guiding light.

Changing the Paradigm and Technology of Learning

In the flow of our everyday lives, the world as we know it follows predictable patterns. Things change, but they change predictably. Every once in while, however, something new appears—an innovation or idea so strange and yet so perfectly in tune with the cravings, resources, and zeitgeist of the time that it changes everything.

Disruptive technologies like electricity, phones, computers, and the internet have produced powerful ripples through the human fabric. Automobiles not only displaced the horse, they enabled the rise of the middle class, the building of suburbs, and intellectual and social freedom for young adults. Paradigm shifts and scientific discoveries create the same effects, changing the way we see the world—changing the possibilities. If not for the ideas of Jesus, Darwin, Gandhi, Confucius, Freud, Einstein, Watson and Crick, Kuhn, and others, we would live in a different world.

The last great disruptive innovation to arise within the learning-and-performance field was the move away from “training” and toward “performance improvement.” Unfortunately, that movement is not yet complete. The hard truth is that we talk more about on-the-job results than we achieve them.

In the move from training to performance improvement, something got lost. Performance gurus often badmouth training as inadequate, but they give short shrift to its strengths, and are blind about how to design the complete training experience to make training work. This kind of blindness is endemic in our field for two reasons, (1) because we have so little understanding of the basics of human learning, and (2) because we rarely evaluate our performance.

Thankfully, Cal Wick and his team (as well as a few others) have tired of training’s big lie. They know that training can be powerful—if only the right processes and procedures are put into place. Because they understand learning, they can envision a systematic set of guidelines that work. Because they measured the performance of their learners, they have been able to fine-tune their recommendations.

The Six Disciplines is poised to become one of the most important books in the learning-and-performance field. Not since the publication of Dana Gaines Robinson and James C. Robinson’s book on performance consulting or the seminal work of Bob Mager on performance-based instructional design, has our field been offered a new system of thinking—a new way to do our jobs as learning-and-performance professionals.

The Book’s Overarching Message

The book proposes six disciplines and offers scores of recommendations, but it’s central message is that what happens after training is just as important—and probably more important—than the training itself.

The six disciplines are:

1. Define Outcomes in Business Terms
2. Design the Complete Experience
3. Deliver for Application
4. Drive Follow-Through
5. Deploy Active Support
6. Document Results

Wick, Pollock, Jefferson, and Flanagan suggest that training ought to be conceptualized with a new finish line.

The “finish line” for learning and development has been redefined. It is no longer enough to deliver highly rated and well-attended programs; learning and development’s job is not complete until learning has been converted into results that matter to the business. (p. 13)

This new finish line enables us to see possibilities beyond the completion of smile sheets. A learner’s job—indeed an organization’s job—is not done when the classroom door swings shut.

The authors also emphasize the importance of visualizing training as something that occurs within an expanded timeline. Before-training efforts and after-training efforts are just as critical as the training efforts themselves. Particularly important are the after-training efforts because they focus learner attention on implementing the learning, reinforce fading memories, and transform the process of learning from an individual pursuit to an organizational responsibility. Learning changes from a love-it-and-leave-it experience to a system of reciprocal reinforcement where the results are measured in on-the-job performance.

The Book’s Evidence

The authors cite lots of organizational research to back up their claims, from thinkers and researchers like Broad, Brinkerhoff, and Newstrom. And, the notion of a new finish line is entirely consistent with the research on fundamental learning factors—the kind of research I’ve been working with for almost a decade. For example, we know learners forget most of what they learn—unless that information is reinforced in the workplace. Each one of the six disciplines push us to design an expanded learning experience, one that focuses on workplace implementation, not training per se.

Other forms of evidence are equally important. In addition to research from refereed journals, the book details dozens of real-world learning executives describing their successes in broadening the conception of training and implementing the six disciplines. Wisdom from learning leaders was relayed from these and other organizations: Sony, Gap, 3M, Humana, BBC, Center for Creative Leadership, General Mills, Corning, Forum, University of Notre Dame, Honeywell, AstraZeneca, Pfizer.

Evidence of the effectiveness of technology-based training follow-through is described using data from the powerful methodology of control-group designs. Graphs and text clearly illustrate the results. For example, page 128 conveys how “Use of a Follow-Through Management System Increased Managers’ Awareness of Their Direct Reports’ Development Goals from 40 Percent to 100 Percent.”

While the majority of books in our field fail to convey more than a few breadcrumbs of credible evidence, The Six Disciplines hits for the triple crown, utilizing refereed research, experience of real-world learning leaders, and data from control-group studies. In our field, it simply doesn’t get any better than this.

The Book’s Design

The book is well organized, with an introductory chapter, a summary chapter, and one chapter for each of the six disciplines described in the title. Each chapter ends with a nice twist—two lists of action points; one for “learning leaders” and one for “line leaders.” There are many design examples such as this that demonstrate that the authors are really serious about on-the-job performance. The book utilizes some valuable repetitions of key points. The text design makes reading a pleasure. Quotations are pithy and relevant. Examples are illustrative of the main points in the text.

I read every page of the book, so I can tell you with confidence that it is well written. There are hundreds of specific recommendations throughout the book. I found many insights that I hadn’t thought of—ideas that I will use in my work as a consultant, instructional-design strategist, and creator of training. The graphs and charts are clear and there are some very useful templates. For example, the first chapter concludes with the “Learning Transfer and Application Scorecard,” a 10-item questionnaire. It’s a powerful tool because—and this is my opinion not the authors’—most current training programs will fail miserably when measured by these questions. I’d bet that most training programs will have low scores on ALL 10 items of the scorecard.

I have two almost insignificant complaints about the book. First, the cover is uninspired. The book deserves better. Second, the six disciplines are shoehorned into starting with the letter “D,” in a way that is more misleading than it should be. For example, the second “D” stands for Design the Complete Experience. The author’s emphasis is on the complete experience, but the shorthand version “Design” connotes the traditional instructional-design notion of design—a notion completely inadequate; as the authors argue persuasively in the actual text.

The Book’s Recommendations

The book is jam-packed with recommendations, so I’ll only convey a few of the specific recommendation here. You really ought to buy The Six Disciplines and read it and share it with everyone you know who cares about doing training right. Here’s my short list:

1. View training follow-up as part of every training intervention.
2. Get learners’ managers involved before and after training.
3. Evaluate your training programs to determine whether they’re working and to improve subsequent training.
4. Before designing a training program, determine what learners will be doing better and differently after the program. Be clear about what evidence will be acceptable to determine success.
5. Understand the business. Be proactive in suggesting training-and-development solutions. Check your understanding with line leaders.
6. Utilize a technology-based training-follow-through system to drive learning application and accountability.
7. Utilize evidence-based practices, including research-based instructional design and after-training evaluation.
8. Avoid “dense-pack education—the tendency to cram every conceivable topic into a program of a few days.”
9. Focus on creating transfer during all phases of training—while designing the training, while delivering it, and during follow-up.
10. Consider using senior executives to teach leadership—it is one of the fastest- growing trends in executive education.
11. During training, stop after each topic and ask participants questions that challenge them to think about applying what they know.
12. Learners should develop “learning transfer objectives” and be prepared to work toward them while back on the job.
13. Send learners’ objectives to the learners’ managers to increase follow-up application and accountability.
14. Utilize Marshall Goldsmith’s “feedforward” techniques to help learners generate ideas for training application.
15. Recognize that there are factors that decrease the likelihood that learners will put their learning into practice, and that the impact of these factors can be minimized only through a systematic follow-through process.
16. Utilize reminders to facilitate memory and spur on-the-job application of training.
17. Hold employees accountable for making effective use of the training they receive.
18. Consider coaching as a complement to training, providing learners with coaches to increase the likelihood of energetic and appropriate application.
19. Learning programs that “demonstrate sound, thorough, credible, and auditable evidence of results are able to garner additional investment; those that cannot are at risk.”
20. Learning and development units within companies need to communicate their results to the organization using multiple communication attempts and various communication channels.

How do Your Learning Programs Rate?

As I mentioned earlier, the Fort Hill Company has developed a Learning Transfer and Application Scorecard (displayed on pages 10 and 11) that targets the most important and leveragable characteristics that make training effective. Every training program ought to be measured with this scorecard. To get an idea of how well your training stacks up, I’ve included three of the ten items. I changed the wording slightly to help you make sense of the items before you read the book. How well do your training programs do the following?

• After the program, participants are reminded periodically of their post-learning objectives and of opportunities to apply what they learned.
• Participants’ managers are actively engaged during the postprogram period. They review and agree on after-learning objectives, and expect and monitor the progress that learners are making in applying what they’ve learned.
• The design of the learning program covers the entire process from initial invitation to attend, through the learning sessions, and through on-the-job application and measurement of results.

Summary

The Six Disciplines is the most important book written in our field in quite some time. It provides a comprehensive system to make training effective. Its radical new nugget of truth is its insistence on training follow-through. The book’s ideas are evidence-based and are consistent with the human learning system. The messages in the book have been tested and refined in the real world. Tools are available (for example, Fort Hill’s Friday5s follow-through management system) that make the recommendations actionable.

Training Follow-Through Systems

I am aware of two training follow-through systems, Fort Hill’s Friday5s, and ZengerFolkman’s ActionPlan Mapper. I have formally reviewed the ZengerFolkman product, but have yet to put my review of Friday5s on paper. Both are powerful programs. Friday5s may have an edge given its longer tenure in the marketplace and its ability to provide learning reminders, not just reminders about learning transfer objectives. My recommendation is that you test them for yourself.

Notes

My contact information is as follows. Will Thalheimer, PhD, is a learning consultant and researcher. He can be reached at 617-718-0067, and will.thalheimer@work-learning.com, and www.work-learning.com, and www.willatworklearning.com.

The Fort Hill Company is available at 302-651-9223, and sales@forthillcompany.com, and www.forthillcompany.com.