Will at Work Learning

Evaluation Objectives: A Brief Description

I got an email today from someone asking me about a term I created called "Evaluation Objectives." I realize that I have not actually written anything for public consumption on this, SO this blog post will suffice until my book on workplace learning is released. Apologies if the following is not completely clear.

The basic idea is that we ought to have evaluation objectives rather than learning objectives in the traditional sense.

Specifically, we need to decouple our learning objectives from our evaluation objectives so that what we evaluate is directly relevant. Of course our evaluation objectives and learning objectives have to be linked, but not necessarily with a one-on-one correspondence.

 

AN EXAMPLE

Suppose you want to train managers to be better at championing change efforts.

Traditionally, we might have objectives like:

The learner will be able to describe how people tend to resist change.

Or, put in a more performance-oriented fashion, a traditional objective might read:

The learner will engage in activities that lessen colleagues' resistance to change.

Examples of evaluation objectives might be as follows:

1. The learner will initiate a change effort within one month after the training ends and be successful in getting 75% of his/her colleagues to sign a public statement of support for the effort.

OR, if real-world compliance cannot be assessed, an evaluation objective might be something like:

2. In the "Change-Management Simulation" the learner will score 65 points out of a total possible of 90.

OR, if a simulated performance can't be created, an evaluation objective might focus on ratings by employees.

3. Two months after the training ends, the learners' colleagues will rate them on average at least 4.5 (of 6 levels) on the multi-rater 360-degree change-management scale on each of the 5 indices.

OR, if this can't be done, an evaluation objective might focus on a series of scenario-based questions.

4. On the 20-question scenario-based quiz on change management given two weeks after the course ended, the learner will get at least 17 correct.

NOTE: More than one evaluation objective can be used for any learning intervention.

THE POINT:

Evaluation objectives are NOT tied to individual learning points that have to be learned, though of course they are linked because both should be relevant to the overarching goals of the learning program.

THE BIG BENEFIT:

When objectives focus on the big picture, as compared to when there is a one-to-one correspondence between learning objectives and evaluation items, (1) they are more relevant, (2) the learners are more likely to see them as valuable and worth achieving, (3) organization stakeholders are more likely to see the evaluation results as having face validity, (4) the evaluation results will give us additional pertinent information on how to improve our learning interventions.

 

Anyway, I'm going to cover this in more detail in the book I'm working on, but for now, that's what evaluation objectives are all about.

Assessment Guru Hints at Trends Relevant to Learning Professionals

Learning professionals (like me) can often gain insights about our industry from people in the field who have different vantage points than our own. I recently talked with Eric Shepherd, CEO of Questionmark, to get a sense of our industry and how it has been affected by the bad economy. Eric has been a good friend and long-time supporter of my research over the years and I've come to value his counsel.

Questionmark is the leading provider of assessment software according to a recent eLearning Guild study. I thought from his perch overseeing all-things-assessment, Eric might be able to give us some unique insight into the learning-and-performance field in general.

Check out my interview with him at the recent Guild conference. I divided it into two parts to make viewing easier.

Part 1: What trends do you see that we may be missing? 

Another Smile Sheet Improvement

Last year I wrote at length about my efforts to improve my own smile sheets. It turns out that this is an evolving effort as I continue to learn from my learners and my experience.

Check out my new 2009 version.

You may remember that one of the major improvements in my smile sheet was to ask learners about the value and newness of EACH CONCEPT TAUGHT (or at least each MAJOR concept). This is beneficial because people respond more accurately to specifics than to generalities, they respond better to concrete learning points than to the vague semblance of a full learning experience.

What I forgot in my previous version was the importance of getting specific feedback on how well I taught each concept. Doh!

My latest version adds a column for how well each concept is taught. There is absolutely no more room to add any columns (I didn't think I could fit this latest one in), so I suppose this may allow diminishing returns on any more improvements.

Check it out and let me know what you think.

Security Issues on Learning Assessments

Eric Shepherd of Quesionmark offers a white paper on his blog that goes into some nice depth to help you think through the level of security you might need in your learning assessments.

As I've written here before, there are many reasons to provide learning assessments, each having it's own design needs--and security needs as well.

Check out Eric's blog (and the first hyperlink) to get the white paper.

Famous Measuring-Training Story from Neil Rackham

Here's the link to Neil Rackham's famous story on the dangers of measuring training only with smile sheets.

Donald Kirpatrick Answers Questions

Donald Kirkpatrick Answers Questions. Click to view.

Learning Measurement eLearning Guild Stuff

I've been busy again thinking about the nexus between LEARNING and LEARNING MEASUREMENT.

You can peruse some of my previous thoughts on learning measurement by clicking here.

Here is a brand new article that I wrote for the eLearning Guild on how to evaluate Learning 2.0 stuff. Note: Learning 2.0 is defined (by the eLearning Guild) as: The idea of learning through digital connections and peer collaboration, enhanced by technologies driving Web 2.0. Users/Learners are empowered
to search, create, and collaborate, in order to fulfill intrinsic needs to learn new information. Evaluating Learning 2.0 differs from evaluating traditional Learning 1.0 training for many reasons, one of which is that Learning 2.0 enables (encourages) learners to create their own content.

Steve Wexler, Director of Research and Emerging Technologies at the eLearning Guild, and I are leading a Webinar on Thursday September 4th on the current state of eLearning Measurement. We've got some new data that we're hot to share.

Finally, Roy Pollock, one of the authors of the classic book, Six Disciplines of Breakthrough Learning, and I are leading a one-day symposium on measuring learning at the eLearning Guild's DevLearn 2008 conference in November. It's a great chance to go to one of the best eLearning conferences around while working with Roy and I in a fairly intimate workshop, wrangling with the newest thinking in how to measure learning. Choose Symposium S-4. Note that it may not show Roy's information there yet--the Guild is still working on the webpage--but let me assure you that Roy and I are equal partners in this one.

What Work-Learning Audit Reveals

I recently completed one of the most comprehensive work-learning audits I've ever been asked to do for a major U. S. retailer. The goal of the audit was to find out how their learning programs AND work-learning environment were supporting the stores in being successful. The audit involved (1) structured and unstructured interviewing with all levels of the organization, especially with store personnel, (2) focus groups held across the country with specific groups of store personnel (e.g., clerk, store managers, assistant managers, etc.), (3) task force meetings with senior line managers and representatives throughout the company, (4) learning audits of e-learning courses, (5) learning audits of a mission-critical classroom course, (6) review of company artifacts (CEO messages, publications, databases, intranet, etc.), (7) interviews with learning-and-performance professionals, (8) discussions of business strategy, (9) discussions regarding corporate information and data-gathering capabilities, (10) job shadowing, (11) store observations, etc.

Who/What Do Workers Learn From?

One of the most intriguing results came out of a relatively simple exercise I did with focus-group participants. The following is a rough approximation of those results.

What I did was ask focus-group participants who they learned from. I would hold up a large 6 x 8 index card with a position label on it, for example, "District Manager," "Clerks," or "Corporate." The group would shout out where they thought that card should go on a large diagram I had created on the wall. I would place it on the wall in a particular category based on the verbal responses and then we would negotiate as a group to determine it's final positioning. So for example, participants could say that they learned the following amounts from that person/position, and we often compromised using in-between placement:

• Learned Most
• Learned a Lot
• Learned Some
• Learned a Little
• Learned Least
• Had Little/No Contact with

See the diagram below for a rough example. This one is actually a composite based on several focus groups and more than one position. It gives a fair representation for how frontline retail clerks responded. Note that the orange boxes represent fellow employees, while the blue boxes represent other groups of people or things that they learned from.

There are several key insights from these results:

• People learn the most from those who they work closely with.
• People learn the most from their experience doing the job.
• People learn the most from their self-initiated efforts at learning.
• The more contact, the more learning (for the most part), however there are benefits from learning from experts (e.g., store managers, head clerks), though the worker has to have at least some signicant contact with them to create this benefit. You'll notice that district staff have only a little impact and regional and corporate staff have none.
• E-learning is seen as somewhat facilitative but not a place where workers learn the most. This result may be organization specific as different e-learning designs and implementations might easily move this result higher or lower.

Frontline clerks didn't get much from company magazines and the like, but managers (not represented in the results above) did find value in these. Store managers also reported that networking with other store managers was on of the "Learn Most" entries for them. For this company, this network was even more important than learning from their district managers (their direct bosses). This makes sense because their network is more accessible throughout the heat of the daily grind.

These results were eye-opening for my client, and they are still wrangling with the implications. For example, district managers and district training staff seemed to produce very little learning benefits. So, should their roles for learning be de-emphasized or re-emphasized?

These types of result have to be understood in the larger data-gathering effort of course. Analyzed alone, they suffer from the problem of de-contextualized self-report data. Combined with multiple other data sources, they paint a really robust picture of an organization's learning environment.

Informal Learning, Social Networks, etc.

Vendors are out and about in our field now selling the benefits of complicated and expensive analysis tools for looking at how people learn through so-called informal on-the-job mechanisms. The example above shows that if you don't have the big bucks, there are simpler ways to get good data as well. 

Evaluating Learning 2.0

The eLearning Guild has asked me to lead a discussion at their upcoming conference (Guild Annual Gathering) on how we might think about evaluating Learning 2.0 interventions.

I'd welcome your examples and insights.

For those who don't know what "Learning 2.0" means, I'll forgo my cynical answer, and say that others describe Learning 2.0 as learning that enables learner-creation of information, comparing Learning 2.0 to the stereotypic traditional model where the teacher teaches and the learner absorbs the information (or the e-learning delivers content and the learner absorbs).

So, Learning 2.0 is said to include such things as Wikis, Blogs, Learner Portfolios, Media Development and Sharing by Learners, Informal Learning, etc.

Here's a few things I'm contemplating:

1. Traditional metrics are certainly appropriate because bottom-line we want to know if Learning 2.0 interventions produce learning, enable on-the-job performance, and produce desirable individual and organizational outcomes.
2. Comparisons to other methods of learning. Especially important to see if Learning 2.0 methods (on the positive side) create more elaborate mental models, produce more satisfaction, etc. and (on the negative side) waste time, create unproductive distractions, communicate incorrect or inappropriate information.
3. We need to measure not only what HAS been learned, but also on what MAY BE LEARNED IN THE FUTURE. It could be, for instance, that Learning 2.0 is inefficient for learning anything specifically, but enables faster future learning in the same area of inquiry.

Here's where I can use your help. Let me know if you know of any of the following:

• Rigorous research studies on Learning 2.0 interventions
• Anecdotal evidence on Learning 2.0 interventions

Better yet, join me at the eLearning Guild's Annual Conference -- Specifically the Learning Management Colloquim and discuss this in real time.

Neon Elephant Award 2007 to Sharon Shrock and Bill Coscarelli

I'd like to announce the winners of the 2007 Neon Elephant Award, given this year to Sharon Shrock and Bill Coscarelli for advocating against the use of memorization-level questions in learning measurement and for the use of authentic assessment items, including scenario-based questions, simulations, and real-world skills tests.

The Neon Elephant Award is awarded to a person, team, or organization exemplifying enlightenment, integrity, and innovation in the field of workplace learning and performance. Announced on the day of the winter solstice—the day of the year when the northern hemisphere turns away from darkness toward the light and hope of warmer days to come—the Neon Elephant Award honors those who have truly changed the way we think about the practice of learning and performance improvement. Award winners are selected for demonstrated success in pushing the field forward in significant paradigm-altering ways while maintaining the highest standards of ethics and professionalism.

See the full announcement by clicking here...