Will at Work Learning

More Evidence that the Internet is Not the Ideal Learning Tool

An article on the behavior of websurfing.

Wiki's -- Failure or Boon of Online Collectivism?

I've been suspicious of the wiki bandwagon for quite some time. An earlier piece I wrote summed up my concerns. Fortunately, someone else has come along and written more deeply and elegantly on the topic. Check out Jaron Lanier's piece, DIGITAL MAOISM: The Hazards of the New Online Collectivism, published 5-30-2006 in The Edge an online publication.

Jaron Lanier's piece points out several things:

1. Sometimes the collective is wise. Sometimes it is stupid. Sometimes it is very stupid.
2. We, as individuals, have difficulty knowing what's good and what's not good.
3. There is very often a need for wise and knowledgable editors to filter messages.
4. Google search results have a similar problem.
5. Websites and blogs that aggregate info have a similar problem.
6. Our democracies may be threatened by this collective stupification.

Relating This to the Learning-and-Performance Field

My main concern with wiki's is that information from real experts can be stupified to the mediocre averaging of above-average minds. I have a wiki design that overcomes this problem, but I need a generous funder to make it a reality. BIG HINT to you rich philanthropic types. SMILE.

Wiki's aren't the only newtech fetish that exhibits the filtering problem. Blogs that simply compile messages from other blogs and web pages offer a prime exhibit. Jay Cross recently noted how the blogosphere was like an echo chamber, where the most prolific bloggers cite the most prolific bloggers into recursion ad absurdium. I see this echoing effect as a potential problem, dulling and twisting the voices of truly brilliant people and fully-vetted ideas.

I don't want these compilers to stop. I like the material to sift through. But I do hope most people realize that there is very little nutrition left in sifted poop.

And we definitely need others to vet our work. For example, many great books that I read have acknowledgements aplenty for people who have provided feedback to the author(s).

My simple little point is that we need expertise too. We need filtering. We need wisdom. Some of that wisdom can come from the collective, but some must also come from people who have done the hard work of learning a knowledge discipline.

Anyway

Go read Jaron Lanier's piece, and the vetting it receives from the collective spectrum of people in The Edge's readership.

New Taxonomy for Learning Objectives

Let me propose a new taxonomy for learning objectives.

This taxonomy is needed to clear up the massive confusion we all have about the uses and benefits of learning objectives. I have tried to clarify this in the past in some of my conference presentations—but I have not been successful. When I get evaluation-sheet comments like, "Get real you idiot!" from more than a few people, I know I've missed the mark. SMILE

Because I don't give up easily—and because learning objectives are so vitally important—I'm going to give this another try. Your feedback is welcome.

The premise I'm working from is simple. Instructional professionals use learning objectives for different purposes—even for different audiences. Learning objectives are used to guide the attention of the learner toward critical learning messages. Learning objectives are used to tell the learner what's in the course. They are used by instructional designers to guide the design of the learning. They are used by evaluation designers to develop metrics and assessments.

Each use requires its own form of learning objective. Doesn't it seem silly to use the exact same wording regardless of the use or intended audience? Do we provide doctors and patients with the exact same information about a particular prescription drug? Do designers of computer software require the same set of goal statements as users of that software? Do creators of films need to have the same set of objectives as movie goers?

Until recently I have argued that we ought to delineate between objectives for learners and objectives for designers. This was a good idea in principle, but it still left people confused because it didn't cover all the uses of objectives. For example, learners can be presented with objectives to help guide their attention or to simply give them a sense of the on-the-job performance they'll be expected to perform. Instructional designers can utilize objectives to guide the design process or to develop evaluations.

The New Taxonomy

1. Focusing Objective
   A statement presented to learners before they encounter learning material—provided to help guide learner attention the most important aspects of that learning material.
2. Performance Objective
   A statement presented to learners before they encounter learning material—provided to help learners get a quick understanding of the competencies they will be expected to learn.
3. Instructional-Design Objective
   A statement developed by and for instructional designers to guide the design and development of learning and instruction.
4. Instructional-Evaluation Objective
   A statement developed by and for program evaluators (or instructional designers) to guide the evaluation of instruction.

I made a conscious decision not to include a "table-of-contents objective" despite the widespread use of this method for presenting learners with objectives. I can't decide whether this should be included. There's no direct research on this (that I've encountered), but there may be some benefit for learners in having an outline of the coming learning material. Your comments welcome. I'm leaning toward including this notion into the taxonomy because it is a stategy that I've seen in use. Maybe I'll call them "Content-Outlining Objectives" or "Outlining Objectives."

One of the clear benefits of this taxonomy is that it separates Focusing Objectives from the other objectives. These objectives—those presented to learners to help focus their attention—have been researched with the greatest vigor. And the results of that research are clear:

1. Focusing objectives guide learner attention to the information in subsequent learning material that has been targeted by objectives, but they also take attention away from the information not targeted by objectives.
2. Similarly, focusing objectives improve learning for the targeted information and hurt learning for the information not targeted.
3. Prequestions are as powerful in creating this focusing effect as learning objectives, and they may be more powerful.
4. The wording of the focusing objective or prequestion must specifically mirror the wording in the learning material. General or abstract wording doesn't cut it.
5. Adding extra words, particularly words that specify the criteria of performance (ala Mager) will actually distract learners and hurt learning.

The Case for Delaying Level 2 Assessments

Why Level 2 Assessments Given Immediately After Learning Are Generally Dangerous and Misleading

Note from Will Thalheimer: This is an updated version of a newsletter blurb written a couple of years ago, where I made too strong a point about the dangers of Level 2 Assessments. Specifically, I claimed that Level 2 Assessments should never be used immediately after learning, which may have been pushing the point too far. Upon rethinking the issue a bit, I've concluded that Level 2 Assessments immediately after learning are still dangerous, but there may be some benefits to using them if the overall assessment process is designed correctly. You can be the judge by reading below.

Introduction to Levels 1 and 2 Assessments

Before we get to Level 2 evaluations, let's talk about Level 1 of Kirkpatrick's 4-level model of assessment. Level 1 is represented by the "smile sheets" that we hand out after training or include at the end of an e-learning course. They typically ask learners to rate the course and to judge how likely they are to use the information they learned. These evaluations are valuable to get learner reactions and opinions, but they provide a very poor gauge of learning and performance. The fact that we rely on these almost exclusively to assess the value of our instruction is unconscionable.

Level 1 Assessments are not always good predictors of learning. Learners may give a course a high score but not remember what they learned. Learners are also famously optimistic about what they will remember. Just because they tell us they'll remember information and use it in their work doesn't mean they will. Learners also fill in smile sheets based on whether they like the course or the instructor. Courses that challenge learners may be rated poorly, even though a challenge might be exactly what is needed to push a significant behavior change.

Level 2 Assessments are intended to measure learning and retention. We want to know whether the information learned is retrievable from memory. Ideally, we want to know whether the information is retrieved and used on the job. If we measure actual on-the-job performance, we're really utilizing a Level 3 Assessment. In comparison, Level 2 Assessments measure the retrievability of information, not it's actual use. This is where the problems start.

What is Meant Here by the Word Assessment?

First let me clarify that I am using the word "assessment" to mean a test given for the purposes of evaluating a learning intervention. Assessments can also be used to bolster learning, as when they are used to promote retrieval practice or provide feedback to the learners. It is the first use of assessments that I am concerned with in this article. Specifically I will argue that Level 2 Assessments given for the purpose of evaluating the success or failure of a learning intervention are dangerous if given immediately after the learning. However, this does not mean that assessments used for the purpose of aiding retrieval or providing corrective feedback are not valuable at the end of learning. In fact, they are excellent for that purpose.

The analysis in this article also assumes that Level 2 Assessments are well designed. Specifically, it assumes that the assessments prompt learners to retrieve from memory the same information that they will have to retrieve in their on-the-job situations. It also assumes that the cues that trigger retrieval will be similar to those that will trigger their thinking and performance on the job. It is true that most current Level 2 Assessments don't meet these criteria, but they should.

The Problems With Immediate Assessments

When we learn a concept, we think about it. When we think about something, it becomes highly retrievable from memory, at least for a short time. Thus, during learning and immediately afterward, our newly learned information is highly retrievable. If we test learners then, they are likely to remember all kinds of stuff they'll forget in a day.

This problem is compounded because learning is contextualized. If we learn in Room A, we'll be better able to retrieve the information we learned if we have to retrieve it in Room A as opposed to Room B (by up to 55% or so). Thus, if we test learners in the training room or while they're still at their desks using an e-learning program, we're priming them for a level of success they won't attain when they're out of that learning situation and back on the job.

Giving someone a test immediately after they learn something is cheating. It provides an inflated measure of their learning. More importantly, it tells us very little---if anything---about how well learners will be able to retrieve information when they get back to their jobs.

On-the-job retrieval depends on both the amount of learning and the amount of forgetting.

Retrieval = Learning - Forgetting

Our instructional designs need to maximize learning and minimize forgetting. If we measure learners immediately after they learn, we've accounted for the learning part of the retrieval equation, but we've ignored forgetting all together. Not only are immediately-given Level 2 Assessments poor tools to use in measuring an individual's performance, but they also give us poor feedback about whether our instructional designs are any good. In short, they're double trouble. First they don't measure what we want them to measure, and then they don't hold us accountable for our work.

But What Happens On The Job?

All this is true in most cases, but there are complications when we consider what happens after the learning event. The analysis above is accurate in those situations when learners forget much of what they learn as they move from learning events back to their jobs. Look at the following graph and imagine doing a Level 2 Assessment at the end of the learning---before the forgetting begins. It would show strong results even though later performance would be poor.

But what happens in those all-too-rare situations when learners take the learning and begin to apply what they've learned as soon as they get back to the workplace? When they do this, they're much less likely to forget---and they may even take their competence and learning to a higher level than they achieved in the actual learning event. Check out the graph below as an example.

In this case, if we did a Level 2 Assessment at the end of the initial learning---before the workplace learning begins, again the assessment wouldn't be accurate. This time it might not adequately assess the ability of the learning intervention to facilitate the workplace learning.

Real learning interventions often generate both types of results. Learners utilize some of what they've learned back on the job---facilitating their memory; but the rest of what they learned is not used and so is forgotten. The following graph depicts this dichotomous effect.

So Why Use Level 2 Assessments At All?

It should be clear that Level 2 Assessments delivered immediately after the learning are virtually impossible to interpret. However, it may be useful to use them in conjunction with a later Level 2 Assessment to determine what is happening to the learning after the learners get back to the job.
If the learners' level of retrieval improves, then we can be fairly certain that the learners have made positive use of the learning event. Of course, a better way to draw this conclusion is to use a comparison-group design, but such an investment is normally not feasible.

If the learners' level of retrieval remains steady, then we can be fairly certain that the course did some good in preventing forgetting. Again, a comparison-group design will be more definitive.
If the learners' level of retrieval deteriorates, then we can be fairly certain that the learning event did not prevent forgetting and/or was probably not targeted at real work skills. Deteriorating retrieval is the one result we want to make sure we don't produce with our learning designs---and because forgetting is central to human learning---if we're preventing forgetting, we're doing something important. Finally, because forgetting is normal, a comparison-group design is not as critical in ruling out alternative explanations. In other words, if we find forgetting after the learner returns to the job, we can conclude that the learning event wasn't good enough.

To summarize this section, it appears that though Level 2 Assessments given immediately after learning have dubious merit because they're impossible to interpret, there may be value in using immediate Level 2 Assessments in combination with delayed Level 2 Assessments.

How to Do Level 2 Assessments

Level 2 Assessments should be utilized in the following manner:

1. Be administered at least a week or two after the original learning ends, or be administered twice---immediately after learning and a week or two later.

2. Be composed of authentic questions that require learners to retrieve information from memory in a way that is similar to how they'll have to retrieve it on the job. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

3. Cover a significant portion of the most important performance objectives.

The Not-So-Straight Poop

What is the median age when children are potty trained? Pick the choice that is closest to the actual figure. Go ahead and pick an answer before you read further—just for the heck of it.

1. 1 year old
2. 2 years old
3. 3 years old

Most Americans are likely to choose the second and third choices because most American children are potty trained starting when they are around two years of age.

What’s interesting is that at least 50% of the world’s babies are potty trained by the time they are 1 year old. This piece of data comes from a front-page story in the Sunday New York Times, October 9, 2005 (citing Contemporary Pediatrics magazine).

And the practice isn’t just relegated to a few cultures. Parents in more than 75 countries use these super-fast potty-training practices.

While I generally keep to adult-learning topics in this newsletter, I decided to include this topic for several reasons.

1. If the NY Times runs it on the front page, it must be fit to print (wink-wink).
2. The topic held the promise of being an attention-grabber.
3. The topic reveals lessons for our training-and-development practices.

One obvious learning point is that cultural and individual differences exist in learning. The potty-training example is just an illustration of how surprised we can become if we assume everyone does training like we do it. We cannot assume everything generalizes!

A second point is that while performance outcomes may not differ, different training methods may have different side effects. Most kids are going to be potty trained by the time they are five, but just imagine how much solid waste could be reduced if kids used less than a year of diapers versus three years of diapers. Different training methods produce many different types of side effects: cost differential, organizational morale, individual sense of efficacy, self-esteem, work-family balance, and even pollution! Do you know your training’s side effects?

Let’s not forget that training affects learner attention, even after the training is over. If we train learners on grammar skills, later when they’re in a meeting, they’re more likely to utilize some of their limited working-memory capacity thinking about grammar than if we’d given them no training on grammar. This can be a beneficial or disruptive side effect.

A third point we might glean from the potty example is that learner readiness affects the training investment needed. As the New York Times article pointed out, the parents who use the super-fast training methods must be in constant vigilance, looking for the faintest signs that their child is going to pee or poop. Some of the parents who use these methods swear by them, describing how it helps them feel closer to their children, but they all talk about what an incredible effort it takes to use the methods. As child-guru T. Berry Brazelton was quoted as saying, “I’m all for it, except I don’t think many people can do it.” It’s just going to be too difficult for most parents—especially the typical parent who has multiple responsibilities.

Similar analogs can be found in workplace learning. Learner readiness can play out in many ways. More training may be required for those who aren’t ready, more prerequisite learning opportunities, more effort, and more hand-holding. Lack of readiness may even trigger a reversal in the training calculus—perhaps some topics for some people in some situations are just too costly to train. Alternative interventions may be required or a decision to withhold training may also be appropriate. Do you know how ready your learners are?

The learner-readiness notion is also relevant to public policy. For example, investments in our public schools affect the investments businesses need to make in learning.

Summary

1. We cannot assume that we understand our learners.
2. Learning designs have different side effects.
3. Workplace attention is a side effect of training.
4. The less ready learners are, the more investment needed.
5. Learner readiness is affected by your elected officials.

Questions to Think About

1. What are the side effects of your learning interventions?
2. What level of readiness do your learners have?