Measuring Learning 8 – Book Review: Shrock & Coscarelli’s Book, Criterion-Referenced Test Development

Sharon Shrock and Bill Coscarelli recently completed the third edition of their important book, Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training. If this book isn't in your collection already, I'll give you a link below to buy from Amazon.com.

In this third edition, Sharon and Bill have updated the book from the second edition (published in 2000) in some critical ways. One of those ways is truly transformational for the workplace learning and performance field. I'll get to that in a minute.

Also updated is the excellent chapter at the end of the book by Patricia S. Eyres, a lawyer with employment-law credentials. Her chapter covers the legal ramifications and guidelines in dealing with employee testing, especially as that testing affects employee selection, advancement, and retention. She has updated her chapter with new case law and legal precedent from that in the second edition. Most people in the training field have very little knowledge of the legal ramifications of testing, and I'd recommend the book for this chapter alone—it's a great wakeup call that will spur a new appreciation of the legal aspects of testing.

In the second edition, Shrock and Coscarelli put forward what they call the "Certification Suite." In criterion-referenced testing, the goal is to decide whether a test taker has met a criterion or not. When they have met the criterion, they are said to be "certified" as competent in the area on which they were tested. The Certification Suite has six levels, some which offer full Certification and some which offer Quasi-Certification:

Certification

1. Real World
2. High-Fidelity Simulation
3. Scenarios

Quasi-Certification

1. Memorization
2. Attendance
3. Affiliation

As the authors say in the book (p. 111), "Level C represents the last level of certification that can be considered to assess an ability to perform on the job."

The truly transformational thing offered by Shrock and Coscarelli is that Level D Memorization, in the second edition of the book, was considered to offer Certification. NO MORE!! That's right. Two of our leading thinkers on testing say that memorization questions are no longer good enough!!!!!!!!!!!!!! Disclosure: In speaking with Bill Coscarelli in 2006, I gently encouraged this change. This is mentioned in the book, so it's not like I'm bragging. SMILE.

I love this, of course, because it follows what we know about human learning. For tests to be predictive of real-world performance, they have to offer similar cues to those that learners will face in the real world. If they offer different cues—like almost all memorization questions do—they are just not relevant. And, from a learning standpoint (as opposed to a certification standpoint) memorization questions won't spur spontaneous remembering through the triggering mechanism of the real-world cues.

This literal and figurative raising of the bar—to move it beyond memorization—should shake us to our core (especially since this is one of the few books on assessment that covers legal stuff—so it may have some evidentiary heft in court). If the compliance tests at the end of your e-learning programs are based on memorization questions, you are so in trouble. If your credentialing is based on completion (and 85% of our respondents in the eLearning Guild research report said they utilized completion as a learning measure), you are in even worse trouble. And, of course, if you ever thought your memorization-level questions supported learning, well, sorry. They don't! At least not as strongly as they might.

Have you bought the book yet? You should. You ought to at least have it around to show management (or your clients) why it's important (absolutely freakin' critical) to use high-value assessment items.

I've got some quibbles with the book as well. They list 6 reasons for testing. I've recently come up with 18, so it appears they're missing some, or I'm drinking too much. I also don't like the use of Bloom's Taxonomy to index some of the recommendations. In short, Bloom's has issues. I don't like the way they talk about learning objectives. They use the methodology of relying on a single objective to guide the process of both instructional design and evaluation. I am now advocating to free instructional-design objectives from the crazy constraint of being super-glued to the evaluation objectives. They need to be linked of course, but not hog-tied. I wish they emphasized more strongly the distinction between testing to assess and testing to support learning. They are different animals and most of us are confused about this.

Overall, it's a great and thoughtful book. I bought it. You should too.

Here's a link that will let you click here to buy.

The Learning Measurement Series will continue in January...

(But watch to see who wins this year's Neon Elephant Award, which I'll announce on Saturday (December 22nd 2007). The winner(s) is/are all about learning measurement.)

Comments

[Dick Carlson]

When I used to do lots of apps training years ago, I would tell people we had two requirements for registration. #1 -- You were breathing. #2 -- You could sign a check. And if they could execute #2 we were willing to let #1 slide.

It amazes me when I see people talking about "Certification" for 1/2 day workshops and such. (ASTD is even doing this, for Bloon's sake!) Would you want a welder or jet pilot to be "certified" for sitting in a room for a few hours, with no measure of actual skill?

But when certification is being done for income at any organization, the temptation is overwhelming to keep lowering the bar. When I worked for Microsoft developing training for large events, our main metric was "customer satisfaction". Guess how much actual assessment was done there. (For extra credit, guess how difficult the materials were to complete.)

To get my assessment of what you learned from this post, print your answers on the back of a $20 bill and send them to me.

[thomas sabo]

I think so!!!