Will at Work Learning

Songs for Learning?

Will Thalheimer — 2006-11-08T09:45:29-05:00

At a recent Bank of America employee event, two employees sang a song to motivate employees to see the benefits in the Bank of America and MBNA merger.

You decide if this is a good learning opportunity. Click to see the video.

Warning: May cause laughter. Don't forget to read the comments.

Actually, I'm guessing that the song worked well for people within the meeting, but damn does it fall apart when the rest of us get to peek through the windows. Suggests to me that motivational attempts like these in our e-learning programs and our business meetings must be lock-boxed to prevent a wider distribution. Even better, when we design we need to assume that our efforts might get posted on YouTube.

Response Cards to Facilitate Active Learning in Lectures

Will Thalheimer — 2006-11-07T08:00:00-05:00

Lectures are widely reviled for putting learners in a passive mode. On the other hand, lectures are relatively easy to implement, even with large numbers of learners. And regardless of the pluses and minuses, lectures are ubiquitous. While there aren't many lectures in kindergarten, by third grade teachers are talking a lot and learners are listening. The college classroom is dominated by lecture. So is the corporate training session, conference presentations, church sermons, public meetings, elder hostels, and the local library's evening speaker series. Lectures aren't going away anytime soon, nor should they. Like all tools for learning, they provide certain unique advantages and have certain unique limitations.

Lectures can be modified in different ways to increase the amount of active learning---to ensure that learners are more fully engaged, have a more robust understanding of the learning material, are more likely to remember what they learned, are more likely to utilize the information at a later time.

One such method to increase active learning are "response cards." Response cards are provided to students so that each one can respond to instructor questions. Two types of response cards are available, (1) those that enable each learner to write his or her answer on the card (for example with a dry-erase marker), and (2) those that enable learners to hold up preprinted answers (either True or False; or A, B, C, or D for example).

Research

While not a lot of good research has been done on response cards, the research seems to suggest that compared with the traditional method of having students raise their hands in response to questions, response cards improve learners' classroom engagement, the amount they learn, and the amount they retain after a delay (Marmolejo, Wilder, & Bradley, 2004; Gardner, Heward, Grossi, 1994; Kellum, Carr, and Dozier, 2001; Narayan, Heward, Gardner, Courson, Omness, 1990; Christle & Schuster, 2003). Learners generally prefer response cards to simple hand-raising. Most of the research has focused on K-12 classrooms, with some research done in community college. The research has tended to focus on relatively low-level information and has not tested the value of response cards on higher-order thinking skills.

Recommendations

Getting learners to actively respond in lectures is certainly a worthwhile goal. Research has been fairly conclusive that learners learn better when they are actively engaged in learning (Bransford, Brown, & Cocking, 1999). Response cards may be one tool in the arsenal of methods to generate learner engagement. Of course, electronic keypads can be used in a similar way, at a significantly increased cost, with perhaps some added benefits as well. Still, at less than $30 a classroom, response cards may be worth a try.

Personally, I'm skeptical that audiences in adult training situations would be open to response cards. While 87% of college students rated the cards highly (Marmolejo, Wilder, & Bradley, 2004), the corporate audiences I've worked with over the years, might find them childish or unnecessary ("hey, why can't we just raise our hands?"). On the other hand, electronic keypads are more likely to be accepted. Of course, such acceptance---whether we're talking about response cards or electronic keypads---really depends on the relevance of the material and the questions used. If the questions are low-level rote memorization, adult audiences are likely to reject the instruction regardless of the technology employed.

Making lectures interactive has to be done with care. Adding questions and student responses can have negative consequences as well. When we ask questions, we signal to learners what to pay attention to. If we push our learners to think about low-level trivia, they will do that to the detriment of focusing on more important high-level concepts.

Limitations of the Research

The research on response cards tends to focus on low-level questions that are delivered all-to-frequently throughout lectures. Learners who have to answer a question every two minutes are being conditioned to focus on trivia, facts, and knowledge. Future research on response cards should focus on higher-level material in situations where more peer discussion are enabled.

Most of the research on response cards suffered from minor methodological difficulties (e.g., weaker than preferred comparison designs and a low level of learners actually tracked) and ambiguity (e.g., in reading the research articles, it was often difficult to tell whether the in-class questions were repeated on the final quizzes---those used as dependent variables; and no inferential statistics were available to test hypotheses).

References

Marmolejo, E. K., Wilder, D. A., & Bradley, L. (2004). A preliminary analysis of the effects of response cards on student performance and participation in an upper division university course. Journal of Applied Behavior Analysis, 37, 405-410.

Cristle, C. A., & Schuster, J. W. (2003). The effects of using response cards on student participation, academic achievement, and on-task behavior during whole-class, math instruction. Journal of Behavioral Education, 12(3), 147-165.

Gardner, R., Heward, W. L., & Grossi, T. A. (1994). Effects of response cards on student participation and academic achievement: A systematic replication with inner-city students during whole-class science instruction. Journal of Applied Behavior Analysis, 27, 63-71.

Kellum, K. K., Carr, J. E., & Dozier, C. L. (2001). Response-card instruction and student learning in a college classroom. Teaching of Psychology, 28(2), 101-104.

Narayan, J. S., Heward, W. L., Gardner, R., Courson, F. H., & Omness, C. K. (1990). Using response cards to increase student participation in an elementary classroom. Journal of Applied Behavior Analysis, 23, 483-490.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

Nutritious Learning Design

Will Thalheimer — 2006-11-06T12:29:17-05:00

For decades, the US Food and Drug Administration (FDA) has been touting schemes to attempt---at least ostensibly---to help Americans eat more healthfully. From the 4-food-group concept to the current food pyramid, the idea has been to help us think more intelligently about what we eat. The development of these formulations have always been under influence by industry groups representing food companies and health advocates, and it has never been clear that the resulting compromises have been effective. Note the current obesity epidemic as evidence.

But now, the Hannaford Brothers supermarket chain is taking matters into their own hands. Hannaford's nutritionists have developed a simple coding system to let shoppers know the relative health value of foods. Three stars is most healthy. Zero stars is least healthy. And while the coding system is simple, the underlying algorithm---the algorithm that assigns the ratings---is complex.

Listen to this: 77% of all the items on Hannaford's shelves rated zero stars. Even most of the Hannaford Store brand items get zero stars. Our choices are not very nutritious!!

As you might imagine, some manufacturers are not happy.

Only time will tell whether shoppers will change their eating habits---or whether food manufacturers will change the formulas for their processed foods to get a better star rating, and Hannaford will be able to track this data very easily. It's hard to tell whether the food pyramid has had an effect. Hannaford's Guiding Star system will be much easier to assess. I hope they've got some control-group stores to make comparisons.

I love the simplicity of the system. It's like a job aid on steroids. It's simple. It's provided exactly when needed (as shoppers make their selections), and it's based on proven nutritional recommendations.

Those of us in the learning-and-performance field can learn a lot from this design.

Ensure that learning designs impact performance situations.
Simplify, to increase how much your learners/performers actually use your stuff.
Base learning designs on proven content. For example, is the stuff you teach in your leadership classes really been proven to improve management performance?
Utilize systems to track your success, so that you can make

To read the excellent NY Times article, click here.

NTL continues its delusions

Will Thalheimer — 2006-11-03T10:30:23-05:00

It's time to publicly vilify NTL Institute for Applied Behavioral Science for propagating the myth that learners remember 10% of what they read, 20% or what they see visually, etc. They continue to claim that they did this research and that it is accurate.

The research is NOT accurate, nor could it be. Even a casual observer can see that research results that end neatly in 5's or 0's (as in 5%, 10%, 20%, 30%) are extremely unlikely. To see a complete debunking of this hoax, click here.

Normally, I choose not to name names when it comes to the myths in our field. We all make mistakes, right? But NTL continues to harm our field by propagating this myth. Here is the document (Download pyramid_2.DOC) they send to people who inquire about the percentages. At least five separate people have sent me this document after contacting NTL on their own initiative.

I have talked to NTL staff people and emailed them (over a year ago), and even with my charming personality, I have failed to persuade them of the problems they are causing.

The people who write me about this are outraged (and frankly confused) that an organization would propagate such an obvious falsehood. Are you?

Here are claims that NTL makes in its letter that are false:

NTL: We know that in 1954 a similar pyramid with slightly different numbers appeared on p. 43 of a book called Audio-Visual Methods in Teaching, published by the Edgar Dale Dryden Press in New York.

Why false? There are NO numbers on page 43 of Edgar Dale's book.

NTL: We are happy to respond to your inquiry about The Learning Pyramid. Yes, it was developed and used by NTL Institute at our Bethel, Maine campus in the early sixties when we were still part of the National Education Association's Adult Education Division.

Very Intriguing: How could NTL have developed the pyramid in the 1960's, when a similar version was published by Edgar Dale in 1954? Professor Michael Molenda of Indiana University has found some evidence that the numbers first appeared in the 1940's. Maybe NTL has a time machine.

NTL: Yet the Learning Pyramid as such seems to have been modified and always has been attributed to NTL Institute.

No. It wasn't attributed to NTL by Dale. Dale thought it was his. And again, Dale did not use any numbers. Just a cone.

Okay, so now half of you hate NTL, and the other half of you hate me for being the "know-it-all kid" from 7th grade. Well, I'll take the heat for that. But still, is this the kind of field you want to work in?

And what is the advantage for NTL to continue the big lie?

Here's what NTL should write when people inquire:

Thanks for your inquiry to the NTL Institute. Yes, we once utilized the "Learning Pyramid" concept in our work, starting in the 1960's. However, we can no longer locate the source of the original information and recent research tends to debunk those earlier recommendations. We apologize for any harm or confusion we may have caused.

eLearning Guild Oct 2006 Conference Review

Will Thalheimer — 2006-11-01T15:23:31-05:00

I recently attended the eLearning Guild's DevLearn06 conference, dedicated to e-learning developers. I gave a preconference workshop on the learning research and two conference sessions, one on the spacing effect and one on assessments (especially as they are illuminated by the learning research).

Here's some quick highlights from the conference:

1. 50 developers showed off their latest and greatest e-learning interventions in the eLearning Guild's Demo Fest. This 2.5 hour opportunity is a great addition to the typical conference fare. I noticed two things:

The production values and aesthetic quality of the entries were remarkable. Beautiful work. So, if you're developing e-learning, you better start hiring talented graphic designers and managers who have an aesthetic sensibility.
The designs were generally much better than page turners, but they still weren't completely consistent with research-based practices. There were good interactions and lots of decision scenarios, but still there were too many missed opportunities for authentic practice and decision-making.

2. The eLearning Guild's Research Posse, led by the ingenious and tireless Steve Wexler, has developed an amazing data-gathering tool, that enables eLearning Guild members to find out the market penetration of all kinds of things, including authoring tools, simulation practices, m-Learning, etc., etc. With this foray into industry research, the Guild easily leapfrog's the feeble attempts by ASTD and Bersin, both of which provide only broad strokes analysis of industry data.

3. Great session by Judy Brown on m-Learning, who showed (a) an amazing installed base of m-Learning tools that makes it clear that m-learning tsunami is on its way here, and (b) some interesting statistics showing that the US is way behind Asia and Western Europe in the amount of data that is utilized on cell phones (the more data, the more learning programs).

Google for Educators

Will Thalheimer — 2006-10-13T17:13:12-04:00

Google announced yesterday a special set of tools for educators (teachers).

Click here to check it out.

Precisely False Industry Data

Will Thalheimer — 2006-09-28T10:55:00-04:00

Recently, the New York Times Public Editor wrote an article on polling. The article skewered many current practices. It also was educational for any of us who want to truly understand the polling data we may hear in the news.

Sadly, the article made me think of the pathetic data that runs around our field---the training, learning, development, e-learning field. I've already mentioned some of the data allegedly posing as learning research.

But there is another wide swath of data that we should be very skeptical about---the data some "research" firms and trade organizations are peddling as industry data. The data is typically gathered by sending out surveys to an unrepresentative sample of companies, by having only a fraction of respondants complete the data, by gathering opinions, and by boisterously proclaiming that the data tell us what the industry best-practices are. Here are some of the problems with this farce:

Biased sampling of organizations.
No control for the biasing effects of non-respondants.
Assuming that opinion is fact.
Relying on the averaging of opinions.
Assuming that the average respondants have the best insights.
The arrogance and lack of caveats in the reporting.
Year-to-Year comparisons with different companies in each year's sample.
Additional biasing due to fraud and corruption, as when these "research" organizations tilt the best-practice results to their paid customers.