New Design for My Smile Sheet

Smile sheets (the feedback forms we give learners after learning events) are an almost inevitable practice for training programs throughout the workplace learning industry. Residing at Donald Kirkpatrick's 1st level—the Reaction level—smile sheets offer some benefits and some difficulties.

On the plus side, smile sheets (a) show the learners that we respect their thoughts and concerns, (b) provide us with a customer satisfaction ratings (with the learners as customers), (c) hint at potential bright spots and trouble spots, and (d) enable us to make changes to improve later programs.

On the minus side, smile sheets (a) do not seem to correlate with learning or behavior (see meta-analysis by Alliger, Tannenbaum, Bennett, Traver, and Shotland, 1997, that showed very weak correlations), (b) are often biased by being provided to learners in the learning context immediately after learning, and (c) are often analyzed in a manner that over-values the data as more meaningful than it really is.

Based on these benefits and difficulties, I recently developed a new smile sheet (one that shows its teeth, so to speak. SMILE) for my workshop Measuring and Creating Learning Transfer.

It has several advantages over traditional smile sheets:

1. Instead of asking learners to respond globally (which they are not very good at), it asks learners to respond to specific learning points covered in the learning intervention. This not only enables the learners to better calibrate their responses, it also gives the learners a spaced repetition (improving later memory retrieval on key learning points).

2. The new smile sheet enables me to capture data about the value of the individual key concepts so that changes can be made in future learning interventions.

3. The smile sheet has only a few overall ratings (when lots of separate ratings are used in traditional smile sheets, most of the time we don't even analyze or use the data that is collected). There is space for comments on specifics, which obviates the need for specific ratings, and really gets better data as well. The average value is highlighted, which helps the learners compare the current learning intervention to previous learning interventions they have experienced. (You should be able to click on the image to see a bigger version).

4. The smile sheet asks two critical questions related to how likely the information learned will be utilized on the job and how likely the information will be shared with others. In some sense, this is where the rubber hits the road because it asks whether the training is likely to have an impact where it was intended to have an impact.

5. The smile sheet shows some personal touches that encourage the learners that the learning facilitator (trainer, professor, etc., or me in this case) will take the information seriously.

6. Finally, the smile sheet is just a starting point for getting feedback from learners. They are also sent a follow-up survey 2 weeks later, asking them to respond to a few short questions. Here are a few of those questions. Again, you might need to click the image to see a bigger version.

The learners get the following question only if they answer a previous question suggesting that they had not yet shared what they learned with others.

Why I Like My New Smile Sheet

I'm not going to pretend that I've created the perfect assessment for my one-day workshop. As I've said many times before, I don't believe in "perfect assessments." There are simply too many tradeoffs between precision and workability. Also, my new smile sheet and the follow-up survey are really only an improvement on the traditional smile sheet. So much more can be done, as I will detail below.

I like my new evaluation sheet and follow-up survey because they give me actionable information.

• If my learners tell me that a concept provides little value, I can look for ways to make it valuable and relevant to them, or I can discard it.
• If my learners find a concept particularly new and valuable, I can reinforce that concept and encourage implementation, or I can highlight this concept in other work that I do (providing value to others).
• If my learners rate my workshop high at the end of the day, but low after two weeks, I can figure out why and attempt to overcome the obstacles.
• If my learners think they are likely to implement what they learned (or teach others) at the end of the day, but don't follow-through after two weeks, I can provide more reminders, encourage more management support, provide more practice to boost long-term retrieval, or provide a follow-up learning experience (maybe a working-learning experience).

I also like the evaluation practice because it supports learning and performance.

• It provides a spaced repetition of the key learning concepts at the end of the learning event.
• It provides a further spaced repetition of the key learning concepts at the beginning of the two-week survey.
• It reminds learners at the end of the learning intervention that they are expected to put their learning into practice and share what they've learned with others.
• It reminds them after 2 weeks back on the job that they are expected to put their learning into practice and share what they've learned with others.
• It provides them with follow-up support 2 weeks out if they feel they need it.

Limitations

• My new smile sheet and follow-up survey don't tell me much about how people are actually using what I've taught them in their work. They could be implementing things perfectly or completely screwing things up. They might perfectly understand the learning points I was making or they may utterly misunderstand them.
• The workshop is an open-enrollment workshop, so I don't really have access to people on the job. When I run the workshop at a client's site (as opposed to an open-enrollment format), there can be opportunities to actually put things into practice, give feedback, and provide additional information and support. This, by the way, not only improves my learners' remembering and performance (and my clients' benefits), it gives me even richer evaluation information than any smile sheet or survey could. 
• While the smile sheet and follow-up survey include the key learning points, they don't assess retrieval of those learning points or even understanding.
• Not everyone will complete the follow-up survey.
• The design I mentioned not only doesn't track learning, understanding, or retrieval; it also doesn't compare results to anything except learners' subjective expectations. If I was going to measure learning or performance or even organiziational results, I would consider control groups, pretests, etc.
• There is no benchmarking data with other similar learning programs. I don't know whether my learners are doing better than if they read a book, took a workshop with Ruth Clark, or went and got their masters degree in learning design from Boise State.
• Bottom line is that my smile sheet and follow-up survey are an improvement over most traditional smile sheets, but they certainly aren't a complete solution.

Learning Measurement is a Critical Leverage Point

Learning Measurement provides us with a critical leverage point in the work that we do. If we don't do good measurement, we're not getting good feedback. If we don't get good feedback, we're not able to improve what we're doing.

My workshop smile sheet and follow-up survey attempt to balance workability and information-gathering. If you find value in this approach, great, and feel free to use the links below to download my smile sheet so you can use it as a template for your evaluation needs. If you have suggestions for improvement, send me an email or leave a comment.

Download the Smile Sheet

• As a PDF (so you can see how it should look). Download SmileSheetJune2008.pdf
• As a Word 2007 document. Download SmileSheetJune2008.docx
• As a Word 2003 document. Download smilesheetjune2008word2003.doc

References

Alliger, G. M., Tannenbaum, S. I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-358.

Comments

[Mark Sheppard]

Will: I like what you've done with this sheet, and it's an improvement on some of the usual fare. I really like Bennett's 7-level framework (shown in Rothwell & Cookson's book) for more detailed evaluation work.

[Mark Frank]

Nice - and very attractively presented. It makes me realise the potential for integrating level 1 assessment into a course/event instead of just having students fill it in as an afterthought when they are keen to get away and avoid the traffic. You could use the level 1 assessment as the basis for a wrap-up and review - maybe you do?

[Savita]

Great tool, thank you. I have been thinking just along these lines and it is exciting to see it put together so well.

[Marcus Barber]

Gidday Will

Much improved and I've been heading down a similar path though not as succinctly as you've captured here.

One possible area for improvement, especially given you intend to follow up. I ask the following question 'What are two areas you believe you are likely to use this information in the next month?'

When I run the follow up sessions I can then ask participants 'You said you'd probably user this in x & y - how did you find it?'

You might be able to structure yours in such a way that you automatically populate the responses from the initial feed back to the future follow up, so that you can track application of the learning to the real work environment

Marcus :-)

[Peter Williams]

This is great. I'm dealing with the problem now of improving the "smiley sheets" I use in online and face-to-face training. We all knew they were inadequate but there was a sense of resignation. Now, there's hope. :) Do you mind if I point other trainers in my workshops to your discussion here?

[Paul Left]

Great that you are asking for feedback on specific sections on content. Another way to approach this is to ask for feedback in relation to specific learning outcomes. That way it links to the purpose of the learning (and its assessment) as well the content.

[Gaye Mara]

Hi, Will,

Thanks for sharing this. It reminded me of your great best learning practices presentation to ISPI Potomac Chapter a few years ago.

A lot of useful ideas for us here! One nitpicky suggestion: Use the same rating scheme (lowest # is worst, highest is best) throughout. When one comes to the "Is there value?" questions, that scheme reverses, with "1" as the best rating and "5" as the worst. That created some cognitive dissonance for me, and I suspect it would for others as well.

Thanks for your good work and your generosity.

[Lisa Allen]

Thank you! I'm a librarian who works with executives in Human Resource Development. The sessions I teach feel more like training, and I was in desperate need of some ideas for capturing feedback. I really appreciate your sharing your documents!

[joanna howard]

Very useful development of the smile sheet, Will. You demonstrate here how an effective evaluation approach can aid learning for the learner as well as capture it for the trainer

[K Stoneman]

Dr. Thalheimer -

A real improvement over the usual smile sheets. I've been toying with a similar idea of a "learning 360" for my particpants to evaluate the usefulness of my trianing focusing on applying the material and what percentage of the time they are using it, etc.

I look forward to your continued thoughts on evaluation and needs assessment.

[Tom Martin]

Another fine piece of work. I really enjoy what you share with us - thank you.

[Amit Jain]

Very well drafted and actually halp the participants to put right feedback and inputs for trainers group.

[Generic Viagra]

Please one more post about that..
I wonder how you got so good. This is really a fascinating blog, lots of stuff that I can get into. One thing I just want to say is that your Blog is so perfect!

[Mobile Computing]

nice job i like it

[ガバペン]

Great tips, I would like to join your blog anyway

[Rhinocort Aqua]

Hola,Ha hecho un trabajo muy bueno. Hay muchas personas en busca de eso ahora van a encontrar suficientes fuentes por tus consejos.espera para obtener más consejos acerca de que

[careprost]

Excellently written article, if only all bloggers offered the same level of content as you, the internet would be a much better place. Please keep it up!

[Seo Services India]

Wow, Great postNice work, I would like to read your blog every day Thanks

[Generic Viagra]

Grande informazione, hai un blog meraviglioso e un ottimo articolo!

[Generic Viagra]

Vänligen en mer inlägg om det .. jag behöver den här artikeln för att slutföra mitt uppdrag i kollegiet, och den har samma tema med din artikel. Tack, bra aktie.!!

[Testking 350-001]

It’s hard to find knowledgeable people on this topic, but you sound like you know what you’re talking about! Thanks

[retoque foto]

Thanks for sharing the idea there would be some apprehensions from segment but i am up for it.
retoque foto

[Generic Viagra]

Es bueno estar visitando su blog de nuevo, se ha pasado meses para mí. Bueno este artículo que he estado esperando durante tanto tiempo..

[Megan, Three Wheel Bike]

I'm dismayed by what seems a primary, possibly sole, focus on level one types of evaluation. That is - learner reaction to a particular learning opportunity. Why do we place so much importance on the subjective feedback of someone who might not even know why his boss signed him up for the class (for example)?

[クレジットカード現金化]

Thank you

[Generic Drugs Exporter]

I have wanted to learn more about particular topics, but not many websites would help me out in informing me the way I expected. This left me with many question, but after reading your article, I got an answer to all my questions. You are too cool dude!!!

[creditcard cash]

It was very interesting to read.
serve as a reference

[kamagra oral jelly]

I '
ben blij voor deze nuttige uitstekende pagina, dit is een vorm van onderwerp dat mij in stand, hoewel uit heb het day.We 'normaal hoort de laatste tijd willen de buurt van in je web-site rechts onmiddellijk afte

[Buy Lioresal Online]

Je viens de découvrir votre site et je voulais vous dire que j'ai vraiment apprécié la navigation de votre posts.In tout cas je vais vous abonnant à votre blog et j'espère que vous publiez à nouveau prochainement

[Spiriva]

Dit is echt interessant, Je bent een zeer ervaren blogger. Ik ben bij uw feed en kijk uit naar op zoek naar meer van uw prachtige post. Ook heb ik deelde in je site in mijn sociale netwerken!

[Allegra]

Dit is echt interessant, Je bent een zeer ervaren blogger. Ik ben bij uw feed en kijk uit naar op zoek naar meer van uw prachtige post. Ook heb ik deelde in je site in mijn sociale netwerken!

[creditcard genkinka]

Excellent site, keep up the good work my colleagues would love this.

[Software Application Testing]

I am happy to find so many useful information here in the post, thanks for sharing it here. I hope you will adding more...
Good article! Thank you so much for sharing this post.

[Mac Cosmetics]

Colour plus texture for the lips. Stands out on the runway. Simmers on the street! What made M·A·C famous.

[essay help]

I came to here today 2nd in a week and enjoyed again to read it's article.

[oil press]

Lot of great information which can be helpful in some or the other way. Keep updating the blog,looking forward for more contents...Great job, keep it up..

[Programming Assignment Help]

Blogs are good in which we get lots of information and convert that information to knowledge.

[Analysis essay]

A comprehensively detailed and attention grabbing review that you wrote in this article. I am really convinced the way you look. The way you describe the whole thing is simple and understandable

[Buy Assignment]

Nice, accurate and to the point. Not everyone can provide information with proper flow. Good post. I am going to save the URL and will definitely visit again. Keep it up.

[iPhone Developer]

I agree with some of your readers and will eagerly look forward to your coming updates.

[logo design contests]

Really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people.

[Facebook Developers]

I totally agree with your article because you mention the some vital point regarding topic.I admire your post and its points

[Buy sildenafil]

A very nice article you have really done a great job by posting such a wonderfull topic i througly enjoyed reading the topic

[Facebook App]

There are many smile sheet you find on internet.Some of them are too ugly and some very good for users.