How to Write Better Smile Sheets: What’s Wrong With Traditional Training Evaluation Forms and How to Make Them Better

If you're in training, you're probably familiar with the sheets that trainers pass out to learners after a training session, asking the learners to evaluate the training session and the trainer.

These are known by a variety of names. Maybe you call them training-evaluation forms, or student-response forms, or trainee-reaction forms. But they're also commonly--maybe most commonly--known as smile sheets.

Why smile sheets? Because it's common for the learners attending training to give the training/trainer high scores that make everyone smile. But the common assumption is that the trainees do that politely, kindly, quickly, uncritically, and without giving any great thought. And so the term smile sheet is generally used somewhat dismissively, with the assumption that the information they contain doesn't really provide a lot of value. Or, they're assumed to hold much valuable information, even if that's not really true due to poor design.

And yet, quite a few trainers continue to use smile sheets, and many of those trainers do nothing to improve them. Maybe they've never even thought of improving their smile sheets. It's all become a bit of a habit to them, one they don't think about because there's so much else to think about, worry about, and to do.

I recently read a very good book called Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Dr. Will Thalheimer. The book explains some of the common problems with smile sheets, but also gives some very helpful tips to help make them better. We definitely suggest that you buy and read the book, and we've included a bunch of information to help you do that at the bottom of this article.

But for now, let's look at some of the general points Thalheimer makes in his book and see what we can learn from them.

If you're extra curious on this topic, I recommend you check out our interview with Dr. Thalheimer on smile sheets after you read this introduction.

THE Book on Smile Sheets

Here's a picture of Dr. Will Thalheimer's book on smile sheets. We're going to explain to you in this article some of the evidence, findings, tips, and suggestions from the book. But we recommend you go out and buy a copy and read it cover to cover. We've got links immediately below to help you do that.

Here's some additional information about Dr. Thalheimer and his book:

• SmileSheets.com: This is where you can buy the book. As Thalheimer explains, if you buy it here instead of elsewhere, he gets a bigger slice of the cut and can therefore afford to spend more time researching stuff that helps us all.
• Will At Work Learning: Thalheimer's blog. Always interesting.
• Podcast with Will Thalheimer and Connie Malamed: Listen in to a 24-minute podcast during which Thalheimer discusses his book and the topic of smile sheets with Connie Malamed (one of my favorite instructional design bloggers, so this is a great two-fer-one).

OK, let's get back to a quick overview of some of the book's lessons. We'll provide the links above at the bottom of this article too so you can click to buy the book from either end. 🙂

Some Basics on Smile Sheets

Let's take a look at a few basic issues about smile sheets and their use, including:

• Who uses them and who doesn't
• What's the point?
• Some thoughts about their place within Kirkpatrick's four levels of evaluation

Who Uses Smile Sheets?

In my own unscientific poll, you can break trainers down into five groups (mea culpa: that's bad that I'm using an "unscientific poll," because we should be using data-driven, evidence-based methods, but the exact accuracy of this breakdown isn't a critical issue for our purposes today, so we'll go with it):

• Trainers who have never heard of nor used smile sheets
• Trainers who have heard of smile sheets, but don't use them primarily out of laziness
• Trainers who have heard of smile sheets, but don't use them because they think they bring no value
• Trainers who use smile sheets, but use traditional smile sheets that bring little value
• Trainers who use smile sheets, and who have somehow put shoulder to rock and, largely on their own, have improved the smile sheets they use so they deliver real value

We think this article will be of interest to those different groups for different reasons. Here's our thinking:

What's the Point of Using Smile Sheets?

So you may be thinking what's the whole point of using smile sheets anyway? Seems like a good place to start the conversation.

There are plenty of good reasons to write smile sheets. Thalheimer gives a list of nine reasons (which he explains that he himself modified from an earlier list created by leaning measurement expert Rob Brinkerhoff). Two that are kind of clever, that raise legitimate points, and that you might not think about are:

"Upholding the spirit of common courtesy by giving learners a chance for feedback" (1)

...and...

"Enabling learner frustrations to be vented-to limit damage from negative back-channel communications." (2)

But the real reason for using smile sheets, to collapse a few of Thalheimer's points into a single point, is to identify weaknesses in current training and use that information to make future training better.

Here, I want to quote Thailheimer at length, because he writes about this passionately and eloquently in his Preface:

"I took this path [studying learning & development] because I believed strongly--and still believe--that learning is a noble cause. It is learning that has enabled human civilization and growth. It is learning that enables individuals to excel and thrive. It is learning that holds the promise of the future.

If learning is so important and our task is such a noble one, don't we, as learning professionals, have an almost sacred responsibility to do our jobs well?

The way I see it, there are two main lynchpins to our performance. First, scientific research must guide our starting assumptions. Second, we must use good learning measurement to get valid feedback so that we can refine our understandings, improve our learning design, and live up to our promise-so that we can maximize the benefit of learning...

While smile sheets should never be the only way we get feedback on learning, by improving them, we can get significantly better information about how we're doing. With better information, we can create virtuous cycles of continuous improvement. We can build more effective learning interventions and meet our obligations as learning professionals."(3)

Kudos! Well said, right?

If we never try to evaluate the effectiveness of our training, we can't use that data to see if our training is working well or not. Which means if our training isn't working, we may not know that, and we won't have helpful information to make it better. And it also means we can't apply the same lessons to other training we may create in the future. So not evaluating the effectiveness of training is a loss/loss.

Consider that phrase Thalheimer uses--"virtuous cycles of continuous improvement." Maybe it's just the third cup of coffee in me talking, but I love that as a professional goal in any discipline. You'd think our friends in lean manufacturing, with their emphasis on kaizen, would be big fans of Thalheimer's efforts.

Smile Sheets and Kirkpatrick's Four Levels of Evaluation

If you're familiar with Kirkpatrick's Four Levels of Training Evaluation, you may know that smile sheets are level one.

If you're not familiar, and you want the extended introduction, check out that link above, or this one for the "New Kirkpatrick Model."

If you are familiar, or just want a quick review, check out the bullets and image below.

Kirkpatrick's Four Levels of Training Evaluation

• Level 1-Reaction: Smile sheets--surveys handed out to the learners, online evaluations, etc.
• Level 2-Learning: Role-playing scenarios during training, focus groups, case studies, tests, etc.
• Level 3-Behavior: Observation of on-the-job behaviors and other evaluation metrics.
• Level 4-Results: Direct measurement of business goal–cost, revenue, production, etc.

The interesting distinction here is whether or not a positive level 1 (smile sheet) evaluation is necessary for effective learning and behavioral change, or even if there's a correlation between having the learner say they "liked" the training or say that the training was "effective" and the reality of whether or not the training prepared them to perform well on a test and/or if the training will lead to desired on-the-job behaviors and movement toward desired business goals.

We're going to discuss that in more detail in the very next section, but before you move on, give this question some thought yourself. Is it more likely that employees learned during training if they said they "liked" the training or that the training was "good?" And what if employees rated one training activity (or one trainer) lower than another---are those employees less likely to learn, to perform desired behaviors on the job, or to contribute toward the business reaching business goals?

What do you think?

Side note: We've presented Thalheimer's work on smile sheets within the context of the well-known Kirkpatrick Training Evaluation model, but that's not necessary. Not only that, Dr. Thalheimer has created his own Learning Transfer Evaluation Model (LTEM) and we plan on interviewing him about that in the near future (we hope).

What's Wrong With Traditional Smile Sheets?

There are quite a few problems with "traditional" smile sheets. Thalheimer goes over them in detail in his book(4), and we'll briefly cover a few of the key points below.

Probably the big bomb-shell Thalheimer delivers is that, according to two meta-analyses of studies on smile sheets (level 1 evaluations) and their correlation to test results (level 2 evaluations), there's ALMOST NO CORRELATION with learning results.

Again, a little quote from the book:

"They found that smile sheets were basically uncorrelated with learning results! To be specific, they were minutely correlated with learning-test results at a correlation of 0.09. You will remember from your statistics that correlations go from -1 to 1. Correlations between -0.30 and 0.30 are considered weak correlations. Having a correlation of 0.09 is practically no correlation at all. It would be like correlating the household level of peanut butter use with the household level of television use." (5)

So that's a biggie right there.

Some of the other problems that Thalheimer notes with traditional smile sheets are:

• We're asking the wrong kind of questions
• We're giving our employees the wrong kind of answer options
• We're getting feedback that doesn't tell us how effective the training was
• We're getting feedback that isn't actionable
• We're translating the answer options we get into a numerical score

We'll look at each of these issues in a little more detail in the section below.

Improving Traditional Smile Sheets: Some Tips on How to Do It

Let's look at some of the biggest problems with smile sheets and get some ideas of how we can improve.

We're Asking the Wrong Kinds of Questions

One of the big problems is that we're asking the wrong kinds of questions.

The kinds of questions that we often ask relate to things like:

• Did you like the training?
• Did you learn from the training?

But studies show that having a high score for "liked the training" and/or "learned from the training" doesn't mean the training was actually effective. In short, that means that the people who attend a training aren't good judges of whether or not they learned.

What we should be do is asking if the learners feel that the training prepared them to apply the skills explained during the training when they return to the job.

Breaking that down a touch further, Thalheimer recommends questions that get at the four following issues:

• Do the learners understand?
• Will the learners remember?
• Are they motivated to remember?
• Are there after-training supports in place? (6)

Go ahead and pick up a copy of the book to learn more about this. Thalheimer goes into great detail on these issues, explains their importance, and includes a lot of sample questions too.

We're Giving Employees/Learners the Wrong Kind of Answer Options

Many smile sheet questions ask the learner a question and force them to answer by choosing one of five options, which typically include options like Strongly Disagree; Somewhat Disagree; Neither Disagree Nor Agree; Somewhat Agree; and Strongly Agree. Or the options are similar to that. This is commonly known as a Likert scale.

There are at least four problems with these kind of answer options:

• They're vague and it's not clear where the "boundary" between one and the other is. When does Strongly Disagree shade into Somewhat Disagree? What's the difference between Somewhat Agree and Strongly Agree?
• They may mean different things to different people. Your Somewhat Agree may be the same as my Neither Agree Nor Disagree, or even our coworker's Strongly Agree.
• They're not related to learning effectiveness and/or job preparedness. Even if the question is phrased well, so that it's targeted at valuable information about training effectiveness and later, on-the-job performance, these answer options don't give the learners a chance to correlate the training to whether or not they've learned the material and are prepared to apply it on the job.
• They don't give trainers actionable information. So you gather all the responses, add up all the numbers, divide by the number of respondents, and you learn that for question 1, you've got an average score of 4.2 (out of 5). Is that good or bad? Where's the cut-off between good and bad, exactly? How can that 4.2 score help you in your "virtuous cycle of continuous improvement" of training materials? If you do want to improve something in the training, what should you improve?

What's a better way to write these smile sheet answer options?

The screen grab below (taken from a recent webinar we hosted on Effective EHS Training) shows some better answer options (and a better question, relating to the last section, too). You can see these answer options are:

• Much less vague
• Related to ability to transfer training to later on-the-job performance
• Written to provide feedback that gives the trainer insight into whether or not the training was effective
• Written to provide feedback that provides valuable information to the trainer who wants to improve the training

We're Getting Smile Sheet Feedback that Doesn't Tell Us How Effective the Training Was

For the reasons we explained above, plus others, the feedback we're getting from our smile sheets doesn't tell us if our training was effective or if it wasn't. This is one of two things that Thalheimer says a smile sheet should do.(7)

We're Getting Smile Sheet Feedback that Isn't Actionable

And, also for many of the reasons we've explained above, we're getting feedback from our smile sheets that doesn't let us--as training developers, trainers, training materials, etc.--act to appropriately follow-up on the information from the smile sheet in order to improve our training. Remember those "virtuous cycles of continuous improvement?"

We're Translating Smile Sheet Feedback Into Numbers and Averages

Not only are we asking the wrong kinds of questions, and not only are we giving learners the wrong kind of answer options, and not only are we getting feedback that doesn't tell us how effective the training was and that isn't actionable, but because we often use a five-point Likert scale as answer options, we often translate those answers into a number--like 3.7 or 4.2 out of 5.

What's a 4.2, again? Is that good or bad? What should I do if I get a 4.2?

Can you reminder me again why we thought it would be a good idea to take poor information and make it one step more abstract?

More About Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Dr. Will Thalheimer

So we encourage you to run, not walk, to go buy yourself a copy of Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Will Thalheimer.

We haven't addressed this in the article yet, but here's more that we can say about the book:

• It's (admirably) short and gets to the point
• It's based in research and is evidence-based
• It's funny; Thalheimer takes his subject seriously but not himself, and you'll have some good laughs along the way

In addition, Thalheimer is a positive influence on social media, and fights the good fight for advancing good, data-driven, evidence-based training learning practices.

Here's some additional information about Dr. Thalheimer and his book:

• SmileSheets.com: This is where you can buy the book. As Thalheimer explains, if you buy it here instead of elsewhere, he gets a bigger slice of the cut and can therefore afford to spend more time researching stuff that helps us all.
• Will At Work Learning: Thalheimer's blog. Always interesting.
• Podcast with Will Thalheimer and Connie Malamed: Listen in to a 24-minute podcast during which Thalheimer discusses his book and the topic of smile sheets with Connie Malamed (one of my favorite instructional design bloggers).

Getting Up to Date: Creating Online Smile Sheets

One last thought. A lot of you have, or still are, creating paper-based learner evaluation forms/smile sheets.

You create these, you hand them out, people complete them with pen or pencil, they return them, then you have to somehow collect and analyze all that paper-based, hand-written data.

One thought is that you can use online, digital tools to move this process into the current day. For example, online websites such as SurveyMonkey have tools to help you with this.

In addition, many learning management systems (LMS) include survey-creation tools such as the one shown below that you can use to do this as well. The benefit here is that it keeps all your learning and development materials in one system.

Hope you enjoyed this article, good luck with your smile sheets, and let us know your thoughts in the comments section.

To read more about Dr. Will Thalheimer, check these interviews we've had with him:

• Evidence-Based Training and Spaced Practice
• Evidence-Based Training and Smile Sheets
• Evidence-Based Training & the Effectiveness of eLearning
• Evidence-Based Training, Learning Maximizers, and Learning Myths

We're also hoping to get Dr. Thalheimer back for more interviews on additional topics soon, so cross your fingers on that and check back in a little bit.

And hey, why not download our free guide to Learning Objectives below?

Notes:

1. Thalheimer, Chapter 1 (What Are Smile Sheets For?), Location 257. Apologies to those of you who might by this as a physical book--I bought it as an e-book and it doesn't display page numbers 🙂
2. Thalheimer, Chapter 1 (What Are Smile Sheets For?), Location 257.
3. Thalheimer, Preface, Location 109-119.
4. Thalheimer, Introduction, Location 133-143 (he's got a list of 10 problems there).
5. Thalheimer, Chapter 2 (Your Smile Sheets Suck!), Location 325. The two meta-analyses he references are one from 1997 (Alliger, Tannenbaum, Bennett, Traver, and Shotland) that included 34 separate studies, and a second from 2008 (Sitzmann, Brown, Casper, Ely, and Zimmerman) that covered 136 separate studies. Both meta-analyses found exactly the same weak correlation-0.09 (what are the odds of that?).
6. Thalheimer, Chapter 3 (Smile Sheets Should Predict Training Effectiveness), Location 808.