John A. Peart

Look At Your Training Objectives

You’ve got numbers from your training. Now what? If you find that your data are unclear, or don’t seem to guide your next steps, you’re not alone. Let’s take care of that.Preparation CountsFirst, let’s review how to set yourself up for success with metrics. Here are the key steps to prepare for action:Set measurable goals for the course. For example, start from learning objectives and set goals for how much people should learn. (See” Track Success with Metrics.”)Find the features of the course that impact the goals. For example, use A/B testing to measure the effect on the course of including videos, or making videos of a certain type. ( See “Find the Levers with Metrics.”)If you you’ve done these things, you’ll be ready for your next steps. If you didn’t do these things, and you’ve got a pile of numbers to make sense of, you’ve signed up to be a data miner. Now, mining data is an honorable profession, and if you have a background in statistics and a large volume of data (say, thousands of students), you could give it a try.However, with the limited size of datasets that we typically get from training, it’ll be hard to find patterns that are significant. So, you’ll come out ahead if you have a system designed from the start around measurable goals and measurable features.Course Focus vs. Portfolio FocusDesigning a course around measurable learning objectives may already be part of your design process. We are used to thinking about training in this way. And, this is work that can happen in isolation from other courses. For example, the learning objectives for sales training for Widget X probably have no relationship to the learning objectives for engineering training for Widget Y. This compartmentalization makes it relatively simple work to set measurable goals for courses.To act on metrics, however, requires a portfolio view. To find the courses that need work, and to accumulate insights into what levers work best, is a long-term effort that spans many courses.First, to find the courses that need work, we have to know what an underperforming course looks like. While we will have some external criteria for this, such as “X number of people need to get at least 80 percent of the quiz answers correct,” the best way to address this is to look across the portfolio and find the lowest courses. In other words, establish a baseline, then find the low outlier that should be prioritized for improvement.Next, your experiments to find levers of quality (for example, through A/B testing) will be spread out across many courses. You won’t test every feature of every course, though. In fact, you might test only one or two features in each course to make the job manageable and data easier to interpret. As you accumulate a “bag of tricks” through your testing, you can apply things you've learned from one course to the design and troubleshooting of other courses.Put Best Practices to WorkCongratulations—you’ve set yourself up with some well-designed goals and levers. Now, your next steps are relatively easy.Redesign an existing course: If your data show that a course is underperforming, and it can be adjusted by one or more of your levers, consider redesigning and republishing. This assumes that course will continued to be delivered long enough to justify the investment, which you can ensure by checking your data early in the life of the course. In fact, it’s best to plan for iteration with any course: publish, collect data, adjust, and publish again.Approach future courses differently: Keep track of the best practices that you uncover through testing, and apply them in the planning stage for new courses. More importantly, once you’ve established best practices, you’re going to include them in all of your new courses. What if one of these new courses still underperforms? At that stage, you can have a discussion about the level of investment in the course. Because even with knowledge about best practices, you’ll naturally make tradeoffs between what works best and what you can afford.If the course is a priority for the business and it’s not working well, it may be time to add more high-end features, such as virtual instructors. And thanks to your work with metrics, you were able to point out that it wasn’t working, and you know what to suggest to improve it.Share Best PracticesAs your final action, share your findings with others. When you find what works best for your training, there’s a good chance it’ll be a benefit to other designers. Of course, mileage for a particular design choice may vary. Again, what you find works for engineers in your industry may not apply to another industry or to sales, for instance. So, also keep track of the demographics of your audience and include that information with your report.In the long run, you could help lay the groundwork to make training a leading profession in the data-driven workplace of the 21st century.

Act on Metrics

ATD Blog

While facilitating leadership development workshops at Emerson (a Fortune 125 company with 90,000 employees in 140 countries), there is an interesting comment I frequently hear: “I did not realize how much time you would give me in this workshop to think!”Is that a compliment or a complaint? I believe most people mean it as a compliment because they are surprised. Most of them haven’t had enough time to think in prior trainings.I’ve heard this comment many times, and it never takes me by surprise. In fact, I would be disappointed if I did not hear it. Most learners aren’t given adequate time to think and reflect when they attend training sessions.Far too many training programs rely on content rather than application and do not offer learners time for intentional reflection. In my 28-year learning and development career, I’ve seen too many programs approach learning by telling learners as much as possible in the time provided. Many of us in L&D call that strategy “ten pounds of training in a five-pound sack.”Fred Harburg, former chief learning officer at Fidelity Investments, sums it up well: “If there is no time for reflection, there is almost no chance for improvement.”L&D professionals want to create value for learners. One of the most valuable activities we can build into the design and delivery of our courses is intentional reflection and implementation planning. To improve our training programs, we must stop talking and pushing content, and let learners think.This represents a paradigm shift for many L&D practitioners. Spend less time on content and more time on intentional thinking and reflection.L&D thought leaders Roy Pollock, Cal Wick, and Andy Jefferson (authors of The Six Disciplines of Breakthrough Learning) popularized the term “learning scrap.” Learning scrap refers to content covered in a workshop that is never applied. Most L&D programs have a typical learning scrap rate of 85 percent. At best, most learners apply 15 percent of what’s taught in a workshop on the job. If you want to decrease learning scrap and increase learning transfer, build more deliberate thinking and reflection activities into your courses.In his classic book, Learning and Memory, Donald A. Norman writes, “To remember is to have managed three things successfully: the acquisition, the retention, and the retrieval of information. Failure to remember means failure at managing one of these three things.” I argue failure usually occurs at the retention level. If there is no reflection, there is no retrieval. If there is no reflection and retention, do not expect retrieval to follow.This issue is well-documented by Ruth C. Clark, Frank Nguyen, and John Sweller in another classic book, Efficiency in Learning. They refer to this concept as “information fatigue.” We need to give learners more time to process during a training program. The authors’ research shows a lack of time for reflection leads to cognitive overload. Overload implies a malfunction; learning is short-circuited. If a person attends a program to learn, and we cram it full of content without giving them time to think, it’s no wonder learning scrap occurs.Several years ago, I codeveloped a workshop with Michael Watkins (a former faculty member at Harvard Business School and founder of Genesis Advisers) to create a workshop based on his bestselling book, The First 90 Days. Our workshop designed intentional thinking activities that asked participants a series of questions at the end of each module. The reflection activities took between 20 and 30 minutes to complete and consisted of five or six questions. These were not simply recall or comprehension questions, but rather synthesis and evaluation questions (the higher levels of cognitive thinking).Some of the top executives who attended this workshop told me that if I had not built in deliberate time to think, their goal would have been to rush through the course so they could move on to the next thing on their list.Here are some ways to build time for reflection into your programs:
Build thinking time into the design and delivery of every learning program you create, no matter the delivery method (in person or virtual).Resist the temptation to add more content to your courses. Do not fall prey to the “myth of more.” More content at the expense of intentional time for reflection will not improve learner performance. It will diminish it.Remember, the goal is not to give the content. The goal is for learners to get it, which will only happen through the discipline of quiet reflection.If trainers are given the option to talk or let their learners think, many would choose to fill the silence with their own voices, but the greatest value trainers bring to a workshop is knowing when to stop speaking and give learners time to reflect.