Few organizations have a well-thought-out measurement and reporting strategy, and there are often scant resources, limited time, and imperfect data to work with when organizations do attempt to create one. Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy offers a framework to simplify the discussion of measurement and helps talent development professionals present their organizations' data and analytics clearly.

What prompted you to write this book?

Vance: We launched an initiative called Talent Development Reporting Principles (TDRp) 10 years ago to bring measurement and reporting standards, best practices, and management principles to the field of L&D. We wanted the same kind of practical guidance for our field that accountants find in their Generally Accepted Accounting Principles (GAAP). During the last 10 years we have refined and expanded our approach to the point where we wanted to share it broadly. We wanted to write the definitive book on TDRp and on measurement and reporting in general—a book that would provide the detailed guidance for anyone to create their own measurement and reporting strategy.

Parskey: In consulting with numerous talent organizations throughout the years, we discovered that leaders and practitioners struggle with measurement. While they understand the broad strokes of measurement, most of our clients do not have measurement frameworks or standard practices. We felt that one way to drive standards in our profession was to provide talent leaders and their staff with the nuts and bolts of talent measurement through this book.

Why did you choose the title Measurement Demystified?

Vance: In our experience working with clients and conducting workshops, most L&D professionals are put off by measurement, reporting, and the general notion of running learning like a business. Most would say they entered the profession out of a desire to help others learn and develop—not to measure or report. Few have experience with measurement or reporting, particularly from a business point of view. For most it actually is a mystery. They now understand why it is important, but they don’t have a framework to organize their thinking or a language with which to talk about measures. They don’t know where to begin and the steps to follow. We wanted a title that would let them know that we understand their lack of comfort with the topic and that help is here.

Parskey: When I first started working in learning and development within a large high-tech company, the measurement methods seemed arcane. I would hear people talking about Level 1 and Level 2, and I had no idea what that meant. I kept hoping that a secret decoder ring would fall into my lap so I could join the conversation. When we were noodling ideas for a title, this one seemed so apt. This book would provide the step-by-nitty-gritty-step about measurement and remove the need for the decoder ring.

There are already several books on measurement. Why another one?

Vance: Jack and Patti Phillips have authored or co-authored more than 100 books on the topic of learning program evaluation, and others have made important contributions as well. So, the field has excellent guidance for determining whether a learning program is effective and impactful. However, this the first book to present the TDRp framework for selecting measures, including a description of more than 100 efficiency measures that are generally not the subject of other works. This is also the first book to integrate the recent important work on human capital reporting by the International Organization for Standardization (ISO). It is also the first book to categorize the various types of reports, create a framework for reporting, and directly tie the selection of a report to the reason for measuring. Last, unlike other books, we provide detailed step-by-step guidance to create your own measurement and reporting strategy, including how to create the all-important plan, year-to-date results, and forecast values in advanced reports.

What are the right steps to create a measurement and reporting strategy?

Vance: The first step is to be clear on the reasons to measure. Talking with the user to understand their need may simply be an answer to a question (to inform), or to ensure the value of a measure meets a threshold (to monitor), or determine if a program is effective (to evaluate), or last to ensure a program delivers promised results over the course of the year (to manage). With purpose clearly in mind, the next step is to select the appropriate and balanced set of measures. The TDRp framework groups measures into three buckets: efficiency, effectiveness, and outcomes. Generally, you will select both efficiency and effectiveness measures and sometimes an outcome measure as well. The book describes more than 120 measures in detail so there are many from which to choose. Last, the selected measures need to be included in the right type of report, which may range from a simple scorecard to a more complex dashboard to a program evaluation report to a detailed management report like your colleagues in sales and manufacturing employ.

Parskey: The process of creating a measurement and reporting strategy needs to be a collaborative effort. The L&D measurement team (or person) should not go off into a corner and create a brilliant plan on their own. Instead, L&D practitioner(s) should engage with the CLO, directors, and program managers to ensure that the measurement and reporting strategy reflects both the stated and unstated needs of both learning and business leaders. Without engagement and approval of the plan by these leaders, the plan is unlikely to be successfully implemented.

Analytics has gotten a lot of attention over the last several years and is seen by many as a trendy topic in the HR field. What role do you think analytics should play in a measurement strategy?

Vance: Analytics is not the highest-level purpose of measurement, although it is a very important part of measurement. The profession needs to move away from merely reporting historical results in scorecards and dashboard—which is all backwards looking—and use measures to evaluate the effectiveness of a program. We are also excited by the opportunity to use analytical methods like regression and other statistical techniques to explore the relationships among measures. However, we firmly believe that an even more important purpose exists for measurement, a purpose that employs all the reporting and analysis that must precede it. We believe the highest-level purpose for measurement is the active management of key programs to deliver significant business value. This requires high-level analytics to properly understand and plan the program, and a continuing high-level of analysis to understand what corrective action must be taken each month to maximize the probability of delivering promised value by year end.

Parskey: Advanced analytics tools, methods, and technology enable organizations to get deep insights into relationships among HR variables and can often predict outcomes such as the likelihood that high performers will stay with the organization or the degree of organizational fit for prospective hires. Analytics is critically important to enable HR to “compete on analytics” as Tom Davenport wrote in 2007. It is also a hot area, and who doesn’t want to be working on hot stuff? However, as Dave indicates, analytics is not enough. When we go beyond analytics to managing the function, we become accountable for our results and start behaving like our colleagues in every other function. While not nearly as hot as analytics, the practice of managing (versus monitoring) enables us to make significant contributions to business success.

What is holding L&D professionals back? Why do they find it hard to get started?

Vance: The lack of a framework and knowledge our book addresses has been holding back L&D professionals in the measurement and reporting area. In addition, even when L&D practitioners know the steps, many lack the confidence to create a measurement and reporting strategy. For some, the task simply seems too big. For others, there are resource constraints. We believe it is important just to get started, even with a few measures contained in a few basic reports. It is too easy to put off getting started if you are waiting for perfection and perfection is an illusion in any case. Plus, the only way to build confidence, is to practice. Better to jump in and gradually grow your sophistication, adding measures and reports each year as you go. Over the years of plugging away, you will have a respectable strategy.

Parskey: Another factor holding L&D back is that we don’t fully emulate business practices of our colleagues in other functions, in particular in goal setting and being accountable for concrete goals not only for activity and effectiveness but also impact. Being accountable can be daunting, particularly when you don’t control all the variables that impact the outcome. But that’s the nature of managing. Consider sales. A sales manager has a target sales level they must meet. They don’t control the buying habits of their accounts, they don’t control purchases practices within their client, and they certainly don’t control the economic landscape that can make or break success. But salespeople find it perfectly natural to be assigned a goal and be held accountable to meet it. L&D needs to view the world from the same lens. We can’t control everything, but we can influence the results we are accountable to achieve. L&D organizations simply need to start setting goals, reviewing progress, taking action to achieve planned results, and reforecasting year end performance. With practice and experience, L&D leaders and their teams will become comfortable with the process and will reap the benefits of managing their function.

What does “running learning like a business” mean and why is it important?

Vance: The concept of running learning like a business means that specific, measurable plans are created at the start of the year for your key initiatives and programs. Those plans are executed with discipline to ensure planned results are achieved. For example, you may have committed to reaching 1,000 participants through five new programs by the end of the third quarter. Once the year is underway, you use monthly management reports to compare your progress against plan. If you are falling behind plan, you identify the reasons why and options to get back on track, and then you enact those options. If that doesn’t work, next month you are at it again. You keep at it until you get back on plan or the year ends. This is how your colleagues in other departments have managed for years. Running learning with this same business discipline is key to earning senior leaders’ trust and resources. The concept also underpins the management reports and the highest level use of measures for L&D.

Why is a balanced set of measures important?

Parskey: Eliyahu Goldratt, in one of his management books says, “Tell me how you measure me, and I will tell you how I will behave.” Given that measurement is such a powerful trigger for behavior change, we need to ensure that our measures will incent the behaviors we desire. When we develop a balanced set of measures, we mitigate the unintended consequences that result from focusing on just one. For example, in the book, we provide a real-life example of a CLO who only focused on efficiency measures. He cut staff, reduced spending, and provided training that while efficient, was not even remotely effective. The people in the business filled the gap by developing their own training, which resulted in further lost time, incremental expense, and a delayed time to competency for new hires. Had the CLO focused on both efficiency and effectiveness, he likely would have identified how to provide the most effective training at the lowest cost, therefore optimizing both measures of success.

What are the next steps for the profession with regards to measurement and reporting?

Parskey: Because of the new SEC rules about disclosure of human capital metrics, HR, talent and L&D in publicly owned organizations must now disclose people measures material to the business. As such, talent functions will need to get their internal reporting house in order, to ensure their public disclosures are accurate and valid reflections of their internal processes. For many organizations, the SEC rules will be the catalyst to develop or adopt standards (such as TDRp), streamline their processes, improve the quality of their data and implement technology that simplifies data collection and reporting. We believe these efforts will enhance not only the quality of reporting but also how L&D and HR manage the function.

About the Authors

David Vance is the executive director of the Center for Talent Reporting, a nonprofit dedicated to the creation and implementation of standards for human capital measurement, reporting, and management. He is the former president of Caterpillar University, which he founded in 2001. Prior to that position, Dave was chief economist and head of the business intelligence group at Caterpillar Inc. with responsibility for economic outlooks, sales forecasts, market research, and competitive analysis.

He is the author of The Business of Learning, now in its second edition, and co-author with Peggy Parskey of Measurement Demystified, just published by ATD Press. He teaches in the PhD programs at Bellevue University and the University of Southern Mississippi and in the executive education program at George Mason University.

Peggy Parskey is the assistant director of the Center for Talent Reporting. Peggy owns her own consulting firm, Parskey Consulting, providing measurement and organizational change consulting services. Peggy leverages more than 25 years of expertise in performance measurement, management of change, human performance improvement, and knowledge management to improve clients’ organizational, team, and individual performance.

Peggy is also a principal consultant in measurement with Explorance, a firm focused on talent measurement across the entire employee lifecycle. She is the co-author of Learning Analytics, second edition, with John Mattox and Cristina Hall, and is the co-author with David Vance of Measurement Demystified, recently published by ATD Press

About ATD and ATD Press

The Association for Talent Development (ATD) is the world’s largest association dedicated to those who develop talent in organizations. ATD’s members come from more than 120 countries and work in public and private organizations in every industry sector. ATD Press publications are written by industry thought leaders and offer anyone who works with adult learners the best practices, academic theory, and guidance necessary to move the profession forward. For more information, visit td.org/books.

Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy

ISBN: 9781950496891 | 432 Pages | Paperback

https://www.td.org/product/book--measurement-demystified-field-guide/112116

To order books from ATD Press, call 800.628.2783.

To schedule an interview with David Vance or Peggy Parskey, please contact Kay Hechler, ATD Press senior marketing manager, at [email protected] or 703.683.8178.