Years ago, as a young instructional designer, I was asked to evaluate training. I knew most of my clients were happy to get “smile sheets” distributed and didn’t go much beyond that.  One day I had a client that wanted a Level 3 evaluation. Curiosity always gets the best of me and I hate not knowing things. I just HAD to learn more about training evaluation! I thought I was doing well until the day I was asked if I could do an ROE study instead of an ROI study.  A what?! 

I know I’m not the only one who has been confused by requests to evaluate training. I know that others feel my pain when it comes to understanding training evaluation,so I thought I would attempt to unravel some of the differences between Kirkpatrick’s “Four Levels” and the Phillips 5-level ROI Methodology.

Just to be different, let’s start at the beginning…

In November 1959, Don Kirkpatrick presented his thoughts about training evaluation in a series of articles published in the Journal of the ASTD.

He used the four words that are currently known as the Kirkpatrick Four Level Evaluation Model: Reaction, Learning, Behavior, and Results He emphasized that all four levels are important to what we mean by evaluating, especially if the purpose of training is to get better results by changing behavior. In his book “Implementing the Four Levels” (Kirkpatrick and Kirkpatrick, 2007) Don and his son Jim talked about the “importance of evaluating the four levels, or as many as you can, in sequence” to “build a compelling chain of evidence as to the value of learning to the bottom line” (p. 123). They emphasized the need to present the value of training in a way that will “maximize the meaningfulness to the hearts and minds of your internal stakeholders” (p. 123). They suggested that presentation of the links between levels will help learning professionals obtain larger training budgets and commitment to carry out the changes needed to achieve organizational objectives. Over time, Don Kirkpatrick’s original work has evolved. Jim and Don’s daughter-in-law Wendy expanded the original work by creating the New World Four Levels and the Kirkpatrick Business Partnership Model, which emphasizes the need to focus on collective efforts to accomplish return on stakeholder expectations. This return on expectations (“ROE”; Kirkpatrick and Kirkpatrick, 2009) should not be confused with the common use of ROE meaning “return on equity,” which is determined by comparing net income to shareholders’ equity. The Kirkpatricks describe return on expectations as the ultimate indicator of value. It is a holistic measurement of all of the benefits (both qualitative and quantitative) realized from a program or initiative brought about through a package of interventions, with formal training typically being the cornerstone (Kirkpatrick and Kirkpatrick, 2009).

To achieve return on expectations, learning professionals must ask questions to clarify and refine the expectations of key business stakeholders as they attempt to satisfy the stakeholders and convert their expectations into observable, measurable business or mission outcomes.  Because a great deal of employee learning occurs on the job, the Kirkpatricks emphasize the need for learning professionals to also partner with supervisors and managers to encourage them to prepare participants for training and to play a key role after training by reinforcing newly learned knowledge and skills through support and accountability.

The Kirkpatricks emphasize that formal training events by themselves do not deliver significant bottom-line outcomes (Kirkpatrick and Kirkpatrick, 2009). Using the Kirkpatrick Business Partnership Model, how do you determine the impact of training? The Kirkpatricks proudly claim, “We are not isolationists” (Kirkpatrick and Kirkpatrick, 2007, p. 107).  They put relatively little emphasis on using estimates, assumptions, and empirical financial formulas to try to isolate the effects of training. However, they do acknowledge that the results always come from a variety of factors, and if you do not measure and know which factors aided in the transfer of learning to behavior and subsequent results, you will not be able to take any credit for the success (Kirkpatrick and Kirkpatrick, 2007). So although they do not attempt to isolate outcomes related to training, they do place emphasis on validating that the transfer of learning to on-the-job behavior is related to training.

Now let’s understand where Jack Phillips enters the picture…

In the early 1980s Jack Phillips wrote his original work on training evaluation. He attempted to stretch the training community to move beyond Level 4 to a financial accounting of program success, return on investment (ROI), which he called Level 5. This level of evaluation was intended to assess the value of expensive, high-profile programs by incorporating the steps of cost-benefit analysis. In addition to adding the fifth level of evaluation to the Kirkpatrick four-level framework, Phillips broadened Levels 3 (Application and Implementation) and 4 (Impact) to include transfer of learning and outcomes to processes other than training (Phillips, 1995). He also developed a process model and guiding principles to support implementation of the evaluation process at each level, and replication of the process across organizations.

An important part of what has become known as the Phillips ROI Methodology is the critical step to isolate the effects of the program. Without this step, Phillips believes training cannot take credit for improvements in Level 4 impact measures with accuracy or credibility. A variety of techniques can be used to isolate the effects of a program. While estimates tend to get the most attention because they can be applied in any situation, other commonly used approaches are control groups, trend line analysis, and forecasting models. These techniques have been used for decades to attribute programs and initiatives with outcomes. The fall-back position is to use estimates that have been adjusted for error. But this step in the process gets you to a credible measure of impact. ROI requires additional information which moves the outcome to a different level, Level 5 ROI.

ROI is a financial metric describing a return on investment for a program. It compares the monetary benefits of a program to the investment in the program itself. Because benefits and costs are both measured using money, they can be compared using the same metric or scale. Further, the ROI statistic can be compared to other programs either inside or outside of the organization (Phillips, 2012). ROI can be used to forecast benefits, and make investment decisions, but can also be used to measure past performance.  It is typically reported as a percentage and represents the annual net benefits returned beyond the initial investment (Phillips, 2012).

Is Phillips saying that all programs need to do a comprehensive ROI study? No. The highest recommended measurement level depends on a variety of factors including the goals, objectives, costs and visibility of the program. Phillips provides guiding principles to make those decisions, and similar to the Kirkpatrick’s “chain of evidence”, he discusses a “chain of impact” and recommends when evaluating at a higher level it’s important to evaluate at lower levels as well.  Doing so will help explain the results at the higher levels, and provide relevant information to stakeholders. Patti Phillips’ book, “The Bottomline on ROI,” contains a table of the suggested percentages of programs to evaluate at each level. The higher you climb up the levels of evaluation, the smaller the percentage of programs she suggests evaluating. She recommends that 90% to 100% of programs should be evaluated at Level 1, but only 5% to 10% should be evaluated at Level 5.

In a nutshell…

The differences between the Kirkpatrick Business Partnership Model and the Phillips ROI Methodology are important to understand if you’re trying to make a decision between focusing on return on expectations or return on investment.

The most obvious difference is the fact that the Kirkpatricks focus on four levels and the Phillipses focus on five levels. ROI is not called out as a separate level in the Kirkpatrick approach, but it is in the Phillipses approach, due to the fact that ROI requires a new set of data not found in the other levels of evaluation. It requires normalizing benefits and costs so the two can be compared on the same scale and the outcome can be compared to other programs. The Phillipses encourage isolating the impact of a program because it ensures credibility of results. They believe senior executives want to see how all factors uniquely influence outcomes.  The Kirkpatricks focus on the Business Partnership approach, emphasize the need to focus on collective efforts to accomplish return on stakeholder expectations, and do not attempt to isolate the effects of training. They factor in, not out, contributions from the efforts of the training graduates and other internal partners; factors that allow learning and performance professionals to be true, strategic business partners. So, do you want to focus on return on expectations or potentially calculate a return on investment? Just like so many things in life, it all depends. In this case, it all depends on the goals of your evaluation, the metrics you intend to use to measure success, and the data needs of your stakeholders.

Resources

Kirkpatrick, D. (1998). “Evaluating Training Programs: The Four Levels” (Second Edition). San Francisco: Berrett-Koehler Publishers, Inc.

Kirkpatrick, D.L. & Kirkpatrick, J.D. (2007). Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs.  San Francisco: Berrett-Koehler Publishers, Inc.

Kirkpatrick, J. & Kirkpatrick, W.K. (April 2009). “The Kirkpatrick Four Levels:  A Fresh Look after 50 Years 1959-2009”. White paper published by Kirkpatrick Partners LLC.

Phillips, J. J. (1983). Handbook of training evaluation and measurement methods. Boston, MA: Butterworth-Heinemann.

Phillips, J. J. (1995, Summer). Corporate training: Does it pay off? William & Mary Business Review, pp. 6-10.

Phillips, P.P. (2012). Bottomline on ROI (2nd edition). HRDQ Press.