Dive into data science using highlights from ATD's latest industry benchmarking report.

Evaluating learning impact is a key skill for talent development professionals. As noted in the TD article "The Overwhelming Task of Communicating L&D's Business Impact," TD practitioners must take a hard look at what analytics they should be measuring and what those analytics signal. Bringing numbers to the conversation is vital to communicating with business leaders, so you should strive to understand the metrics behind what the TD function does, why the metrics matter, and how to interpret them. That means incorporating the latest techniques to examine the data you have and the meaning, as well as the uncertainty, behind them.

To show what that looks like in practice, let's use data from the State of the Industry, the Association for Talent Development's annual report on TD efficiencies and expenditures. Although the report presents only averages to protect participants' confidential information, in this article, we'll go beyond those numbers using a simple data science technique that you can use to better understand the data available to you. The technique is called bootstrapping.

About the bootstrap

Imagine that you sat in on 30 random sessions of the same training program to get a sample of how long the course takes to complete. Normally, you would take the average length of the 30 sessions, and then you may run a basic statistical test to get an idea of where the true average realistically would fall. That strategy works great if the data follows the normal distribution the test assumes you're working with—that bell-shaped curve with which you're probably familiar.

But what if you had numerous sessions that went about the same length and five or six went much longer? You can't trust the results, because the traditional statistical test only works if the data is in the shape of a bell curve.

Bootstrapping solves that problem by simulating the data set. It uses a computer program (R, Python, and SPSS are ideal, but Excel could also work) to take the data on the training sessions you observed and resample it with replacement to build thousands of new data sets. In our scenario, the bootstrapping process would randomly take the length of one of the training sessions you observed; add it to a new, simulated sample; and put it back in the pool of training sessions to draw from. Now the session you just drew has the same chance of being drawn again as any of the other ones in the data set.

You have the computer program keep doing that until there's a new data set. Then you repeat the process thousands of times until you have thousands of new, simulated data sets. Each of those simulated data sets has its own average time per training session, and by looking at all the averages, you can see where the real average likely falls.

Now, let's apply that technique to key TD metrics from the 2022 State of the Industry report for which 174 unique organizations submitted data for fiscal or calendar year 2021.

Total expenditure on formal learning

On average, organizations spent $1,280 per employee on direct learning expenditures in 2021, up from $1,267 in 2020 and down from $1,308 in 2019. So, what's the real average? According to a bootstrap test, there's a 90 percent chance that the true average direct learning expenditure falls between $956 and $1,636 per employee, as shown in Figure 1.

That range is obviously pretty wide. The reason is because the range of expenditures submitted for the State of the Industry was likewise wide. Every organization is unique, faces vastly different circumstances from other organizations, and has its own strategy to train and develop its employees. ATD's research has found that is even true within the same industry and workforce size group.

Therefore, if your company spends the State of the Industry average of $1,280 per employee, your organization is typical. However, if it spends more than $956 and less than $1,636 per employee, it's probably within a reasonable distance of the average organization overall.

To calculate your company's average direct learning expenditure, divide the total expenditure on formal learning by number of employees. For example, if an employer spent $10 million on formal learning and had 10,000 employees, its direct learning expenditure is $1,000 per employee.

Direct learning expenditures include TD staff salaries (including taxes and benefits), travel costs for TD staff, administrative costs, nonsalary development costs, and delivery costs (such as classroom facilities and online learning technology expenditure, learning supplier expenses, and tuition reimbursement).

In general, ATD's data has historically shown that the larger an organization is, the more likely it will be lower on the scale in terms of direct expenditures. Larger employers have traditionally realized cost advantages because they can spread the cost of their learning programs across more workers, which results in a lower cost per employee.

However, that trend reversed last year. In 2021, large companies with more than 10,000 employees had higher per-employee expenditures than midsize companies with between 500 and 9,999 employees and small companies with between 1 and 499 employees.

The changing trend for 2021 may be due to organizations adjusting their strategies to cope with changing economic conditions associated with the pandemic. According to Gallup research, 26 percent of full-time employees worked exclusively from home in December 2021, down from a peak of 54 percent in April 2020 at the start of the pandemic. With more employees returning to the office than small companies, large companies generally had more learning programs they needed to adjust to address all their workers' needs.

The change may also stem from last year's tight labor market. Research from the Organization of Economic Cooperation and Development found that all 38 of its member countries saw increased job vacancies even though employment had returned to pre-pandemic levels.

TD staffing levels

The State of the Industry found that, on average, organizations had one TD staff member for every 353 employees in 2021 after adjusting for outsourcing. That was higher staffing than in 2020, when companies had an average of 385 employees per TD staff member but lower than in 2019, when companies had 330 employees per TD staff member. However, as with the learning expenditure numbers, those are the averages for the organizations that participated in the study.

According to a bootstrap test, the real-world average for TD staff per employee after adjusting for outsourcing is somewhere between 224 and 499, as shown in Figure 2. Like expenditure per employee, this range is quite wide. Some companies had as few as a dozen or so employees for every TD staff member, but at the other end of the spectrum, some organizations had thousands. That only further highlights the fact that companies' different business situations, strategies, workforce sizes, and industries have an impact on how to approach TD.

Historically, ATD's research has found that the ratio of TD staff members to employees increases as an organization's workforce size increases. As traditionally seen with direct expenditure per employee in the State of the Industry, the reason is because of economies of scale—the costs of learning are spread among more employees, which drives them down.

Cost per learning hour used

The average cost per learning hour used in 2021 was $103, according to the State of the Industry. When applying a bootstrap test to the data, the average cost per learning hour used falls somewhere between $82 and $126 (see Figure 3). The bottom of the range is even above the $76 in 202The Overwhelming Task of Communicating L&D’s Business ImpactThe Overwhelming Task of Communicating L&D’s Business Impact0 and $77 in 2019, which means the average cost per learning hour used in the real world for 2021 was almost certainly higher than in 2020.

Employees returning to the physical office likely played a role in the increasing cost to deliver formal learning. With more workers back in the office, the average organization increased the percentage of formal learning hours it delivered via an instructor-led classroom to 30 percent in 2021, from 18 percent in 2020. Additionally, the State of the Industry found a positive, statistically meaningful correlation between the cost per learning hour used and the percentage of learning that companies delivered via instructor-led physical classroom training. In other words, companies that used more in-person classroom training generally spent more to deliver each learning hour. Therefore, it's fair to suspect that the return of this more expensive delivery method helped drive the increase.

A learning hour used is an hour that an employee spent in formal training. For example, if a worker participates in one hour of formal training, it counts as one learning hour used. When aggregated, this metric measures the total number of hours employees spent in formal training.

To calculate your organization's cost per learning hour used, divide the company's total expenditure on formal learning by the number of hours employees spent in formal learning. For example, if an employer spends $10 million on formal learning across 100,000 learning hours, its average cost per learning hour used would be $100.

Measuring up

How do the learning expenditures and efficiencies data at your organization compare to the ranges presented here? Are your numbers within the confidence intervals? If so, that means your organization is reasonably close to the real-world average.

It's not necessarily bad if your numbers are different, though. All companies face their own circumstances and have unique strategies to develop their talent. As such, it's better to understand why your numbers may be higher or lower than the ones presented here, along with the root causes of those differences. If you can justify the root causes, then your company's efficiencies and expenditures probably make sense in context. If not, then it may be time to have a serious conversation with your business leaders, because your organization's expenditures are outside the norm without strong justification.

Download the 2022 State of the Industry report at td.org/SOIR.

Talent Development Staff Members per Employee

To calculate your organization's number of talent development staff per employee, divide the number of TD staff by total number of employees. For example, if Company A has 1,000 employees and 10 TD staff members, the TD staff-to-employee ratio is 1:10.

It's a good idea to account for outsourcing when calculating TD staffing levels. Outsourced learning activities such as purchasing content or software licenses reduces the burden on TD staff.

To adjust your company's TD staff per employee by outsourcing, multiply the organization's overall TD staff members per employee by the percentage of its formal learning expenditures that it spent on learning suppliers. Then subtract that number from the total learning expenditure. For instance, if Company A spent 20 percent of its training budget on outsourced activities, after adjusting for that, the TD staff-to-employee ratio would decrease by the same percentage, from 1:10 to 1:8.