Case Studies in the ROI of Training
Case Study 1: Real Estate Company
When the Microsoft Office suite became the standard across a real estate company, there was a need to provide training to all employees on the most frequently used Microsoft applications—Outlook, Excel and Word. The company briefly considered outsourcing the training to an off-site training provider, but it was concerned that the costs for travel and time away from the office would require a much larger budget than an e-learning solution.
In the end, the company chose an e-learning approach and then proceeded to find the right vendor to meet its needs. Implementation of the solution incurred both technical labor and non-technical labor costs. The technical side of implementation required only a few hours of investment. The company invested significantly in non-technical labor costs for a variety of activities, however, including overall program development and program marketing within the organization. These non-technical costs represented approximately 67 percent of the total first-year budget. But once the program was launched, ongoing costs were minimal, with ongoing program coordination requiring the work of only one person for an hour or two a week.
Within the first eight months of the program, more than 150 courses were completed in full, and more than 500 courses were completed at least in part. In general, the employees were excited to have an opportunity to receive training in the software that was a critical part of their workday. The company believes that this increase in employee satisfaction had a positive impact on employee retention.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the company realized several benefits from training, only two were selected for analysis—cost savings and employee retention. These benefits were chosen because they represented the greatest near-term benefit for the company. The benefit values were also weighted (reduced) to produce a conservative quantification of their impact. Note that the time-to-payback (five to six months) is the result of estimating “straight-line” costs and “straight-line” benefits. That is, it is assumed that the costs and the benefits are realized at the same rate each month. In reality, this is not likely to be the case. Thus, the time-to-payback is an approximation.
Table 1: Cost and Benefits: Real Estate Company
One of the things that stands out in this case is the decision to limit the benefits measured to two items—cost savings and employee retention. The training group did not try to measure too many variables.
The cost of employee turnover represents a tremendous drag on any organization and was likely already being measured. Measuring the before-and-after difference was a practical and low-cost way of calculating ROI. And the positive impact on the organization—as depicted in Table 1—must have drawn the favorable attention of management.
The cost savings of e-learning was an interesting choice for inclusion in an ROI study. The measurement may have benefited the training group as much as it did senior management because it serves as a benchmark for future investments.
The training group’s grasp of the expenses involved was compelling and insightful. Identifying the expense of marketing the program internally was appropriate and showed the group understood its cost structure.
The format for the ROI presentation selected for the research project—the simple graph, statement of ROI and additional benefits—is quickly accessible. It is the type of presentation that, I believe, would be well received by a chief executive officer and a chief financial officer. The inclusion of qualitative benefits is impressive, but not overdone. On the whole, the case is credible and convincing.
It also gives me the confidence that I could do something similar and that the intangibles of measuring the impact of training are not quite as elusive as I might have originally thought.
Case Study 2: Energy Company
Business units at the energy company needed IT technical training for their employees to support new projects and products. Often there was little advance planning, and the training had to be delivered quickly. At the corporate level, employees needed ongoing access to a broad array of technical training subjects.
The company chose an e-learning delivery approach. The company did not want employees to wait for classes or be obliged to sit through entire courses. Also, the company wanted to give employees the flexibility to take as many courses as they needed. Nearly all of the training was delivered via asynchronous, Web-based, self-directed learning. Some employees, however, have begun to discuss what they learn in virtual classrooms. There are only four significant cost areas for the solution: product licensing fees, product implementation, salaries for three in-house part-time program coordinators and administration of course evaluations.
The last 12 months yielded a running average of approximately 3,000 completed courses and 7,000 partial completions. The company believes that the opportunity for employees to complete only a portion of the course was critical for improving employee productivity, which increased because employees spent less time searching for answers to technical questions and had access to training during off-hours. Because employees had answers at their fingertips, they relied less on the help desk and more on the courseware. The result was a reduction in help-desk calls and associated costs.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the company realized at least six benefits from training, Table 2 shows that only two were selected for analysis—an increase in employee productivity and a reduction of help-desk call volume. The time-to-payback is an approximation.
Table 2: Cost & Benefits: Energy Company
The reservation I have about this case is that I’m not completely clear on how the training group measured employee productivity. Clues are given in the benefits box about greater employee speed to competency, reduced re-work, higher quality of service and less downtime. Are we to assume these factors are the basis of the productivity improvements? With the exception of employee speed to competency, these items are clearly quantitative, and measurements likely already exist. Speed to competency might be easily quantifiable for this company; we just don’t know enough of the context.
I’d be more comfortable with this presentation of ROI if the training group had been clearer on what went into the employee productivity measurement, which is the crux of their case. They could have easily broken productivity into components and raised the credibility of the report, at least from our outside perspective. As this study is presented, it almost looks too good to be true, in my opinion, and has a lower level of credibility than the first case.
Case Study 3: Defense & Aerospace Company
The third case study subject is a telecommunications consulting unit within a Fortune 100 aerospace and defense (A&D) company. The consulting unit builds highly complex information technology systems for its clients and trains clients to use those systems. Although the A&D company provided high-touch technical instruction to its clients, it did not train its consultants—the 70 individuals who were charged with building those systems. The company recognized that to better serve its clients and to stay competitive, it needed to keep its consultants on the cutting-edge of technology.
The A&D company set out to build a robust training program for its consultants. It had to be very selective when choosing course content, as the subject matter is highly technical and changes rapidly. In the end, the A&D company selected best-of-breed content from multiple vendors. This approach was more costly—more than $100,000 per year—but the investment was justified by the value of the content. As the delivery method for the training, the A&D company chose a blended approach—primarily instructor-led classes supported by self-study Web-based content, CD-ROMs and workbooks. The company determined that live, face-to-face instruction would help students gain a deeper understanding of the content. The supporting materials would reinforce what was learned in the classroom. In the end, instructors delivered roughly 60 percent of the content. Students received an additional 30 percent of the content through supplemental course materials. The remaining 10 percent was devoted to individual coaching, which underpinned the entire program. This formal coaching and mentoring process ensured that real performance improvement took place and that the training had a clear impact on the consultant’s professional development. The field service organization reported receiving several distinct benefits from the training, including higher technician motivation and competency and greater corporate productivity.
The return on investment was calculated by making conservative assumptions regarding the benefits received. Although the A&D company realized seven benefits from training, only three were selected for analysis—an increase in employee productivity, product and service quality and customer satisfaction. The time-to-payback shown in Table 3 is an approximation.
Table 3: Cost & Benefits: Defense & Aerospace Company
Two things impress me about this case history. The first is the high cost of training per student. Senior management must have had supreme confidence in its training organization and believed deeply in the potential value of the program to fund it. The second is that this company had the data to put a dollar value on customer satisfaction, typically one of those intangibles.
In a slow-growth economy, high customer satisfaction leading to customer loyalty is strategic. As you prepare your own ROI statements, talk with your peers in the sales, marketing and finance organization to determine whether they have been able to pin down a dollar-value correlation with customer satisfaction. If they have, this may be an extremely important ROI factor to measure and to compare before and after training.
A theme running through all these cases is that data already exist in different departments and that critical components of business success like customer satisfaction are being measured. For your own ROI reports, look for these. Existing data that is already being collected may give you an effective way of measuring a before-and-after difference. Knowing what data is available also helps to focus your planning for an ROI report.
Getting Started: Demonstrating Positive ROI From Training
To be fair, most chief executive and chief financial officers simply want assurance that the investment in training has a positive impact on the organization. They are not asking for astronomical return on investment. They are not asking for a 100-page research report. They simply want to know that the training organization has control over its costs and an eye to the bottom line. They will likely be thrilled to see a start made on quantifying ROI. How can you get started or, if started, refine your thinking? The CompTIA/Hurwitz report suggests:
- Start with qualitative data. Most people think of ROI as a number, but in reality, a return can be qualitative, too. For example, level-three evaluations, in which managers report students’ change in performance, can produce highly useful insights for the training manager. This qualitative analysis can serve as the foundation for building a robust quantitative measurement system.
- Conduct ROI case studies. Randomly select a few students or small teams to study. Assess changes in key performance metrics. Collect data by interviewing both the students and their managers and by reviewing data from management systems (e.g., call center data). With this approach, it is possible to project the results (with varying degrees of accuracy) to the larger population of students.
- Start by targeting 100 percent payback. Does your program at least pay for itself? To find out, determine the minimum performance improvement that will yield 0 percent ROI. Sure, achieving payback might not knock the socks off your CFO, but at least it shows that your training program justifies itself. You can use this ROI model to set minimum goals for your training program.
A suggestion not made in the report, but one that would be helpful, is to make course evaluations more business-oriented. Course evaluations have typically been paper-based, asking students to write down their feelings about the training experience. Course evaluations can be transformed by focusing on objective measurements of what is happening in key performance areas at the end of the class, as well as months after the class. Data collection can be facilitated by Internet-based systems, and there are companies that offer these kinds of services.
Although the report mentions certification, it does not explain this measurement in detail. Certification via an examination following a training program is a benchmark of how well the student mastered the body of knowledge. Certified individuals speak the same language and share the same concepts. This helps to enhance productivity. They have absorbed the standards you are promoting through training. Those who do not earn a certificate give the training organization the opportunity to take a more tailored approach. You have effectively identified those individuals who may learn differently or require more from the training organization to achieve their full potential.
An interesting study commissioned by one of the most respected companies in computing found that every dollar invested in technology certification yielded more than $300 in return. If you are not already certifying your students, consider piloting a program. All the research conducted by CompTIA and others that I’ve read indicates that the result of certification is higher productivity and job satisfaction.
My purpose in writing this article was to use the case studies to take some of the mystery out of ROI calculation. I hope that readers now have a better understanding of what to measure and how to start. As learning professionals, we know the intrinsic value of education. Now we can work with our colleagues in other departments—colleagues who are already involved in ROI measurement—and use readily available data to forcefully and credibly demonstrate to the CEO and the CFO the compelling impact that training is having on the organization’s bottom line.
Gene Salois is vice president of certification programs at CompTIA, www.comptia.org. Gene has more than 23 years of industry expertise in training and certification. He previously served as director of certificaton programs and performance assessment at Lucent Technologies.
March/April 2003 Table of Contents
Keeping a staff of experts up to date on complex and expanding technical product architecture is no small challenge on its own. Doing so while the organization rapidly grows requires determination, creativity — and as much support as your tools can provide.
Trusted as the experts in visualizing big data, this fast-growing analytics and business intelligence software company has found the support it needs to maintain and expand its industry-leading team of developers in another visualization tool: video.
Today, with a little support from Panopto, video has become an essential part of how this organization shares information. Learn how a leading software company uses video training to solve four common learning & development challenges in the case study below. You can also download the case study here.
How Video Can Solve 4 Learning & Development Challenges
1. Formal Internal Technical Product Training
The first contribution of video training to the company’s developer training process was the most straightforward. In order to ensure every member of the development team is fully up to speed on the company’s offerings, the team had already been holding a regularly scheduled 2-day all-hands conference to review product architecture.
These product architecture discussions bring together all the company’s developers to explain to each other in full exactly how everything in the company’s products worked. The course makes it easier to understand the intricacies of the software — assuming, of course, an employee can attend. In the past, those who had to miss the event for any reason could only wait for the next one — and hope they didn’t miss too much regarding anything new.
With Panopto, the company was able to record and share those classes, ensuring that employees could still experience the class even if they missed a session — and that they could go back and rewatch a session later should a question arise.
“Sharing our knowledge via on-demand video in Panopto gives our people the power to access the team’s expertise whenever and wherever they need it,” said a member of the employee training team. “For example, I had a very senior engineer in my group that wanted to start watching our Panopto videos immediately. On day two, he was watching our product architecture conference videos so he could learn about all the backend components of our product.”
Other employees instead rely on the videos as guides they can access on-demand whenever they have a question. “As they watch the videos, people often like to follow along in the product, reading the code that’s being discussed. Panopto helps speed up the product knowledge acquisition process immensely.”
2. Informal, Social Learning
To complement its formal training, the company also uses video for social, peer-to-peer knowledge transfer. As part of launching new or revised product features, the developer who created the new architecture is asked to record a walkthrough of the details and rationale behind their work.
“Recently one of our developers was asked by a new hire how to perform a specific task,” recalled a member of the training team. “A light bulb seemed to go off in his head and he said, ‘Why don’t I make a video?’ And so, that is what he did. Using Panopto he demonstrated the process step-by-step in a very short recording. Now we have that information ready for anyone else to view and learn from at any time.”
These individual videos are then stored in the organization’s video library, where teammates can search for and reference them when working on those components at a later date. In a short time, this practice has built an extensive library of internal expertise – all readily available at the click of a mouse, so that the developer’s knowledge can be shared on-demand, even if they aren’t available to answer questions in person.
3. On-Demand Cross-Departmental Training
Today the company has rolled out Panopto to all functional groups including Marketing and Sales, ensuring everyone in the organization can share information and trade expertise. As the development team leverages Panopto for training, teams all across the company make use of the knowledge stored there.
“Every time a new version is shipped, new features are there,” notes one of the company’s leaders. “All these new features need to be shared with our customers. And so, we use Panopto videos to train customer support and salespeople on these additions, so that they can then effectively communicate them to our end users.”
As a provider of exceptionally technical data analytics software, it’s no surprise that the company has an almost constant stream of questions on how the tools work — both from new prospects working to integrate the tools into their own systems and from long-time customers seeking to better utilize the product suite. That means everyone, including sales and customer service, must be up to speed on the details of how the product works.
To make sure they are, the company leverages Panopto to share its internal expertise with those key stakeholders. Managers watch for trends in common product or technical questions, and when those arise, quickly share the relevant video from the development team to ensure everyone has the right answer at hand.
“Our customer support people, as well as our new developers and those working in offices outside the US, all have access to information about the feature well before that feature makes it to our customers. The videos are an invaluable part of our overall communication strategy.”
4. Formal Knowledge Sharing that Connects Developers to the Company
Technology teams in almost any organization can sometimes feel a certain disconnect with business teams — too often finding themselves in the role of order-taker, with little if any communication shared about customer feedback or product strategy. To bridge this divide, the company uses video to help its teams connect throughout product development.
This starts with the kickoff meeting — when stakeholders across the company come together to define new feature requirements. The internal employee training team records these presentations, sharing them with every member of the new feature’s working team so as to tell the whole story of that feature, including who came up with the original idea, what data points led to it being included in the new version and more.
“With Panopto, our developers can record their demo videos for others very quickly,” the team notes. “We keep track of the story of each feature from the beginning of the process of including it in a release. Then, after it’s finished, it’s really satisfying for us to go back and watch how all that played out.”
These videos also become a key part of the launch of the new feature, shining a spotlight on the creators and giving the team the opportunity to share their perspective on the importance of the feature — connecting their work to the company as a whole.
Find Out How The Company Has Found A Way to Massively Scale Employee Training On-Demand — In Our Newest Case Study!
See how this cutting edge organization found a single solution to four opportunities to enhance employee training and enable the organization to grow. Download your copy of our newest case study today!