These metrics address the five levels of evaluation recognized in training literature, combining the four basic levels of learning measurement introduced by Donald Kirkpatrick in 1994 along with a fifth level added by Jack Phillips that addresses return on investment (ROI). These five levels of what has become known as the Kirkpatrick-Phillips Learning Measurement Model consist of:

Level 1 - Reaction: measures learners’ satisfaction with the program & captures planned actions.
Level 2 - Learning: measures changes in knowledge, skills and attitudes (competency).
Level 3 - Application: measures changes in on-the-job behavior.
Level 4 - Business Impact: measures changes in business impact variables.
Level 5 - Costs and Benefits Assessment: compares program benefits to the costs.

The specific measurements used in the transit industry with this model include:

• Supervisor and worker training satisfaction surveys.
• Promotion, retention and wage increases resulted from training.
• Pre-test and post test results and practical examination pass rates before and after training.
• Transit maintenance efficiency and quality improvements following training, including:
          a. Mean distance between failures/road calls
          b. Late pullout/Bus days lost/Runs cancelled
          c. Maintenance cost per mile/vehicle operated/vehicle available (labor and parts)
          d. Repeat and chronic mechanical failure/Part rework
          e. Percentage of unnecessary part replacements
          f.           Percentage of preventive maintenance jobs performed on schedule
          g. Ratio of scheduled versus unscheduled maintenance.
• Vehicle operator improvements resulting from training including enhanced customer service and satisfaction, more effective pre-trip vehicle inspections, driving practices that save fuel, improved vehicle handing in adverse conditions, etc. 
• An assessment of the financial benefits and costs of training program.

In Pennsylvania, the Keystone Development Project and the Center helped introduce a unique partnership-based, data-driven training program that has produced very positive results for the state of Pennsylvania. A combined investment of $2.6 million in this training through state government funding and employer match has produced an estimated cost savings of between $10 million and $22 million in bus maintenance and fleet procurement over a four year period. The resulting four-year ROI is estimated between 293 and 745 percent. The annual ROI rate has climbed to a stable 5 to 12 times the investment in subsequent years.

As stated by Doug Greenwood, Director of Maintenance Cambria County Transit Authority, Johnstown:

“Everyone we sent [to Keystone training] has come back home showing results. Our electrical problems are fewer with faster repairs because they now know what to look for. Our A/Cs have become more reliable. All in all, this program is a plus.”

An ROI study can show your agency through a numerical analysis why a partnership-based training program is worth the time and expense. THE BOTTOM LINE in today’s economy: designed formulas to achieve top return.