In 1959, Donald Kirkpatrick outlined his now famous four levels of evaluation for training programs: reaction, learning, behavior and results. Building upon this foundation, Jack Phillips has since added a fifth level, that being the Return On Investment (ROI) produced by a training program using the financial formula:

Unfortunately, both the Kirkpatrick model and Phillip’s ROI formula may not provide the type of information needed by trainers and instructional designers to evaluate and improve training systems. To address this need a new model has been developed.

A New Training Systems Evaluation Model
To begin, let’s start by reviewing the elements of a simple systems model. As illustrated below, we can represent any system as being composed of three basic elements.

First, let’s recognize that all systems produce results. Albeit, the results they produce may at times be difficult to predict, but nevertheless, systems produce results. Secondly, to achieve results, something must happen; something must be done. In other words, activity must take place. Third, in order to have activity, resources must be used. These three elements of a system hold true whether the system is mechanical, electrical, biological, financial, social or educational. 

The next step to understanding this new evaluation model is to examine the dynamic relationships that exist between the elements of a system. We will start with the relationship between results and activity. 

Effectiveness
We can say that something is effective only when an activity produces a predetermined result. For instance, when the Wright brothers were designing their airplane, I am sure they built a number of contraptions with wheels and wooden structures covered with cloth. However, these things were not airplanes. Why? Because these machines did not fly, they may have rolled, jumped, or shuddered, but they were not airplanes. It was only when the brothers had built a machine that could fly under its own power, could Orville and Wilber call it an airplane. When something does what it is suppose to (with a high degree of predictability), then and only then, are we able to say that it is effective. 

Effectiveness is concerned with "how well" something works (produces predetermined, desired results). This concern for "how well" is the basis for another concept known as quality. As designers of systems, we must strive first for effectiveness. For without it, there is little reason to proceed with the design of any system. 

Efficiency
Efficiency is the dynamic relationship that exists between resources and an activity. For example, when Mr. Ford built his automobile that would travel between Dearborn and Lansing using 20 gallons of gas, I am sure he began to think about improvements. One of his first thoughts about improving the efficiency of his car may have been, "I wonder if I can build a car that can go the same distance on only 10 gallons of gas?" Note: he still wanted to go from Dearborn to Lansing (same results), only he wanted to do it more efficiently. 

Efficiency asks the question, "How much?" However, this is a dangerous question to ask. Dangerous in that if we cut resources too much, we run the risk of producing poor results. 

This is not to say that we should not be concerned with resources and their associated costs. We should, because there is an optimum balance within every system: a balance between resources and activity, and activity and results. This optimum balance is known as productivity.

Productivity
We can see this balance between effectiveness and efficiency in the illustration below. System (A), which is neither effective nor efficient, cannot be said to be productive. System (B), while very effective, cannot be said to be productive, because it is not efficient. System (C), while efficient, is not effective. Therefore it cannot be considered productive. Only system (D), which is both effective and efficient, can be said to be productive. In application, it is possible to set performance standards for any system in terms of effectiveness and efficiency, and to determine corrective courses of action when evaluative data is generated and graphed. 

Applying the Model to Improving the Productivity of Training Systems
If we are to measure and improve the productivity of a training system, we must focus our attention on two distinct measures. The first is the quality of the instruction, i.e., what should be taught? And the second is the quantity of resources needed to deliver the instruction, i.e., by what means should the curriculum be taught? 

Let me expand on this concept. Instructional quality is the major concern of curriculum development. Curriculum development answers the question, “What should be taught?” It is during the development of the instructional curriculum that a performance standard for the curriculum is established and program content is identified. Instructional quality is often measured by how well participants are able to predictably achieve the objectives of the program. This data is most often gained through traditional testing procedures. For example, a particular curriculum may produce results, wherein an average participant achieves a 95% level of competency on the material presented. 

Instructional quantity, on the other hand, falls into the domain of  instructional strategy, which is the process of determining and selecting the most efficient method and media for delivering a program of instruction (curriculum). The goal of instructional strategy is to answer the question, “By what means should the curriculum be taught?” Again, a performance standard must be established by which to measure this dimension. Common standards are money, time, instructional staff, equipment required, i.e., instructional resources. 

In order to determine the productivity of an instructional system, we must consider both the results produced by the curriculum and the instructional resources required to deliver the curriculum. Keep in mind that productivity is a ratio or composite measure of both the effectiveness and efficiency of a system. 

For example, let’s say that we have an instructional system in which 95% of the participants achieve competency using a media/method to deliver the training which has a per participant cost of $125. 

Taking Corrective Action
The power of this evaluation strategy is that it enables trainers and instructional developers to identify where they should take corrective action. Using the illustration below, it can be seen that to improve our instructional system, attention should be focused upon decreasing the cost of delivering the training. 

In other situations, given different data, trainers and designers may want to improve their curriculums. Which brings us to the following. 

Factors that Influence the Effectiveness of an Instructional System (Curriculum Development)
- Needs Assessments 
- Assessment of Learners 
- Analysis of Work Settings 
- Job, Task, or Content Analysis 
- Statements of Performance Objectives 
- Performance Measurements (test items) 
- Sequence Performance Objectives 

Factors that Influence the Efficiency of an Instructional System (Instructional Strategy)
- Instructional Techniques 
- Designs for Instructional Materials 
- Instructional Resources, i.e., money, time, instructional staff, or required equipment. 

Comparing Two or More Instructional Systems
Trainers and instructional developers are often asked to compare and contrast two or more instructional systems and to make a recommendation. Using the Productivity Model, this becomes a simple task as illustrated below. 

Let us say that we have two instructional systems that deliver the same information and skills. 85% of the trainees, who use Instructional System A, achieve competency for an average delivery cost of $100 per participant. However, 95% of the trainees who use Instructional System B achieve competency using media and methods which have a per participant cost of $125. Now then, which is the most productive instructional system of the two being considered? To determine this, we must graph out the two instructional systems. 
 
 

It can be seen in the above graph, that Instructional System B falls higher and closer to the line we call the "Slope of Productivity," and therefore is the more productive choice. 

A Grounded Model
The concepts and relationships detailed in this article are based upon established principles from a variety of disciplines. 

The basic elements of a system: resources, activity and results, are described throughout the literature on systems theory and are sometimes referred to simply as an input/output model. 

Accountants and financial people, who often speak of return on investment, have known the relationship between resources and results: (ROI). ROI is where returns are results and investments are costs, and in this way, are able to determine the health of a business enterprise (activity). 

Physicists and engineers recognized long ago that in order to describe the performance of various phenomena and systems, they had to be described in terms of dynamic relationships between two variables. For example, miles per gallon or feet per second. 

Thomas Gilbert, a founding father of the human performance field, developed the First Leisurely Theorem, which says that worth is equal to value divided by costs. In other words, activity adds value to resources (cost) and results in something of greater worth. 

Peter Drucker pointed out in the 1970s that effectiveness was doing right things, while efficiency was doing things right. In the field of education, these concepts were expanded upon by Ivor Davies and applied to decision-making concerning instructional methods. 

_______________________________ 
References

Davies, Ivor K. (1981). Instructional Techniques. New York: McGraw-Hill Book Company. 

Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers. 

Gilbert, Thomas F. (1978). Human Competence. New York: McGraw-Hill Book Company. 

Kirkpatrick, Donald (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc. 

Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods, 3rd Edition. Houston: Gulf Publishing Company. 

Rothwell, William J. and Kazanas, H. C. (1998). Mastering the Instructional Design Process, 2nd Edition. San Francisco: Jossey-Bass Publishers.