An Important Evaluation Model You Should Know

Donald L.Kirkpatrick’s four levels for evaluating training programs is probably the most frequently used evaluation model. Since the model was first introduced in 1959, Kirkpatrick has continued to hone it and provided numerous case studies to point out effective methods for measuring training success from the time of the workshop through the time when the training’s effect on the organization’s bottom line can be judged.

Kirkpatrick’s four levels are:

Level One: Reaction. In Level One, the focus is on the workshop itself.

Was it enjoyable?

Were its learning objectives relevant—and clearly stated?

Did the facilitator address the needs of the audience?

Was the facilitator knowledgeable?

How confident is the learner that the learning objectives were met?

How confident is the learner that she or he will be able to apply the learning back on the job?

Were the surroundings pleasant?

Was the pace appropriate?

Would the learner send a colleague to this workshop?

Obtaining answers to these questions is essential to measuring the workshop’s effectiveness. The primary time to gather this feedback is during and at the very end of the workshop itself. Waiting until later will not provide as true a picture of the learner’s reaction. Therefore, this data needs to be collected from all participants before they leave the training room (or exit the final session of the e-learning course). But, clearly, even with overwhelmingly positive responses to all of the above questions, there is no real proof that the workshop will have lasting value to the learner.

Level Two: Learning. Level Two goes beyond the learner’s perception of the learning and begins to measure the knowledge and skills actually transferred to the learner by asking the following:

Did the training experience result in an increase in knowledge for the learner?

Did the learner master new skills as a result of the training experience?

Were there any changes in the learner’s attitudes or beliefs as a result of the training?

Notice that each of the questions posed at Level Two are in the form of before/after pictures. In other words, Level Two evaluation will require tests before and after the workshop to obtain an accurate measurement of results. Since this is not always possible or practical, many trainers rely on quizzes, exercises, and mastery tests to demonstrate that learning has occurred. In relying solely on such measures, you are unable to prove that the learning actually resulted from the workshop.

A more serious shortfall is the fact that most of these quizzes and master tests occur within the timeframe of the training and do not take into account the issue of skill and knowledge retention. In an ideal world, you would want pre-tests, tests of learning progress, and tests administered long enough after the workshop to determine how well the learner retained the skills and knowledge acquired. The first time out with your project management training, you may settle for some mastery tools used during the workshop itself and introduce follow-up tests at a later date.

Level Three: Behavior. Level Two can be useful in finding out what skills and knowledge your participants acquired during the workshop. Level Three takes things a step further to find out whether the participants actually apply skills and knowledge on the job by seeking to demonstrate that the workshop had practical value and resulted in changes in the way work is performed after the workshop. Level Three asks: 1) Did the participant apply the knowledge and skills acquired in the workshop on the job? 2) Have there been positive changes in the participant’s attitude since the workshop?

Level One solicited the learner’s evaluation of the workshop. Level Two sought to prove knowledge and skill transfer though a combination of tests and demonstrations involving some kind of interaction with the participant. Level Three seeks feedback within the organization—usually from the participant’s supervisor or from the manager of the project team to which the participant belongs—to find out if the training resulted in positive change. In addition, there may be follow-up self-assessments whereby the participant can report which knowledge and skills have helped make the job easier.

Level Four: Results. At this level, you are looking not just for proof that the participant’s behavior and attitudes have changed but also for proof that these changes have benefited the organization in some way. Level Four asks: 1) Have the participant’s newly acquired knowledge and skills resulted in an overall improvement in job performance? 2) Have these performance improvements translated into increased revenues; reduced costs; improved service; or other tangible, quantifiable benefits to the organization?

Level Four not only relies on feedback from the participant’s supervisor but also has to take into account past performance; the stated goals of the training as defined during the training needs assessment; and perceptions of customers, other departments, the project management office, and even such indirect stakeholders as auditors and regulatory agencies. For this reason, it isn’t surprising that a significantly smaller percentage of courses wind up being evaluated at Level Four. Nevertheless, Kirkpatrick’s model provides a valuable framework for planning and evaluating your project management training program—whether you stop at Level One evaluations or develop a comprehensive system of evaluation. As your program grows, you can expand your use of the framework to include all four levels.

Article added at: 11.09.2006 by Sally Jordan