Kirkpatrick's Four-Level Training Evaluation Model

Level 1: Reaction

This level measures how your trainees (the people being trained), reacted to the
training. Obviously, you want them to feel that the training was a valuable
experience, and you want them to feel good about the instructor, the topic, the
material, its presentation, and the venue.

It's important to measure reaction, because it helps you understand how well the
training was received by your audience. It also helps you improve the training for
future trainees, including identifying important areas or topics that are missing from
the training.

Level 2: Learning

At level 2, you measure what your trainees have learned. How much has their
knowledge increased as a result of the training?

When you planned the training session, you hopefully started with a list of specific
learning objectives: these should be the starting point for your measurement. Keep in
mind that you can measure learning in different ways depending on these objectives,
and depending on whether you're interested in changes to knowledge, skills, or
attitude.

It's important to measure this, because knowing what your trainees are learning and
what they aren't will help you improve future training.

Level 3: Behavior

At this level, you evaluate how far your trainees have changed their behavior, based
on the training they received. Specifically, this looks at how trainees apply the
information.
It's important to realize that behavior can only change if conditions are favorable. For
instance, imagine you've skipped measurement at the first two Kirkpatrick levels and,
when looking at your group's behavior, you determine that no behavior change has
taken place. Therefore, you assume that your trainees haven't learned anything and
that the training was ineffective.

However, just because behavior hasn't changed, it doesn't mean that trainees
haven't learned anything. Perhaps their boss won't let them apply new knowledge.
Or, maybe they've learned everything you taught, but they have no desire to apply
the knowledge themselves.
Level 4: Results

At this level, you analyze the final results of your training. This includes outcomes
that you or your organization have determined to be good for business, good for the
employees, or good for the bottom line.

Level 2: Learning

At level 2, you measure what your trainees have learned. How much has their
knowledge increased as a result of the training?

When you planned the training session, you hopefully started with a list of specific
learning objectives: these should be the starting point for your measurement. Keep in
mind that you can measure learning in different ways depending on these objectives,
and depending on whether you're interested in changes to knowledge, skills, or
attitude.

It's important to measure this, because knowing what your trainees are learning and
what they aren't will help you improve future training.

Level 3: Behavior

At this level, you evaluate how far your trainees have changed their behavior, based
on the training they received. Specifically, this looks at how trainees apply the
information.
It's important to realize that behavior can only change if conditions are favorable. For
instance, imagine you've skipped measurement at the first two Kirkpatrick levels and,
when looking at your group's behavior, you determine that no behavior change has
taken place. Therefore, you assume that your trainees haven't learned anything and
that the training was ineffective.

However, just because behavior hasn't changed, it doesn't mean that trainees
haven't learned anything. Perhaps their boss won't let them apply new knowledge.
Or, maybe they've learned everything you taught, but they have no desire to apply
the knowledge themselves.

Level 4: Results

At this level, you analyze the final results of your training. This includes outcomes
that you or your organization have determined to be good for business, good for the
employees, or good for the bottom line.
Kirkpatrick Plus

Articulated by Kaufman, Keller, and Watkins (“Kaufman”) (1995), this

evaluation framework connects performance to expectations. Kaufman
proposes 5 levels of evaluation

Level 1: Resources and processes

•Level 1 is actually divided into two levels, 1a and 1b.
•Level 1a focuses the evaluation lens on inputs, such as the availability and quality
of materials needed to support a learning effort.
•Level 1b considers processes. What’s their quality? Are they efficient? Are learners
satisfied with them?

•Compared to Kirkpatrick’s Level 1 (Reaction), Kaufman’s Level 1 focuses not only

on learner satisfaction, but on the organizational factors that can impact learner
satisfaction.

Level 2: Acquisition
•This level is focused on individual and small group payoffs—what Kaufman
calls “micro” benefits. Are the objectives or desired outcomes of the learning
intervention met? It’s pretty analogous to Kirkpatrick’s Level 2 evaluation (Learning),
but Kaufman notes that the learning intervention may not necessarily be training.

Level 3: Application
•This is still a micro analysis, examining individual and small group impacts. The
relevant inquiry here is whether newly acquired knowledge and skills are being
applied on the job. Level 3 also is quite similar to Kirkpatrick’s Level 3
(Behavior/Performance).

level4: Organizational payoffs

•Here, the analysis examines macro benefits. What are the benefits from an
organizational standpoint? Level 4 is analogous to Kirkpatrick’s Level 4 (Results).

Level 5: Societal contributions

•Kaufman considers this a mega analysis. How is the organization contributing to its
clients and society? Is it responsive to client/societal needs?
•Issues of health, continued profits, pollution, safety, and well-being are central [in
this level]. The basis for mega-level concerns is an ideal vision, which is a
measurable statement of the kind of world required for the health, safety, and well
being of tomorrow’s children.
•Level 5 has no analog in Kirkpatrick’s Evaluation Model.
CIRO APPROACH

•Developed by War, Bird, and Racham, is a unique way to classify evaluation

process.
•Four general categories of evaluation are described, which form the letters CIRO
1. Context evaluation
2. Input evaluation
3. Reaction evaluation
4. Outcome evaluation

context evaluation
Context evaluation involves
•collecting information about a performance deficiency,
•assessing the information to establish training needs and,
•on the basis of those findings, setting objectives.
Context evaluation involves obtaining and using information about the current
operational situation (or context) to determine training needs and
objectives.

Three types of objectives may be evaluated

1.Ultimate objectives (the particular deficiency in the organization that the

programme will eliminate or overcome)
2.Intermediate objectives (the changes in employees’ work behavior that will be
necessary for the ultimate objectives to be attained)
3.Immediate objectives (the new knowledge, skills, or attitudes that employees
must acquire to change their behavior and reach the intermediate objective)

Input Evaluation involves

•obtaining and using information about possible training resources to choose
between alternative inputs to training.
•It involves analysing the resources available (both internal and external) and
determining how they can be deployed so that there is a maximum chance of
achieving the desired objectives.

Input evaluation refers to the process of collecting evidence and using it to decide on
the training methods
evaluation involves
•obtaining and using information about participants’ reactions to improve the HRD
process.
•The distinguishing feature of this type of evaluation is that it relies on the subjective
input of the participants.
•Their views can prove extremely helpful when collected and used in a systematic
manner.

Outcome Evaluation
• involves obtaining and using information about the results or outcomes of training,
and is usually regarded as the most important part of evaluation.
• If outcome evaluation is to be successful, it requires careful preparation before the
programme begins.

Stages of outcome evaluation-

1.Defining trend objectives
2.Selecting or constructing some measures of those objectives
3.Making the measurements at the appropriate time
4.Assessing the results and using them to improve later programmes.

The Philip’s Five Level ROI Framework

1.Reaction and planned action

Programme participants’ satisfaction is measured, along with a listing of how they
plan to apply what they have learned.
This level of evaluation is important as a participant satisfaction measure, a
favorable reaction does not ensure that participants have learned new skills or
knowledge.

2. Learning

Measurements focus on what participants learned during the programme using

tests, skill practices, role plays, simulations, group evaluations, and
other assessment tools.
A learning check is helpful to ensure that participants have absorbed the material
and know how to use it.

3. Job application

•A variety of follow up methods are used to determine if participants apply what

they learned on the job. Level 3 evaluations are important to gauge the success of
the programme’s application, it still does not guarantee that there will be a
positive impact in the organization.
4. Business results

•The measurement focuses on actual results programme participants achieve as

they successfully apply the programme material. Typical level 4 measures
include output, quality, costs, time, and customer satisfaction.

5. Return on Investment

Comparing the monetary benefits from the programme with its costs. It is
usually presented as a percent or cost/benefit ratio.
Very few organizations actually conduct evaluations at ROI level, perhaps because
ROI evaluation is often characterised as a difficult and expensive process

Steps to calculate ROI

1.Identify outcome(s) (e.g., quality, accidents)
2Place a value on the outcome(s)
Determine the change in performance after eliminating other potential influences
on training results.
Obtain an annual amount of benefits (operational results) from training by
comparing results after training to results before training.

Determine training costs (direct costs + indirect costs + development costs +

overhead costs + compensation for trainees)
Calculate the total savings by subtracting the training costs from benefits
(operational results)
Calculate the ROI by dividing benefits (operational results) by costs.
The ROI gives you an estimate of the rupee return expected from each rupee
invested in training.

Outcomes Used in Evaluating Training Programs:

Cognitive Outcomes
Determine the degree to which trainees are familiar with the principles, facts,
techniques, procedures, or processes emphasized in the training program.
Measure what knowledge trainees learned in the program.
Skill-Based Outcomes
Assess the level of technical or motor skills.
Include acquisition or learning of skills and use of skills on the job.

Affective Outcomes
Include attitudes and motivation.
Trainees’ perceptions of the program including the facilities, trainers, and content.
Results
Determine the training program’s payoff for the company.
•Return on Investment (ROI)
–Comparing the training’s monetary benefits with the cost of the training.
•Direct costs
•Indirect costs
•Benefits

Good training outcomes need to be:

•Relevant
•Reliable
•Discriminate
•Practical

Importance of Training Cost Information

To understand total expenditures for training, including direct and indirect costs.
To compare costs of alternative training programs.
To evaluate the proportion of money spent on training development,
administration, and evaluation as well as to compare money spent on training for
different groups of employees.
To control costs.