A Structured Approach to

Building Predictive
Key Risk Indicators
by
Aravind Immaneni, Chris Mastro
and Michael Haubenstock

L
eading risk indicators with good predictive capabilities are criti-
cal to the successful management of enterprise risk. This article
describes how a process that incorporates some Six Sigma meth-
ods for developing and using key risk indicators was used at Capital One.
The Role of KRIs in ERM most important measures may suf- divided into four different cate-
Key risk indicators (KRIs) play fice as a risk profile update to man- gories: coincident indicators, causal
a critical role in any risk manage- agement. However, this is easier said indicators, control effectiveness indica-
ment framework. Tools for monitor- than done, and one of the current tors, and volume indicators.
ing controls, risk drivers, and expo- challenges of operational risk man- • Coincident indicators can be
sures, they can provide insights into agement is how to structure senior thought of as a proxy measure
potential risk events. For example, management reporting to be as use- of a loss event and can include
where self-assessments are used ful as possible. Especially where internal error metrics or near
periodically to identify risks and KRIs are concerned, most measures misses. An example of a coinci-
controls, KRIs can monitor them in are business or process specific and dent indicator in a payment-
the intervening intervals. KRIs also difficult to aggregate. Even meas- processing operation may be
can provide a means to express risk ures that are common to many areas number of misapplied pay-
appetite. KRIs often serve their of an organization, such as turnover, ments identified through inter-
most practical purpose in conjunc- training, and other human resources nal quality assurance sampling.
tion with a system of thresholds; measures, may track risk well in rel- • Causal indicators are metrics
when a KRI breaches its associated atively small business units but that are aligned with root caus-
threshold, it triggers a review, esca- track it very poorly when measured es of the risk event, such as
lation, or management action. at the enterprise level. system down time or number
As a rule, KRIs should be moni- of late purchase orders.
tored closer to the “front” than in Types of Key Risk Indicators • Control effectiveness indicators
the higher reaches of management. Key risk indicators encompass provide ongoing monitoring of
In the absence of any major risk different types of metrics. For the the performance of controls.
changes, monthly summaries of the purposes of this article, KRIs are Measures may include control

© 2004 by RMA. Aravind Immaneni is a senior process redesign specialist, Chris Mastro is group manager of process
engineering, and Michael Haubenstock is director, Operational Risk Management, at Capital One, Richmond, Virginia.

42 Operational Risk: A Special Edition of The RMA Journal May 2004

 A Structured Approach to
Building Key Risk Indicators

effectiveness, such as percent Key Risk Indicator Methodology sionally and the information given
of supplier base using encrypt- The successful identification is not accurate.
ed data transfer, or bypassed and application of effective KRIs
controls, such as dollars spent require a structured approach. We 1. Identify existing metrics.
with nonapproved suppliers. used a six-step process that incor- Developing key risk indicators
• Volume indicators (sometimes porates various Six Sigma tools: often starts with a risk assessment.
called inherent risk indicators) fre- 1. Identify existing metrics. Risk events in a business are iden-
quently are tracked as key per- 2. Assess gaps. tified, assessed, and catalogued
formance indicators; however, 3. Improve metrics. along with their associated controls
they also can serve as a KRI. As 4. Validate and determine trigger and an analysis of their root causes.
volume indicators (e.g., number levels. Quite sensibly, businesses some-
of online account applications) 5. Design dashboard. times then opt to focus their KRI
change, they can increase the 6. Establish control plan. development on the events with
likelihood and/or impact of an This is a process that could be high inherent or residual risk.
associated risk event, such as applied to develop, validate, and Then the first step in the KRI
fraud losses. Volume indicators implement KRIs across any busi- process is to identify existing met-
are often associated with multi- ness, but here we illustrate it with rics for each high-risk potential
ple risk types in a process or an example from a call-center event. Interviewing subject matter
business unit. operation. The risk here is that a experts (SMEs) in the business typi-
customer is not handled profes- cally uncovers five to 10 existing

Figure 1
Gap Assessment Template
Dept: Call-Center Operations
Risk Event: Customer contact not completed in an accurate or professional manner
Metric 1: Customer Satisfaction Index
Metric 2: Associate Attrition
Metric 3: Average Handling Time (AHT)
Metric 4: Transfer Rate

ric 1

ric 2

ric 4
ric 3
Met

Met

Met
Met
Q# Dimension Assessment Question Low Values (rating 1) Medium Values (rating 3) High Values (rating 5)
–Frequency is clearly defined.
–Frequency is clearly defined. –Frequency is at least daily or lowest
Is the frequency of measurement –Frequency is not clear –Frequency is at least weekly. required for the specific metric.
adequate to flag a risk event –Frequency is monthly or less –It is not clear whether the frequency –Frequency is low enough to identify
1 Frequency prior to occurrence? frequent. is sufficient to prevent the risk event.
and prevent potential risk events. 3 2 5 4
Do trigger levels exist and if so Trigger levels have not been Trigger levels have been identified Trigger levels exist and they are
2 Trigger Levels are they analytically sound? identified. but are not analytically sound. sound. 3 1 5 1
Are there clear escalation criteria Escalation criteria exist but no clear Clear escalation criteria with responsible
3 Escalation Criteria tied to the trigger levels? No clear escalation criteria. owner or documentation. owner with documentation. 1 1 5 1
The metric is tied to a control or rootThe metric is tied to one of the major
Is the metric a leading or a Is the metric tied to the risk cause, but not leading enough to root causes and has sufficient lead to
4 Leading/Lagging indicator? event occurrence? prevent a risk event. prevent the risk event from occurring. 1 3 4 2
Some ownership, but changes from Clear ownership for creation and analysis
Is there a clear owner for the No clear owner. Metric is more ad- time to time or is not a clearly of the metric as part of an established
5 Ownership creation and analysis of the metric? hoc in nature. established job function. job function. 5 5 5 3
Past data available but has not been Historical data available and has been
Does historical data exist on New or recently created metric tracked. It can be retrieved with tracked as a metric for significant period
6 Historical Data the metric? with no past data. some effort. of time. 4 5 5 3
Data reliability and accuracy can Reliable data collection process is
not be ascertained (or are unknown). in place and is not subjective. Reliable, repeatable data collection
How accurate and reliable is Process/procedure for data Measurement error is high procedures. Measurement error is low
7 Data Accuracy the data? collection is subjective in nature. (inadequate) or unknown. (adequate) and it is well known. 1 5 5 4
Average Score (Q1 to Q7) 2.6 3.1 4.9 2.6
Each metric is evaluated along seven dimensions to identify gaps. Metrics that score higher are better candidates for serving as a Key Risk Indicator.
In this example, Metric 3 (Average Handling Time) has the highest overall rating against the dimensions.

43
 A Structured Approach to
Building Key Risk Indicators

metrics as potential KRIs and you whether the metrics in the current the drivers of the risk event are
should expect at least one or two of form would be effective as a KRI; scored using the 0-1-3-9 scoring cri-
each type—coincident, causal, con- typically, a composite score of 4.0 teria. The driver rating is a binary
trol, and volume. or higher is desired. An example of rating of yes/no, with a “yes” only
a KRI gap assessment for a call if the risk-event driver scores a 9 on
2. Assess gaps. Once that center is shown in Figure 1. any one of the metrics. The metric
inventory is complete, the next step The second tool, the design rating is the weighted average of
is to evaluate the suitability and matrix, is a variant of the quality the ratings that the metric was
effectiveness of each of these exist- function deployment commonly scored for each driver. An example
ing metrics as leading risk indica- used in Six Sigma exercises. The of a design matrix for existing met-
tors. Two tools are used: the gap drivers of the risk event are listed rics is shown in Figure 2.
assessment and the design matrix. on the left by row, and the existing With these tools, we can assess
The gap assessment tool has metrics are listed along the top by where existing business metrics
seven dimensions along which column. Risk-event drivers are the fall short in terms of suitability and
each metric is rated on a scale of 1 root causes that allowed the risk effectiveness for use as key risk
to 5. The dimensions are frequen- event to occur, such as data-entry indicators. They point the way to
cy of measurement, trigger levels, error, incomplete communication, where we need to make improve-
escalation criteria, leading/lagging, or an associate not following proce- ments to existing metrics and find
metric ownership, historical data dures. Each risk-event driver is additional ones.
availability, and data source accura- given an importance weighting,
cy. For each dimension, a clear dis- which reflects the percent contri- 3. Improve metrics. First, we
tinction is made between what bution of this driver to the likeli- focus on metrics that scored a 9 in
constitutes a weak, moderate, or hood that the risk event in ques- the design matrix (particularly
strong rating. Evaluating metrics tion would occur. The relationship along multiple risk-event drivers)
along these dimensions identifies between each of these metrics and but that have a low score in the gap
assessment. When that
low score is associated
Figure 2
Design Matrix Template with something that
Dept: Call-Center Operations can be fixed, such as
Risk Event: Customer contact not completed in an accurate or professional manner insufficient frequency,
Date: 6/2/2003 inadequate trigger lev-
Assessor Name: John Doe III
Existing/Potential KRIs els, or an absence of
Customer Satisfaction

Average Handle Time

established escalation
Associate Attrition

Scoring Criteria
Driver Rating
Transfer Rate

criteria, improvements
Weighting

0-—No relationship
(AHT)
Index

1—Weak relationship along these dimensions

3—Moderate relationship can transform an exist-
9—Strong relationship
ing metric into an effec-
Risk-Event Drivers
tive KRI.
Associate not knowledgeable on company policy/procedure 10% 3 9 3 3 Y
Next, we look at
Associate has poor communication skills 50% 3 1 3 1 N
metrics that have
Associate misunderstands customer request 10% 3 3 3 1 N
scored high on the gap
Associate cannot explain policies accurately to the customer 25% 3 3 3 3 N
assessment but don’t
Customer call transferred to the wrong queue 5% 1 1 3 9 Y
score a 9 along any risk-
event drivers on the
Metric Rating 100% 2.90 2.50 3.00 2.10
design matrix. We ask
whether there are mod-
A design matrix is used to assess the relationship between each potential KRI and each risk-event ifications to the metric
driver. In this example, no existing metric aligned strongly to the primary risk-event driver (poor
communication skills). that might strengthen

44 Operational Risk: A Special Edition of The RMA Journal May 2004

 A Structured Approach to
Building Key Risk Indicators

Figure 3 In such cases, a risk-

Design Matrix Template event driver can be
Dept: Call-Center Operations used as a proxy. Since
Risk Event: Customer contact not completed in an accurate or professional manner the business usually
Date: 6/2/2003 has a good understand-
Assessor Name: John Doe III
Existing/Potential KRIs ing of the targets and

Customer Satisfaction

Communication Score
trigger levels (or con-

Associate Attrition

Knowledge Score
Scoring Criteria

Driver Rating
trol limits) for the risk-

Weighting
0-—No relationship

Index
1—Weak relationship event drivers, the cor-
3—Moderate relationship relation between the
9—Strong relationship
driver and the risk indi-
Risk-Event Drivers
cator lets us set corre-
Associate not knowledgeable on company policy/procedure 10% 3 9 3 9 Y
sponding targets and
Associate has poor communication skills 50% 3 1 9 3 Y
trigger levels for the
Associate misunderstands customer request 10% 3 3 9 9 Y
KRI metrics. A good
Associate cannot explain policies accurately to the customer 25% 3 1 0 9 Y
example is shown in
Customer call transferred to the wrong queue 5% 1 1 0 0 N
Figure 4, where the
risk-event driver, cus-
Metric Rating 100% 2.90 2.00 5.70 5.55 tomer satisfaction
index (CSI), is plotted
The design matrix also is used to assess new KRIs that are created. In this example, two new metrics
(Communication Score and Knowledge Score) were created that better align with the primary risk- against the new com-
event drivers and have a higher metric rating, i.e., they provide better risk-event coverage. munication score met-
ric. The goals and the
the relationship with at least one of assess the strength of the relation- trigger levels for the CSI are trans-
the risk-event drivers. For exam- ship between the risk-event drivers lated into trigger levels for the
ple, payment-data-entry errors may and the metrics. In many cases KRI.
be a key driver of the risk event these correlations are self-evident. Validation is not necessary for
“payments not processed in a time- For example, a metric that meas- each KRI and risk-event driver.
ly or accurate manner.” An existing ures cycle time (or turnaround time) Ideally, each risk will have one or
metric that measures all data-entry of an evaluation process is strongly two major metrics that need to be
errors across the department may correlated with the risk-event driver validated to ensure that appropriate
have a weak relationship measuring “evaluation is not completed in a trigger levels are set to enable
to payment errors; however, the timely manner.” In such cases the intervention.
driver rating can be increased by validation is not necessary, and trig-
modifying the metric to capture ger levels are set based on business 5. Dashboard design. The
only payment-keying errors. and/or regulatory requirements. In next step is to design dashboard
Before moving into validation, all other cases, especially when new reports on these critical metrics for
the list of is drivers pared down to metrics are created as described in business managers, process owners,
five or fewer KRIs. Each metric in the previous section, these metrics and senior management. A dash-
the design matrix that has no strong should be validated to ensure that board can be useful on a stand-
relationship to any risk-event driver the metric is indeed a predictive alone basis or as part of another
is removed. An example of a design risk indicator. management process, such as a
matrix for the new metrics is shown Ideally, validation will involve monthly business review process.
in Figure 3. a statistical analysis of historical Dashboards typically use graphs
data between the risk event itself and tables to give a concise and
4. Validation and trigger-level and the metric. However, in most comprehensive risk picture, high-
identification. The previous two cases historical data is not avail- lighting KRIs that are above con-
steps used subjective judgment to able, particularly on the risk event. trol-plan trigger levels and report-

45
 A Structured Approach to
Building Key Risk Indicators

Figure 4 version could be a one-page

Regression Plot description of all the key risk indi-
cators associated with that specific
85
risk event in a tabular form. In
80 ⽧
both cases, the control plan should
Customer Satisfaction

75
state the KRI metric, the measure-
70 ⽧ ⽧ ment frequency, a description of
65 ⽧
⽧ ⽧ the measurement system, goals,
60
trigger levels, escalation criteria,
55
and the owner for the escalation
50
criteria. The control plan could be
45
presented as an appendix to the
40
dashboard to bring attention to the
1 1.5 22.5 3 specific actions taken with respect
L3 Target: 2.6
Communication Score Lower Spec Limit: 2.2 to each KRI over the course of the
reporting period. An example of a
control plan and the associated
Trigger Limit: 2.3
escalation criteria for our call-cen-
A regression analysis is used to validate each KRI and to establish appropriate trigger ter example are shown in Figure 6.
levels. For this example, the Communication Score KRI correlates well with overall
Customer Satisfaction.
Challenges in Instituting KRIs at
an Enterprise Level
ing on the actions that have been new owner can come up to speed
A firm-wide initiative for KRIs
taken as a result. An example of a on the procedures quickly and
creates challenges in development,
KRI dashboard for our call-center understand the level of risk the
aggregation, and reporting. The
example is shown in Figure 5. business is willing to accept in
potential applications are enor-
managing such risk.
mous for development of the
6. Control plan and escala- The control plan can be a
measures, touching every business
tion criteria. The purpose of the more detailed one-page description
area in the organization. The
control plan is to ensure that clear of all the actions and accountabili-
approach can be either top-down
escalation criteria and roles for ties around a specific KRI. In such
or bottom-up. A top-down
intervention have been established a case, a separate page would be
approach would look at overall
when a KRI is triggered. This doc- needed for each metric. A simpler
objectives and risks and determine
umentation
enables a Figure 5
process Regression Plot
owner to
Communication Score for Team A Knowledge Score for Team A
follow an
3.0 3.0
agreed, con- UCL=2.967
UCL=2.920
2.9
sistent pro-
Individual Value
Individual Value

2.8 䡲
tocol each 䡲 Target=2.1 䡲 䡲 Target=2.75
2.5 䡲 䡲 2.7
䡲 Mean=2.404
time a KRI 䡲 2.6 䡲 Mean=2.616
is triggered. Trigger
䡲 䡲 Limit=2.3 2.5 䡲 Trigger
䡲 䡲 Limit=2.55
In the event 2.0 2.4
the process LCL=1.841 2.3 LCL=2.311
is transi- 1 2 3 4 5 6 7 1 2 3 4 5 6 7
tioned to a Week Week
different A dashboard is used to display and monitor each Key Risk Indicator. Individual control charts are used to monitor
owner, the the Communication Score and Knowledge Score KRIs from the call-center example.

46 Operational Risk: A Special Edition of The RMA Journal May 2004

 A Structured Approach to
Building Key Risk Indicators

Figure 6
Control Plan

Metric Metric Metric Metric Trigger Escalation Procedure Last

Name Description Measurement Frequency Owner Limit(s) Procedure Owner Update
Communi- A measure of 1=customer impact. 3 calls QA 2.3 Associates receiving <2.3 Site 9/03
cation Score associate’s 2=no customer measured Manager score (rolling biweekly basis) Director
communication impact. weekly per will receive L3 refresher
skills on a call, 3=not heard on call. associate. training and be reassessed.
measured on a If refresher training does not
3-point scale. solve issues consistently for
over 3 months, associate
should be removed from
the role.
Knowledge A measure of 1=customer leaves 3 calls QA 2.55 Associates receiving <2.55 Site 9/03
Score associate’s ability call without resol- measured Manager score (rolling biweekly basis) Director
to explain poli- ution to issue or weekly per will receive K3 refresher train-
cies and proced- question appropri- associate. ing and be reassessed. If re-
ures to the cus- ately addressed. fresher training does not solve
tomer measured 2=customer gets the issue consistently for over
on a 3-point resolution but has to 3 months, the associate
scale on a call ask multiple times. should be removed from
monitored by QA. 3=customer is ser- the role.
viced appropriately
upon first request.

A control plan is used to document each KRI along with associated information such as trigger levels, escalation criteria, and ownership.

the risk indicators that might best ious measures is the next chal- indicators when we first started
measure them. These are then cas- lenge. While at the business area work on our approach. Drawing
caded down the organization. level one can simply pick the most from the Six Sigma tool set, we
Often this is a good approach for significant measures for manage- have developed it and tested it
key performance indicators that ment reporting, at a corporate level with encouraging results. While
can be common across business the number of resulting risks difficult issues remain—such as
units or processes (e.g., human would be overwhelming. There- reporting on disparate KRIs for
resource measures) or for those fore, the reporting process has to senior management—we believe
that can be applied to very similar be either more selective or trans- applying our basic six-step
processes across business areas formed into an index. Selective approach will often lead to more
(e.g., payment processing). Top- reporting might pick a few signifi- effective key risk indicators and
down approaches also are effective cant measures or report only those consequently stronger risk man-
for key performance measures. over limit, for example. Reporting agement. ❐
Alternatively, bottom-up at business area levels is more
approaches start in each business actionable than combining them Haubenstock can be reached by e-mail
area with their specific processes into a single corporate number. An at michael.haubenstock@
and risks. Here KRIs can be very index would combine results from capitalone.com; Immaneni can be
specific but are often different in various different measures and reached at aravind.immaneni@
each area and then are difficult to report on them in aggregate. capitalone.com; and Mastro can be
aggregate centrally. Since the ben- reached at chris.mastro@capitalone.
efit is more for business areas than Conclusions com.
for a central group, the bottom-up While the potential of key risk
approach results in a more effec- indicators has been widely accept-
tive approach for business areas ed for some time, we were not
with unique processes. aware of a structured approach to
Aggregating and reporting var- developing and applying key risk
47