Model Risk Management

2016 CAS ERM Seminar

October 6, 2016

Hsiu-Mei Chang, FCAS

AIG – Risk Director, ERM/ Model Risk Management
[email protected]
 Disclaimer

The information, opinions, and recommendations contained in this

presentation are my own and do not necessarily reflect the policies,
procedures, or opinions of AIG.

2
Discussion Points

1. A comprehensive model management framework

2. Model governance
3. Model risk measurement
4. Analytic asset management

3
A Comprehensive Model Management Framework
Three phases of evolution
Model Governance
Policy to define roles & responsibilities

Accurate model inventory

Model development/implementation

Use/Implementation/Change controls Model Risk Management

Model validation Model risks measured at the model level

Model network fully diagrammed

Validation components: Model risk aggregation

Data/Assumptions Model risk reporting infrastructure Analytics Asset Management

Correct application of theories Model returns measured

Conceptual soundness Model risk/return calculated

Appropriate for business purpose Model risk measurements: Strategic analytic resource allocation

Benchmark/Challenger models Likelihood of model failure

Estimates of adverse financial impacts

4
Model Governance
Model Definition & Components

SR 11-7a / AAA Modeling 3rd Exposure Draft

Model Inputs - Data and assumptions; may be partially or

wholly qualitative, or based on expert judgment

Model Processing - Transform inputs into estimates or

forecasts of unknown quantifies

Model Outputs - Translate estimates into

business information
Model Use Examples
1. Asset/Liability valuation
• Financial reporting
• Budget/Business Planning
• Baseline scenario analysis - systemic insurance risk drivers (e.g., underwriting
cycles, mortality, morbidity, and catastrophe risks), macroeconomic risk drivers
(e.g., Interest rate, currency, commodity price risk)

2. Products pricing
• Overall profitability
• Effective risk segmentation
• Marketing strategies, product (re)designs

3. Capital management
• Cost effective financing and risk transfer/hedging strategies
• Aggregation/concentration risk mitigation
• Stress scenario analysis – liquidity risk management
• Portfolio optimization reflecting efficient capital use/allocation

6
 Model Governance
3 Lines of Defense
MRM governed by Operational Risk Committee /
Board of Directors
 Assess and provide assurance of the overall
effectiveness of MRM framework. 3rd Line
 Document/Report breaches to Audit Committee
of Board of Directors. of Defense
 Test compliance of 1st and 2nd lines MRM
controls. Assurance
 Define the model governance framework.
 Provide effective challenge to the 1st line through
2nd Line of Defense critical reviews of end-to-end model
development/implementation and model
Risk Control / outputs/uses.
 Identify, assess, report, and monitor key model
Compliance - ERM risk events.
Accurate & complete model inventory
Disciplined model development,
implementation, and use 1st Line of Defense
Detailed model documentation
Key metrics to identify, asses, control,
monitor specific model risks.
Risk Owners - Business/
Corporate Functions
Model policy & procedures set forth overall
model governance & risk control framework
Model Risk Management
Model Risk & Causes
1. Model risk - Potential adverse consequences from
decisions based on incorrect models or misuse of model
outputs.
2. Causes of model risk
• Intrinsic – data deficiencies, estimation uncertainty,
complexity of model process, business applications, new
models, inadequate testing
• Extrinsic – model implementation/use controls, systematic
risk drivers (e.g., (e.g., uncertainty in volatility / correlation,
unexpected movements in interest rates).

8
 Model Risk Management
Costly Model Risk Events
Examples of costly model errors Other costly model errors with Model tie-ins
 Model-related errors  Operational errors
 Bank of America (2014) – data/process  S&P and Moody’s (2008) – errors in models for
error causes $4B reduction in reported rating complex debt products. Huge
capital reputational damage
 London Whale (2012) – models error  Knight Capital Group (2012) – trading software
caused $5.8B of trading losses malfunction led to more than $450M losses
 Banamex (2002) Modeling teams destroy  Goldman Sachs (2013) – software glitch caused
approximately 5 years worth of default erroneous flood of stock option orders, creating
data due to faulty data processing. significant trading losses
Computer literature suggests that the
value of 100 megabytes of data is valued at  Basic model errors
approximately $1 million,  Long Term Capital Management (1998) – over
 Between 2001 and 2012 SEC public reliance on short term history to calibrate models,
registrants announced over 12,000 use of VaR. Resulted in bankruptcy
financial restatements, most due to data  2008-2009 financial crisis – CDO default models
processing and/or model errors ignored dependence on rising national housing
prices

The revenue loss from other undiscovered and unreported

models deficiencies cannot be estimated, but must be huge 9
Model Risk Management
Model Risk Quantification - Challenges
1. All risk measurement is hard
2. Model “failure” criteria hard to fully define
3. Apples & oranges problems
4. Direct and indirect effects

But we can take inspiration from some (unlikely) heroes:

• Simon Kuznets – inventor of GDP
• Frank Knight – “If you can’t measure it, measure it anyway”, Economic Freedom;
Toward a Theory of Measurement, Walter Block, 1991

Practitioners need to maintain an inventor/entrepreneurial

attitude. Read Frank Knight’s “Risk, Uncertainty, & Profit”.
10
Model Risk Management
Model Risk Quantification - Challenges

1. Risk is a psychic concept, i.e. it is “perceived”

2. Technical risk analytics requires assumptions about underlying
preferences – typically expressed through a utility function.
Such analysis is usually used to:
• Rationalize behavior we observe
• Provide guidance/control over our own behavior
3. The theoretical foundation for the existence of utility
functions is the ability of the agent to rank order preferences
over a choice set

Thus, we do not necessarily need utility functions to create

an institutional model risk framework – but we do need
preference ordering
11
Model Risk Management
Model Risk Quantification - Framework

1. Enumerate bad outcomes

2. Identify preference rank ordering
3. Associate models with bad outcomes
4. Enumerate modes of failure by model type
5. Associate failure modes with bad outcome likelihood

Risk must be based on somebody’s preferences

12
Model Risk Management
Model Risk Quantification - Framework

1. Enumerate bad outcomes

• Any model failure that could impact revenue, profitability, market share,
stock price, reputation, or survival
2. Identify preference rank ordering
• No ranking is necessary, the only bad outcome is a negative impact on stock
price
3. Associate models with bad outcome potential
• For different model classes, how likely are failures to affect stock prices?
5. Associate failure modes with bad outcome likelihood
• For each model class, how likely are different failure modes to affect stock
prices?

With enough data such a framework may be feasible, but it still

must reflect somebody’s preferences
13
Model Risk Management
Model Risk Quantification - Framework

1. Enumerate bad outcomes

Management/BU
• Losses (of different types), revenue drag, reputational damage,

developed
regulatory censure, etc.
2. Identify preference rank ordering, e.g.
• Don’t fail CCAR
• Prevent headline “OpRisk” losses
• Enhance margins
• Additional criteria
3. Associate models with bad outcome potential

Risk Analytics
developed
4. Enumerate modes of failure for model types
5. Associate failure modes with bad outcome likelihood

These components, along with their probability measures

and weightings comprise the framework
14
Model Risk Management
Model Risk Quantification - Framework

Model Risk Score

Model Quality Score

Intrinsic Model Score (Risk inherent in type of

model; i.e., related to model complexity, quality
of data, etc.) Risk
Exposure Mitigation
Index
(Quality
Quality of Data of
Model
Used to Build
Specification
Implementation Use Controls)
Model

To harvest risk component data from the validation process

requires that process to be highly structured 15
Model Risk Management
Model Risk Quantification - Limitations
1. Model-to-model effects
• Risk propagation (amplification, neutral transmission, or mitigation)
within a system

2. Exposure attribution
3. Weak link to financial metrics
4. Redundant analyses/findings
5. Poor subject matter expertise matching
• All validators need to be data quality experts?

All of these issues are significantly ameliorated by elevating the

unit of observation to the model stream level
16
Analytic Asset Management
Why is this important?

1. Profitability and market share (and ultimately firm survival)

will depend critically on it
2. Regulatory expectations (requirements) in this area continue
to grow*
3. They are essential for a comprehensive and integrated model
management framework

“Model risk should be managed like other types of risk. Banks should identify the sources of
risk and assess the magnitude… Banks should consider risk from individual models and in
the aggregate.”,
SR Letter 11-7 Model Risk, page 4.

*In rare but actual cases, failure to meet regulatory expectations and survival can become
intertwined.
17
Analytic Asset Management
What is a model stream?
1. A group of models and their infrastructure related by
• Function
• Dependence (nesting)
• Common data sources
• Common platform
2. The stream includes all movements of data and
calculated values
3. It includes data transfer/processing/transformation
components as well as models
4. It is wing-to-wing: data sources to final use/reporting

Risk measurement at the stream level can directly embed data

quality risks and model risks adjusted for interdependencies
18
Analytic Asset Management
Considerations of a model stream
1. Product outlook
• Core/non-core, growth/stable/shrinking
• Profitability, competitive positions
• Performance volatility
• Product evolution (dynamism, segmentation)
2. Tactical objectives
• Improve risk segmentation, predictive accuracy
• improve implementation infrastructure - more controlled production application,
ease of use, more automated data capture
• Interconnectivity of related models
3. Economic assessments
• Known deficiencies
• Key costs and effected margins
• Tail loss avoidance
• Product differentiation, pricing power, demand elasticity, client services
• Potential impact – risk/reward trade offs, combined ratio effect, etc.

Strategic and tactical action based on this information is model

risk management
19
Analytic Asset Management
A Diagram View of an Illustrative Stream

20
Analytic Asset Management
A Strategic View of the Overall Information Processing Complex
1. Model based view
• Wing-to-wing independent validation (data,
performance, controllership, technology)
• Risk score (based on comprehensive model risk Basic underlying
analysis
assessment framework)
2. Stream based view
• Assessment based on use/scope/corporate function
Supports the
• Clear executive ownership development of a
• Includes an appropriate measurable definition of model strategy
exposure alternatives
3. Meta view
• Explicit mapping of all system components: data,
Enables the
applications, models, reports & other uses implementation of
• Typically will lie between “model” and “block” based the strategy
views
• Assessment throws off - aggregated model risk measure,
risk-based data quality measure, explicit tactical
remediation plans
21
Analytic Asset Management
Concept of an Integrated Objective Environment (IOE)

1. Models are typically embedded in systems/processes that

include data sources, inter-related models, platforms, and
other model-delivery systems – they all contribute to risk and
to return
2. Effective model validation requires some consideration of
this broader context/infrastructure anyway – putting
structure on this part of the process will increase efficiency
3. Business strategic planning to enhance analytic capabilities is
typically done at the stream level – this planning is also
critical contextual input for the validators
4. Model risk measures aggregated to the stream level will be
more meaningful and more actionable
22
Analytic Asset Management
IOE Framework
1. Each business line has its own infrastructure, sometimes linked, but not explicit or visible.
2. Development of calculations are soloed and independently managed.
3. Data & calc lineages are not easily determined.
4. Analytic infrastructures require forensic analysis to determine components and assess
5. controllership, performance gaps and outputs.

Information Assets Analytical Infrastructures Business Operations

CALC Pricing

Reserving/ Claims
CALC
Management

Capital
CALC
Management

CALC Investing

Models

End User Tools (non-model)

CALC Compliance
Model Risk
Data Governance, Standards, Quality & Risk
Database Rationalization

2
3
Analytic Asset Management
IOE Framework - The Analytics “Supply Chain and Factory”
1. Analytical linkages established and maintained data, models, platforms, end uses.
2. Makes the infrastructure “streams” visible with insight into cost, controls, and profitability.
3. Contains its own embedded analytic & reporting capabilities for management.
4. Surgical approach to scale and to extension.

Information Assets Analytical Infrastructure Business Operations

Pricing
CALC

CALC Reserving/
Claims Mgmt

Integrated Object Environment

CALC
Capital
Management

CALC

CALC Investing

Data Architecture “Lake”

Data Lifecycle Management Compliance

Data Rationalization

24
25