2011 PMOC Annual Meeting

FEDERAL TRANSIT ADMINISTRATION

Risk Review Process Basics

Michael P. Wetherell, PE - Urban Engineers

David N. Sillars, PE Sillars Consulting
FEDERAL TRANSIT ADMINISTRATION
2011 PMOC Annual Meeting

Background
As with many large, public projects, FTA
experienced a significant amount of project cost
overrun
In this environment of overrun risk, planning a cost
budget has been difficult
The risk review process is intended to:
Inform the FTA about the projects risk

Provide the grantee with recommendations to

strengthen the project
 Uncertainty
Uncertainty is the only certainty there is, and
knowing how to live with insecurity is the only
security.
- John Allen Paulos
The fundamental assumption is the grantee may
decrease the level of uncertainty by studying the
problem, exposing the risks, and taking actions to
mitigate known risks
 Certainty and Uncertainty

No Information Partial Information Complete Information

Unknown unknowns Known unknowns Knowns

Scope of project risk assessment

General Specific
Total Uncertainty Total Certainty
Uncertainty Uncertainty
 Risk Review vs. Risk Assessment
Risk Review
FTA provides a review of the risk and contingency plan
within a Grantees project management plan
This review evaluates the 4 fundamental parts of risk
management, i.e., Identification, Assessment, Response,
Management
Risk Assessment
1 of the 4 parts that quantifies the impact of all risks
FTA treatment of risk assessment depends on the projects
size
 Risk Management Process
Identifying risk situations for which outcomes
are likely to vary in amounts that may
significantly affect the projects goals
Assessing the likelihood and magnitude of
such variance
Responding by developing plans for
minimizing hazard and maximizing gain
Managing the execution of those plans
 Risk Review Benefits

Reviews an agencys technical capacity and

capability to perform the project
Identifies and develops action plans for risky
events
Creates informed assessment of likely cost
and schedule outcomes
Assures plans are in place to protect the
project
 FTA Risk Review

Oversight Procedures (OPs) prescribe fundamentals of the risk

review process
Grantee
Technical
Capacity
(OP-21)

Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)

Monitoring Reports (OP-25)

 Presentation Outline
Project Status Evaluation and Risk Assessment
Project Status Evaluation Procedures
Project Cost and Schedule Reviews
Risk Identification
Risk Categories
Risk Register
Assessment Evaluation and Risk Management
Overview of Beta Model Development/ Interpretation
Overview of Cumulative Cost Risk Curve/ Schedule Risk
Curve
Risk Mitigation
Risk and Contingency Management Plan (RCMP) Basics
 Project Status Evaluation
Technical capacity and capability
Project capacity and scope
Cost estimate
Project schedule

Grantee
Technical
Capacity
(OP-21)

Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)

Monitoring Reports (OP-25)

The project status evaluation is a precursor to the detailed Risk Review

 Risk Review Meetings
The Risk Review process involves 3 primary
review meetings between the grantee, the
PMOC, and FTA before each evaluation
Supplemented by working groups designed to
develop the details of the risk review
Its important each meeting is attended by those
most able to provide seasoned, professional input
 Risk Review Meeting 1

Introduce PMOC team and grantee team

Grantee presents the project to PMOC team
Agency organization and project team plan for staffing
Description of work and reviews over the previous year
Review of the project by discipline
Review of schedule, cost estimate, and risk register, if
available
PMOC presents the risk process to grantee
Tour of alignment, station, and support facility
locations
 Project Status Evaluation

Fundamental project documents must be provided to the

PMOC to complete the project status evaluation
Program documents
Project agreements
Design documentation
Cost estimates, with contingencies exposed
Schedule, with contingencies exposed
If these elements are flawed or incomplete, risk
assessment may be undermined before it even begins
 Technical Capacity and Capability
Evaluates grantee management, organization, and project
definition data
Requires:
Project program information
Interviews of grantees organization
Grantee Project Management Plan (PMP)
Assesses technical capacity and capability to implement
the project in conformance with the Development Strategy
Provides recommendations that should be addressed in
the PMP
 Project Capacity and Scope

Evaluates the projects operating capacity, in terms of line capacity

and ridership
Includes current and projected ridership
Considers phasing of facility
Requires:
Project program, alignment, and ridership information
Capacity calculations (like TCRP-100 ridership capacity worksheets
and station stop worksheets, if used by Grantee, etc.)
Identifies gaps (or requirements risks) in the forecasted
balance between potential ridership and system capacity
 Project Capacity and Scope

Assesses the physical components of the project scope and

how those components relate to their surroundings

Requirements for assessment depend on the projects phase

Compares against project description and initial goals

Requires:

All program and design documents and investigative reports

Site evaluation visits

Scope checklist
 Cost Estimate Review

Review basis for Escalation Factor(s)

Standard Indices
Basis for Assumptions
Review grantees project cost estimate and general
conditions in base year and escalated dollars
Requires grantees detailed cost estimate and back-up,
Standard Cost Category (SCC) workbook, and project
scope review results
Provides a mathematical check of the estimate
May include recommendations for adjustments to the
grantees budget estimate
 Project Schedule Review
Reviews grantees project schedule
Requires grantees schedule, SCC workbook,
and project scope review results
Provides evaluation of schedule
Continues discovery of items that are
uncertain and constitute risks
 Project Status Evaluation

Upon completion of the Project Status

Evaluation, FTA (through reports from the
PMOC) has a thorough understanding of the
project, grantee, cost and time required to
complete the project, and uncertainties to be
resolved
Information revealed will be used by FTA for
programmatic decisions and will form the basis
for identification of risk events
 Risk Identification
Accumulate issues Categorize by SCC code
discovered through the Categorize by risk type:
various reviews and Requirements
identify potential risk Design
events Market
Develop a Risk Register Construction
listing of risk events
 Risk Identification
Risk identification is a collaborative process
Purpose is to objectively discover potential
threats to the projects goals
Starting points for risk event identification:
1. PMOCs listing of issues discovered through the
project status reviews
2. Grantees own risk analysis (if undertaken)
 Risk Review Meeting 2

PMOC summary reviews of technical capacity and

capability, scope, cost, and schedule
Discuss risks identified by PMOC and record
additional risks discovered during the workshop
Enter into discussions with grantees project team
to resolve open questions
Discuss actions required to facilitate the next
stage of quantitative risk assessment
Inform grantee of next steps
 Key Individuals
Project owner
Staff of project owner
Management staff, preferably those who participated in
any earlier risk reviews
Appropriate technical staff (e.g., operations staff)
Senior members of the project team
Including cost, schedule, design experts, etc.
Participating or affected agencies
If risk mitigation measures have been allocated to
contractors, their management and technical staff
should be involved
Risk facilitator
 Risk Factoring

Probability (P) chance of occurrence

Magnitude (M) value
These 2 factors determine the expected value
(E) of a risk event

E = P*M
 Risk Factoring

It is helpful to provide guidance regarding

what constitutes each category of probability
or magnitude assessment
One possible risk factoring:
Assessment Low (1) Medium (2) High (3)
Probability < 10% 10% 50% > 50%
Cost ($) < $1.0 million $1.0 $5.0 million > $10.0 million
Time (months)
< 1 month 1 12 months > 12 months
potential delay
 Risk Register

An initial evaluation of the expected value of each risk is made

Risk Register

Identification Assessment

Cost Schedule
Probability Expected Expected
Magnitude Magnitude
(A) Value Value
(B) (C)
SCC Code
Risk event Risk
description Type 3 = high 3 = high 3 = high
Schedule
2 = medium 2 = medium 2 = medium Cost Score
Score
1 = low 1 = low 1 = low

4
 Risk Review
Risk assessment by PMOC
after risk identification
workshop with Grantee

Grantee
Technical
Capacity
(OP-21)

Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)

Monitoring Reports (OP-25)

 FTA Standard Cost Categories (SCC
Codes)
10 Guideway and Track Elements

20 Stations, Stops, Terminals, Intermodals

30 Support Facilities: Yards, Shops, Administration Buildings

40 Sitework and Special Conditions

50 Systems

60 Right-of-Way, Land, Existing Improvements

70 Vehicles

80 Professional Services

90 Unallocated Contingency

100 Finance Charges

 Risk Assessment Steps

1. Strip the estimate and schedules of all contingencies

2. Summarize the costs and activities
3. Apply ranges of cost and durations to the summaries
4. Calculate the range of cost and schedule for the
project
Cost and Schedule Risk Assessment

1. The schedule and cost estimate are

scrubbed of contingencies and adjusted by
the PMOC
Identify and remove hidden and exposed
contingencies
Remaining schedule and cost estimate are used to
forecast risk on the project
After the first step, the cost and schedule assessment processes differ
 Cost Risk Assessment

2. Summarize the grantees scrubbed and

adjusted estimate into the SCC
SCC Category 10%

SCC 10 Guideway

10.01 Guideway: At-grade exclusive right-of-way 42,501

10.04 Guideway: Aerial Structure 51,900

10.11 Track: Ballasted 45,176

SCC 20 Stations, Stops, Terminals, Intermodals

20.01 At-grade station, stop, shelter, mall, terminal, platform 45,122

20.02 Aerial station, stop, shelter, mall, terminal, platform 46,551

20.06 Automobile parking multi-story structure 215,380

20.07 Elevators, escalators 9,248

 Cost Risk Assessment

3. A range multiplier (Beta) is applied to the

scrubbed and adjusted estimate, on a
category-by-category basis, to yield the range
from highly optimistic (10%) to highly
pessimistic (90%)
The range multiplier varies depending on the
phase of project development
FTA provides recommended values for the Beta
factor
 Cost Risk Assessment

How is the Beta chosen?

Depends on what phase the elements of the project are in: the earlier in
the process, the more unknowns and the higher the risk
4 stages of project completion are considered important; completion of
each phase successively reduces risk

Stage of Project Completion Risk Example

Unknown stakeholder requirements for

Requirements definition Requirements risk
station
Project design Design risk Vertical alignment issues

Construction contracts procurement Market risk Significant spike in cost of steel

Unforeseen conditions, project
Project construction Construction risk
interference
 Cost Risk Assessment
Requirements Risk,
=?

Note: Beta () is a factor

applied to the base cost
2.50
estimate indicating amount
of risk.
Design Risk, =0.50
2.25
Cumulative Risk Beta

2.00
Market Risk, =0.25

1.75
Early, =0.40

1.50
Constr. Risk, =0.70

Mid, =0.15
1.25
Late, =0.15

1.00 Early Mid Late

Design
Concept Bid
Project Phase Construction
 Cost Risk Assessment
Sample Beta assessment
Post
Requirements Design Marketing Construction
SCC Category 10% Construction Total Beta
Beta Beta Beta Beta
Beta

SCC 10 Guideway

Guideway: At-grade exclusive

10.01 42,501 0.00 0.50 0.25 0.70 0.05 2.50
right-of-way

10.04 Guideway: Aerial Structure 51,900 0.00 0.50 0.25 0.70 0.05 2.50

10.11 Track: Ballasted 45,176 0.00 0.50 0.25 0.70 0.05 2.50

Stations, Stops, Terminals,

SCC 20
Intermodals
At-grade station, stop,
20.01 shelter, mall, terminal, 45,122 0.00 0.50 0.25 0.70 0.05 2.50
platform

Aerial Station, stop, shelter,

20.02 46,551 0.00 0.50 0.25 0.70 0.05 2.50
mall, terminal, platform

Automobile parking multi-

20.06 215,380 0.00 0.50 0.25 0.70 0.05 2.50
story structure

20.07 Elevators, escalators 9,248 0.00 0.50 0.25 0.70 0.05 2.50

Note: Total Beta includes starting base of 1.0

 Cost Risk Assessment

4. Categories of the project are then aggregated

probabilistically to yield likely project
outcomes
PMOC will use the Cost Risk Assessment
Workbook to simplify the process of project risk
assessment range modeling
 Cost Risk Assessment

PMOC uses the OP-40 Cost Risk Assessment Workbook

Consists of several parts:

Conditioned
base estimate, PMOC
in standard SCC adjustments to Adjusted YOE
categories raw YOE estimate Betas
Betasassigned
assigned
estimate without totoBetas
Betasassigned
SCC
SCC assigned
contingency to SCC
Categories
to SCC by
Categories
Categories byby
phase Categories by
phase Risk reduction
phase
phase graph
Cost Risk Assessment
 Cost Risk Assessment
Project is forecasted out to each stage of development to identify
potential risk reduction; the results are plotted across the project life-
cycle, as if all mitigation is successful
 Cost Risks

Completion, phase by phase >>>>>>>>

Project Project
Conception Completion
 Schedule Risk Assessment

2. A summary schedule is developed with a

reasonable number of activities
Activity One
Start
10 12 16
Criticality = 63% Finish

Criticality = 37%

Activity Two
8 10 8
 Schedule Risk Assessment
3. The duration range is established based on the risk register
and applied to the summarized schedule
MOST
Orig Dur Rem Dur % Comp Early Start Early Finish OPTIMISTIC PESSIMISTIC
LIKELY

Pico/Aliso Station

Segment at grade works

9815 Segment 2 Utility diversions and duct banks 225 71 68 30NOV05A 10DEC07 65 75 93

9835 Segment 2 1st Street bridge track bed/track 129 129 0 08JAN07A 28FEB08 100 129 149

9819 Segment 2a/b Complete systems installation 221 221 0 11DEC07 14OCT08 199 221 243

9820 Segment 2 Sequencing/resource constraints 20 20 0 5OCT08 11NOV08 5 20 40

Station

980 Pico/Aliso Final design and MTA approvals 118 0 100 30NOV05A 11MAY06A

981 Traffic plan approvals and street closures 61 0 100 15MAY06A 31JUL06A

9810 Peco/Aliso station demo and structure 95 47 51 01FEB07A 06NOV07 40 52 69

9830 Peco/Aloso MEP and Str restore and punch list 190 190 0 07NOV07* 29JUL08 180 200 212
 Schedule Risk Assessment
4. Range-loaded activities are then simulated
(using Monte Carlo methods) to yield likely
project outcomes
The results are reviewed and rerun as necessary
for schedule integrity
The results may affect the cost analysis
 FTA Schedule Risk Assessment
The results indicate the likelihood of meeting the target completion date

*See back of module for larger graphic

 Risk Review Meeting 3

Describe process used to develop quantitative risk

assessment
Summarize key findings of the assessment
Provide recommendations regarding risk
Review which risks strongly influence overall project
risk
Review specific recommended mitigation measures
and solicit completion dates
Discuss action items and next steps
 Risk Response
Risk response
recommendations
and planning

Grantee
Technical
Capacity
(OP-21)

Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)

Monitoring Reports (OP-25)

Risk response
monitoring
 4 Risk Responses (Risk Aversion Order)
1. Avoidance
Pursue a different project or project element
Alternatives Analysis is used here
2. Transfer
By contract or agreement with the performing parties
Major concepts to be considered:
Efficient allocation of risk
Equitable allocation of risk
 4 Risk Responses (Risk Aversion Order)

3. Reduce
Modify either the probability (P) or magnitude
(M), or both, reducing the expected value (E) of
the risk

E = P*M
Modify the design or PMP
Protect against theft or damage
Influence the projects environment
 4 Risk Responses (Risk Aversion Order)

4. Retain/absorb
Recognize a risk event, but take no action
If it appears to be of minimal consequence
Retain it and protect against it
Where risks are unavoidable
Establish contingencies that can absorb realized risks
Develop a Plan B in case risks become realized
 Risk Response Recommendations

Two types of information are used:

1. Qualitative information is used to develop
mitigation action plans for the grantee to
undertake, through the use of Primary
Mitigation
2. Quantitative information is used to establish Risk
Protection against the inherent project risk
through the use of Contingencies and
Secondary Mitigation
 Primary Risk Mitigation
Executed by the grantee through completing
efforts described in the grantees most current
Risk Contingency Management Plan in the PMP
Grantees plans should be consistent with the risk
review and mitigation recommendations
Work is monitored on a continual basis during the
PMOCs monthly monitoring and oversight
Failure to adequately complete work translates to
greater uncertainty as the project moves forward
 Risk Protection
Risk protection is approached from 2 directions:
Contingency budgeted amounts to overcome
normal overruns
Secondary Mitigation development of plans to
recover cost or schedule overruns using redesign
techniques where contingency is insufficient
 Cost Contingency

FTA risk review provides quantitative modeling to assess

the sufficiency of the grantees budget at various project
phases
Specific target values are identified at primary phases:
Entry to Preliminary Engineering
Entry to Final Design
FFGA
Construction
The target values vary, depending on the project phase
current at the time of calculation
 Cost Contingency Targets

Cost Contingency Targets

35%

30%

25%

20%

15%

10%

5%

0%
PE Phase FD Phase FFGA Award 100% Bid 50% Construction
 Cost Contingency
An additional analysis of specific, potential
project risks is conducted to determine how
contingency must be maintained and at what
amounts throughout the project completion
This is called the backward pass method
This analysis identifies points during the
construction period at which certain
minimums of contingency should maintained
to protect the project;
 Schedule Contingency
FTA also established criteria for suggested
minimum schedule contingency levels
The targeted amount of schedule contingency,
for the project as a whole, is 20% of the
stripped duration from Final Design through
Construction completion
 Schedule Contingency

In an analysis similar to that for cost

contingency, a separate, detailed analysis of
minimum schedule contingency levels occur
This analysis identifies points during the project
phases at which certain minimums of schedule
contingency must be maintained
 Cost and Schedule Contingency

Contingency needs are analyzed on an individual

project basis by looking at specific risks
Contingency is not released until these risks are
no longer of concern
Project Milestones indicate the times and
amounts at which the contingency minimum may
be reduced
Final recommendation is developed by PMOC in
consideration of 3 forecast methods
 Secondary Mitigation
To determine the amount of capacity a grantee
should maintain, the FTA establishes targets for
specific phases of the project
Secondary Mitigation does not reduce the
functionality of the initial project
At a given stage of a project, the grantees
estimate (including contingency) is compared
against a target point within the potential range
of cost risk
The difference is called the gap
 Secondary Mitigation

Target @ Entry
into FD=
30% likely value

Mitigation in
amount of gap
must be available

Grantees
estimate
including
contingency
 Risk Management Planning
General process for risk management begins with a written
Risk and Contingency Management Plan (RCMP):
Documentation of the various analyses
Listing of major project risks, their description and the
nature of their consequences
How they are to be handled (by whom, when)
Watch List
A list of secondary and/or residual minor risks that need to be
kept an eye on
Timing for re-examination of the Risk Management portion
of the PMP
 RCMP Requirements
Section of the Grantees Project Management Plan (PMP)
Fully updated and active PMP
Highlight specific areas of management focus as identified
through the risk review process
Means for monitoring Grantees progress
Actions to strengthen technical capacity and capability, project
performance, cost and schedule analyses, mitigations of identified
project risks, and others.
Information contained within the RCMP should complement
and not be in conflict with information contained elsewhere
within the PMP or in other FTA guidance documents
 RCMP Overview
Topics:
Primary Mitigation, organized by significant
project activities, such as:
Technical Capacity
Project Scoping and Design;
Delivery Methods and Contracting;
Construction Process;
Project Tracking, including:
Cost Estimating, Financing and Financial Management;
and
Project Schedule Management.
 RCMP Overview
Insurance:
Professional services, construction phase, wrap-up, or other specialized
insurances purchased for reduction of risk exposure
Contingency Management:
Cost Contingency Management Plan; and
Schedule Contingency Management Plan
Secondary Mitigation:
Establishment of Secondary Mitigation actions and cost targets which
may trigger the implementation of Secondary Mitigation
Risk Management:
Risks management and mitigation monitoring, change identification,
and management controls
 Risk Management Planning

Risk Management Plan is a combination of elements within the

grantees PMP and its sub-parts
Primary Mitigation occurs through professional planning, design,
and execution
Risk Protection involves planning for the what-ifs (e.g., What if
poor soils unexpectedly arise?, What if the market tightens?)
Written plans to handle project overages result in a well-
managed response to any crisis and should be documented
within the PMP
Plans are updated as the project progresses
 Risk Management Process

The process of:

Identifying risk situations, for which outcomes are likely to
vary in amounts that may significantly affect the projects
goals
Assessing the likelihood and magnitude of such variance
Establishing plans for minimizing hazard and maximizing gain
Managing the execution of those plans
A rational means for working in an uncertain world
 Risk workshops and meetings
Meeting Purpose When Attendance Note

Introduction Introduce team, Before formal risk Project May be reduced

get to know the workshops management, during risk
project, discuss risk technical experts, refreshes
process lead designers
Risk workshop #1 Establish current After scope, cost, Project If Grantee has a risk
risk register, schedule etc. reviews. management, process, focus on
evaluate risks, PMOC list of risks technical experts, suggested
identify potential required. lead designers amendments to
mitigations Grantees risk
register
Risk workshop #2 Discuss findings After PMOC risk Project
from PMOC risk modeling and after FTA management
modeling; next concurrence with
steps recommendations;
discuss key mitigations
Risk workshop #3 Discuss mitigations After workshop #2, Project May not be
(optional) if not discussed before finalizing OP40 management, required if covered
earlier report technical experts, in workshops #1&2
lead designers