The Predictability Index ―

Benchmarking Project Outcome Predictions

Construction
Industry Implementation Resource 291-3
Institute®
CII Member Companies
Abbott AMEC
Air Products and Chemicals AZCO
Ameren Corporation Alstom Power
American Transmission Company Audubon Engineering Company
Anglo American Baker Concrete Construction
Anheuser-Busch InBev Barton Malow Company
Aramco Services Company Bechtel Group
ArcelorMittal Bentley Systems
Architect of the Capitol Bilfinger Industrial Services
BP America Black & Veatch
CITGO Petroleum Corporation Burns & McDonnell
Cameco Corporation CB&I
Cargill CCC Group
Chevron CDI Engineering Solutions
ConocoPhillips CH2M HILL
Consolidated Edison Company of New York CSA Central
DTE Energy Coreworx
The Dow Chemical Company Day & Zimmermann
DuPont Dresser-Rand Company
Eastman Chemical Company eProject Management
Ecopetrol Emerson Process Management
Eskom Holdings SOC Faithful+Gould
ExxonMobil Corporation Fluor Corporation
General Electric Company Foster Wheeler USA Corporation
General Motors Corporation GS Engineering & Construction Corporation
GlaxoSmithKline Gross Mechanical Contractors
Global Infrastructure Partners Hargrove Engineers + Constructors
Huntsman Corporation Hatch
Intel Corporation Hilti Corporation
International Paper IHI E&C International Corporation
Irving Oil Limited IHS
Kaiser Permanente Industrial Contractors Skanska
Koch Industries International Rivers Consulting
Eli Lilly and Company JMJ Associates
Linde North America JV Driver Projects
LyondellBasell Jacobs
Marathon Oil Corporation KBR
National Aeronautics & Space Administration Kiewit Corporation
NOVA Chemicals Corporation Kvaerner North American Construction
Occidental Petroleum Corporation Lauren Engineers & Constructors
Ontario Power Generation Leidos Constructors, LLC
Petroleo Brasileiro S/A - Petrobras Matrix Service Company
Petroleos Mexicanos McCarthy Building Companies
Petroliam Nasional Berhad McDermott International
Phillips 66 Midwest Steel
Praxair Parsons
The Procter & Gamble Company Pathfinder
Public Service Electric & Gas Company POWER Engineers
Reliance Industries Limited (RIL) Quality Execution
SABIC - Saudi Basic Industries Corporation Richard Industrial Group
Sasol Technology The Robins & Morton Group
Shell Global Solutions US S&B Engineers and Constructors
Smithsonian Institution SKEC USA
Southern Company SNC-Lavalin
Statoil ASA Technip
SunCoke Energy Tenova
Teck Resources Limited TOYO-SETAL Engenharia
Tennessee Valley Authority URS Corporation
TransCanada Corporation Victaulic Company
U.S. Army Corps of Engineers WESCO International
U.S. Department of Commerce/NIST/ Walbridge
Engineering Laboratory Wanzek Construction
U.S. Department of Defense/Tricare Wilhelm Construction
Management Activity Willbros United States Holdings
U.S. Department of Energy Wood Group Mustang
U.S. Department of Health and Human Services WorleyParsons
U.S. Department of State Yates Construction
U.S. Department of Veterans Affairs Zachry Holdings
U.S. General Services Administration Zurich
Vale
The Williams Companies
The Predictability Index ―
Benchmarking Project Outcome Predictions

Research Team 291, Improving the Accuracy of Project Outcome Predictions

Construction Industry Institute

Implementation Resource 291-3

January 2014
© 2013 Construction Industry Institute™

The University of Texas at Austin

CII members may reproduce and distribute this work internally in any medium at no cost to internal recipients. CII members
are permitted to revise and adapt this work for their internal use, provided an informational copy is furnished to CII.

Available to non-members by purchase; however, no copies may be made or distributed and no modifications may be
made without prior written permission from CII. Contact CII at http://construction-institute.org/catalog.htm to purchase
copies. Volume discounts may be available.

All CII members, current students, and faculty at a college or university are eligible to purchase CII products at member
prices. Faculty and students at a college or university may reproduce and distribute this work without modification for
educational use.

Printed in the United States of America.

Contents

1. Introduction 1

2. The Predictability Index 3

3. Predictability Index Assessment and Interpretation 7

4. The Predictability Index Tool 11

 1
Introduction

CII established Research Team 291 to help project teams produce more reliable project
cost and schedule forecasts. To do this, the team worked to isolate the factors that
promote accurate, timely, and consistent project outcome predictions. Having such
reliable forecasts empowers project teams to adopt appropriate mitigation actions
sooner than they would with faulty predictions. This ability to be proactive, rather than
reactive, can significantly improve a team’s ability to minimize the deviations between
predicted and actual outcomes. For example, while the early recognition that the
baseline cost of a project has been significantly underestimated cannot guarantee
meeting cost targets at project completion, it can enable the project team to proactively
address and improve cost performance. Conversely, the early recognition that a
project’s baseline cost has been significantly overestimated enables organizations to
invest the overcommitted funds as early as possible in other financial endeavors in
order to fulfill the expected rate of return. A timely and proactive response to external
events leads to earlier and more accurate predictions of project outcomes. Indeed,
improved predictability enables project teams to address project performance in a
proactive and timely manner and, thus, increase value. While project teams should not
be expected to eliminate all surprises, they can actually mitigate the potential negative
effects of such surprises by recognizing them early and producing timely information
on them. Unexpected events, whatever their quantity or magnitude, can always be
properly managed with the right people, processes, and behaviors.

Because a project team’s response to such events can make such a difference, its
cost and schedule performance must be evaluated on the basis of its ability to mitigate
deviations through the timely prediction of cost and schedule outcomes—as opposed
to the prevalent practice of basing evaluations on the magnitude of the final deviations
alone. However, evaluating how effectively (i.e., how early and how accurately) a team
forecasts cost and schedule project outcomes is not a common industry practice.
Indeed, to date, the industry has not had a method or tool to measure a team’s ability
to make timely and accurate project outcome predictions.

To address this lack, RT 291 developed the Predictability Index (PI), a metric that enables
organizations to assess and benchmark their predictability efforts on completed projects.
For a given project, the lower the Predictability Index is, the better the predictability
performance. The Predictability Index is expressed as the sum of cost and schedule
predictability measures, as Chapter 2 explains.

1
1: Introduction

Chapter 2 also defines the Predictability Index and its components, while Chapter 3
provides guidance on making PI assessments and interpreting PI values. Chapter 4
provides step-by-step instructions for using the PI tool.

2
 2
The Predictability Index

This chapter discusses the equations that underlie the Predictability Index (PI) measures
and their meaning. Equation 1 expresses the PI as the sum of cost and schedule
predictability measures.

Predictability Index (PI) = Cost Predictability (CP) + Schedule Predictability (SP)

Equation 1

These cost and schedule predictability measures distinctly assess the cost and
schedule forecasting errors along the project completion timeline as the product of the
normalized timeliness error times the absolute deviation error (expressed in percentage
of deviation). (See Equations 2 and 3.)

CP = [ Normalized Cost Timeliness Error ] x [ % Cost Deviation Error ]

Equation 2

SP = [ Normalized Schedule Timeliness Error ] x [ % Schedule Deviation Error ]

Equation 3

The computation of these two error components—deviation and timeliness—for cost and
time predictability is produced as follows. First, the absolute values for the percentage
of cost and time deviations are represented with the pair of equations immediately
below. (See Equations 4 and 5.) The percentage of deviation is measured in absolute
(positive) values and, hence, theoretically ranges from “0” to any percent deviation
value, eventually going beyond 100 percent.

| Actual Installed Cost – Baseline Cost |

% Cost Deviation Error = x 100
Baseline Cost

Equation 4

| Actual Completion Time –

Baseline Completion Time |
% Schedule Deviation Error = x 100
Baseline Completion Time

Equation 5

3
2: The Predictability Index

The normal timeliness error accounts for how timely the cost and schedule predictions
were along the project completion timeline, regardless of the magnitude of the final
deviation. In short, getting to the correct outcome sooner, whatever that outcome
might actually be, is much better than being surprised later when no mitigation actions
are possible. The normal timeliness error is independently computed for cost and
schedule predictions based on the definition of a normalized or unitary (equal to 1)
deviation. This unitary deviation corresponds to the difference between baseline and
completion values. Based on the unitary deviation, the normal timeliness error is equal
to the area above and/or below the actual outcome value at project completion. In
Figure 1, the area chart for Project A conceptually illustrate the computation of the
normalized cost timeliness component for a situation in which all forecasted costs
had been underestimated. It shows that, in such a situation, the area of error is strictly
below the actual total installed cost.

In the figure, Project A is assumed to be an early and accurate predictor of the final
project cost, while Project B is assumed to recognize the actual project cost growth
very late in the project execution. Thus, Project B experiences large errors between
forecasted and actual performance until late in its project execution. As reflected in
the Cost Predictability expression, at equal percentages of cost deviations, Project A
is a much better predictor of cost performance than Project B, since its normal cost
timeliness error is much smaller. The same normal timeliness concept can be applied
to compute the schedule predictability performance. By using this calculation CII
members will be able to gauge and benchmark the predictability of their completed
projects.

4
 2: The Predictability Index

Total Installed Cost

(Cost at tn = 1)
Normal Cost Timeliness Error
(colored area)

Forecasted/Reported
Authorized Cost Total Installed Cost
(Cost at t0 = 0)

Authorization t1 t2 ... tn–1 Completion

Date (t0 = 0) Date (tn = 1)

Project A

Total Installed Cost

(Cost at tn = 1)

Normal Cost Timeliness Error

(colored area)

...

Forecasted/Reported
Authorized Cost Total Installed Cost
(Cost at t0 = 0)

Authorization t1 t2 ... tn–1 Completion

Date (t0 = 0) Date (tn = 1)

Project B

Figure 1. Normal Cost Timeliness Error

Chapter 3 presents a detailed discussion of the PI assessment, its use, and the
interpretation of its components.

5
 3
Assessment and Interpretation

This chapter provides guidance on the PI assessment and its interpretation. Specifically,
it discusses the tool’s metrics on overall predictability, cost and schedule predictability,
and their normal timeliness error components.

The Predictability Index

The Predictability Index provides a quantitative assessment of how early and accurate
a project team’s cost and schedule predictions were, with respect to final outcomes.
In general, low PI values are associated with good―early and accurate―predictability
of project outcomes. The larger the PI is, the worse the combined predictability for
cost and schedule outcomes along the project delivery timeline.

In order to provide an assessment on the predictability performance for a specific PI

value, the research team sought to provide the range of PI values that differentiate early
and accurate predictors as against late and inaccurate project predictors. To this end,
the team computed the PI scores for 135 projects, using the data from the projects that
participated in the research. (See Research Summary 291-1 for a detailed discussion of
the data collection effort.) The team then divided these projects into quartiles according
to their PI values. The first quartile projects had PI values associated with very good
project predictors, the second quartile with good predictors, the third quartile with poor
predictors, and the fourth quartile associated with the very poor project predictors.
Table 1 outlines the PI threshold values that separate the four quartiles and, hence,
from a statistical perspective, differentiate the ability of project teams to predict cost
and schedule outcomes. As the table shows, the minimum possible value of the PI
is zero, and it is indicative of a project that has been completed with no cost and
schedule forecast errors.

Table 1. Predictability Index Threshold Values

Predictability Index Range

Assessment
Minimum Maximum
0 8 Very good predictor
>8 18 Good predictor
>18 32 Poor predictor
>32 Very poor predictor

7
3: Assessment and Interpretation

Cost and Schedule Predictability

In concordance with the CII definition of a world-class project, RT 291 emphasizes the
need to attain good predictability performance for both cost and schedule measures
by minimizing both cost and schedule deviations in order to maximize net present
value. The reader should keep in mind that the PI metric is actually a composite index
that results from computing both cost and schedule predictability performance and,
hence, is not particularly indicative of either cost or schedule predictability alone. For
instance, assume that a schedule-driven project was perfectly forecasted and completed
on time (and, thus, had a null value for the schedule predictability component), but
had a significant cost deviation at completion and large cost forecast errors during
project delivery (and, thus, had a large value for the cost predictability component).
In such case, the PI measure will be rather large and, thus, unable to reflect the exact
prediction of the final schedule outcome. Thus, the PI assessment values presented
in Table A should guide the overall (cost and schedule) predictability evaluation for
any given project.

Nonetheless, the research team acknowledges that a project can be driven by either
cost or schedule—or both—and, thus, a project team can emphasize a clear priority
to satisfy one of those performance measures. Tables 2 and 3 respectively present
the sets of cost and schedule predictability values that belong in the four quartiles
of predictability performance. The reader should note that the team independently
generated these values by sorting out the 135 projects in its dataset in terms of both
cost predictability and schedule predictability. Thus, values in Tables 2 and 3 respectively
prioritize cost and schedule predictability performance alone and, hence, when added,
are not necessarily consistent with the PI values presented in Table 1.

Table 2. Cost Predictability Threshold Values

Range
Assessment
Minimum Maximum
0 3.5 Very good cost predictor
>3.5 7.5 Good cost predictor
>7.5 15 Poor cost predictor
>15 Very poor cost predictor

8
 3: Assessment and Interpretation

Table 3. Schedule Predictability Threshold Values

Range
Assessment
Minimum Maximum
0 3.5 Very good schedule predictor
>3.5 8 Good schedule predictor
>8 18 Poor schedule predictor
>18 Very poor schedule predictor

Normal Timeliness Error

To assess cost and schedule predictability values, the normal timeliness error should
also be evaluated. The reader should recall that the normal timeliness error expresses
the accuracy and timeliness of the forecasts generated during project delivery—
independently for cost and schedule—irrespective of the actual magnitude of their
deviations. Thus, as explained in Chapter 2, at equal percentages of deviation from
the baseline, smaller normal timeliness errors are indicative of a better predictability
performance. (See Figure 1.) To provide guidance on assessing the normal timeliness
error component of both cost and schedule predictability measures, the team sought
to investigate a common set of threshold values. This investigation was irrespective
of cost or schedule deviations, and resulted in the threshold values for normalized
timeliness error shown in Table 4. The threshold values in the table should be used
to assess the timeliness of both cost and schedule outcomes at completion. Such
assessment is achieved by separately comparing the threshold values in Table 4 with
the normal cost timeliness error and/or the normal schedule timeliness error.

Table 4. Normal Timeliness Error Threshold Values

Error Range
Assessment
Minimum Maximum
0 0.4 Very timely forecasting
>0.4 0.55 Timely forecasting
>0.55 0.75 Late forecasting
>0.75 Very late forecasting

9
 4
The Predictability Index Tool

RT 291 designed the Predictability Index (PI) tool to evaluate the predictability of cost and
schedule outcomes for completed projects. The PI tool enables owner and contractor
organizations to quantitatively assess and benchmark the predictability performance
of their completed projects. This chapter explains how to use this Excel-based tool.

The PI tool consists of 10 sections organized into tabs: 1) Introduction; 2) Instructions;

3) Input Project Data; 4) Input Cost Forecast; 5) Input Schedule Forecast; 6) Cost
Predictability Index; 7) Schedule Predictability Index; 8) Predictability Index; 9) Cost
Predictability; and 10) Schedule Predictability. The first two tabs give general information
and instructions on the purpose and use of the tool. In the third section, the user is
asked to provide personal information and general project data. The fourth and fifth
sections ask for the cost and schedule forecasted values of the project being assessed.
The remaining sections present the tool outputs. Specifically, the sixth, seventh, and
eighth sections respectively show the quantitative assessment for cost predictability,
schedule predictability, and overall predictability. Finally, the ninth and tenth sections
respectively present the cost and schedule timeliness error charts. The remainder
of this chapter gives instructions for using the tool and offers guidance on entering
the requested information, assessing the output results, and printing the information
contained in the tool.

11
4: The Predictability Index Tool

Input
The user should take the following steps in Section 3:

1. Make sure that the records in ALL the input tabs are empty. Not doing so
can alter and hinder the computed values.
2. Fill the fields in the Project Data section, which asks for both project data
and evaluator information. Notice that the cost and schedule units defined
by the user in the tab must be consistently utilized throughout the tool for
accurate results. Thus, if installed cost and completion time are defined
in millions of dollars and months, the forecast data must be introduced in
these units for the results to be consistent. (See Figure 2 for an example of
input data.)
3. Enter cost forecast log data, up to a total of 200 forecast records.
Sequentially introduce each instance of generated cost forecast with two
parameters (i.e., time at which the forecast was generated/reported, and
forecasted installed cost). Make sure that, in the last record introduced
by the user, the time at which the forecast was generated/reported
is equal to the total completion time of the project as introduced in
the Project Data section. Failure to meet this requirement will result in the
incorrect measurement of predictability. (If either of these two conditions
is violated, the tool will display an error message to alert the user.) Figure 3
provides an example of input data that are consistent with the units
defined in the Project Data section (See Figure 3.)
4. Enter schedule forecast log data, up to a total of 200 forecast records.
Sequentially introduce each instance of the generated schedule forecast
with two parameters (i.e., time at which the forecast was generated/
reported, and forecasted project duration). Make sure that, in the last
record introduced by the user, the time at which the forecast was
generated/reported is equal to the total completion time of the
project as introduced in the Project Data section. Failure to meet this
requirement will result in the incorrect measurement of predictability. (If
either of these two conditions is violated, the tool will display an error
message to alert the user.) Figure 4 provides an example of input data that
are consistent with units defined in Project Data section. (See Figure 4.)

12
 4: The Predictability Index Tool

Figure 2. Project Data Section

Last data record must match

completion outcome values
from Input Project Data tab

Enter records in
time-sequential order

Figure 3. Cost Forecast Data Input

Last data record must match

completion time value from
Input Project Data tab

Enter records in
time-sequential order

Figure 4. Schedule Forecast Data Input

13
4: The Predictability Index Tool

Output
The tool contains five output tabs, inclusive of two charts. The first two output tabs
determine the predictability of cost and schedule forecasts, as the product of their
deviation error times the normal timeliness error. Cost and schedule predictability
values are independently calculated based on corresponding input data. The third
output tab measures the Predictability Index as the sum of the cost and schedule
predictability measures. The fourth and fifth outputs present the cost and schedule
forecast error charts.

1. After introducing the cost and schedule forecast log data, review the
results under the “Cost Predictability Index,” “Schedule Predictability
Index,” and “Predictability Index” tabs. Figure 5 presents an example of
the PI output tab. Throughout these tabs, if any output value is labeled
“ERROR,” it implies that the user has not addressed erroneous data inputs
that the tool has announced in error messages. Until such errors have
been resolved the tool cannot generate accurate results.
2. Each of the numeric outcome tabs (Cost Predictability, Schedule
Predictability, and the Predictability Index) has two components that can
be independently assessed. For instance, the Cost Predictability measure
is generated on the basis of the project’s normal timeliness and deviation
errors. The assessment of such components is necessary to understand
cost predictability performance.
3. The performance for each of these three predictability metrics (Cost
Predictability, Schedule Predictability, and the Predictability Index) is
illustrated on a dashboard dial in quartiles corresponding to very good,
good, poor, and very poor. The team statistically determined and adjusted
the quartiles by comparing the accurate and early predictors to the
inaccurate and late predictors, among the projects in its dataset.
4. To complement the previous quantitative measures, the “Cost
Predictability” and “Schedule Predictability” tabs contain the timeliness
chart representation for both cost and schedule forecasts. The area
between the actual project outcome (i.e., installed costs for cost forecast
and completion time for schedule forecast) and the forecasted values
during the project timeline illustrates the timeliness error. In other words,
the cost timeliness chart illustrates how early and accurately the final value
of total installed costs was ascertained for a given project. Similarly, the
schedule timeliness chart illustrates how precise the schedule forecasts
were during the project delivery process. Figure 6 provides an example of
a cost forecast error chart.

14
 4: The Predictability Index Tool

Figure 5. Predictability Index Output Screen

Figure 6. Cost Timeliness Error

Printing
1. Go to the INPUT tabs and print.
2. Go to the OUTPUT tabs and print.

15
Research Team 291, Improving Predictability of Project Outcomes

* W. Edward Back, The University of Alabama

Glen Cullop, Eastman Chemical Company
Russell T. Cusimano, eProject Management, LLC
* David Grau, Arizona State University
John Greco, Air Products and Chemicals, Inc., Co-chair
Kay Harlow, Southern Company
Doug Helmann, Architect of the Capitol
Micki Kohn, Hargrove Engineers + Constructors
Brian Kong, U.S. Department of Energy
Guillermo Mejia, The University of Alabama/Universidad Industrial de Santander
Robert Mozzi, Saudi Basic Industries Corporation
Oscar Rodriguez, URS Corporation
Rick Sirven, ConocoPhillips, Chair
Patrick Sweeney, SNC Lavalin Constructors, Inc
William A. Taylor, Alstom Power
Douglas E. Weaver, Walbridge

Former Members
Chris Sorrell, Day and Zimmermann
Mark White, Faithful+Gould

* Principal authors

Editor: Jacqueline Thomas

 Construction Industry Institute
The University of Texas at Austin
3925 W. Braker Lane (R4500)
Austin, Texas 78759-5316
 IR 291-3

The Knowledge Leader for Project Success

Owners • Contractors • Academics