See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/266666768

The Risk of Using Risk Matrices

Article in SPE Economics & Management · September 2013

DOI: 10.2118/166269-MS

CITATIONS READS
96 60,654

3 authors:

Philip Thomas Reidar B. Bratvold

2 PUBLICATIONS 98 CITATIONS
University of Stavanger (UiS)
148 PUBLICATIONS 1,636 CITATIONS
SEE PROFILE
SEE PROFILE

J. Eric Bickel
University of Texas at Austin
87 PUBLICATIONS 1,164 CITATIONS

SEE PROFILE

All content following this page was uploaded by Reidar B. Bratvold on 06 April 2015.

The user has requested enhancement of the downloaded file.

 The Risk of Using Risk Matrices
Philip Thomas, SPE, and Reidar B. Bratvold, SPE, University of Stavanger; and
J. Eric Bickel, SPE, University of Texas at Austin

Summary This paper summarizes the known flaws of RMs, identifies

The risk matrix (RM) is a widely espoused approach to assess and several previously undiscussed problems with RMs, and illus-
analyze risks in the oil and gas (O&G) industry. RMs have been trates that these shortcomings can be seen in SPE papers that ei-
implemented throughout that industry and are used extensively in ther demonstrate or recommend the use of RMs. The paper is
risk-management contexts. This is evidenced by numerous SPE organized as follows: The next section describes RMs. The fol-
papers documenting RMs as the primary risk-management tool. lowing section discusses current practices and standards for risk
Yet, despite this extensive use, the key question remains to be management, including an example. We then illustrate the flaws
addressed: Does the use of RMs guide us to make optimal (or and dangers resulting from the use of RMs before we provide a
even better) risk-management decisions? very short overview of methods and references that discuss a con-
We have reviewed 30 SPE papers as well as several risk-man- sistent approach to risk management. Finally, we provide a sum-
agement standards that illustrate and discuss the use of RMs in a mary and a discussion.
variety of risk-management contexts, including health, safety, and
environment (HSE), financial; and inspection. These papers pro-
mote the use of RMs as a “best practice.” Unfortunately, they do RMs
not discuss alternative methods or the benefits and detriments of An RM is a graphical presentation of the likelihood, or probabil-
the use of RMs. ity, of an outcome and the consequence should that outcome
The perceived benefit of the RM is its intuitive appeal and sim- occur. Consequences are often defined in monetary terms. RMs,
plicity. RMs are supposedly easy to construct, easy to explain, as their name implies, tend to be focused on outcomes that could
and easy to score. They even might appear authoritative and intel- result in loss, rather than gain. The purported objective of the RM
lectually rigorous. However, the development of RMs has taken is to prioritize risks and risk-mitigation actions.
place completely isolated from scientific research in decision Within the context of RMs, “risk” is typically defined as con-
making and risk management. This paper discusses and illustrates sequence multiplied by its probability, which yields the expected
how RMs produce arbitrary decisions and risk-management actions. downside consequence or the expected loss. Rather than refer to
These problems cannot be overcome because they are inherent in expected downside consequence as “risk,” we will use the more
the structure of RMs. In their place, we recommend that O&G pro- precise term expected loss (EL).
fessionals rely on risk- and decision-analytic methods that rest on Pritchard et al. (2010) gave an example of using RMs to assess
250 years of scientific thought and testing. the risk of a drilling hazard. This paper was one of three in a spe-
cial issue of World Oil devoted to advances in drilling. Pritchard
et al. (2010) note the example as a “typical industry risk assess-
Introduction ment matrix.” We have adopted this example as Fig. 1 and use it
In the O&G industry, risk-intensive decisions are made daily. In to explain the flaws inherent in RMs.
their attempt to implement a sound and effective risk-manage- As can be seen in Fig. 1, the consequences and probabilities in
ment culture, many companies use RMs1 and specify this in “best an RM are expressed as a range. For example, the first conse-
practice” documents. Furthermore, RMs are recommended in quence category might be “< USD 100K,” the second might be
numerous international and national standards such as ISO,2 API,3 “USD 100–250K,” and so on. The first probability range might be
and NORSOK.4 The popularity of RMs has been attributed in part “< ¼ 1%,” the second might be between 1 and 5%, and so forth.
to their visual appeal, which is claimed to improve communications. A verbal label and a score are also assigned to each range. (Some
Despite these claimed advantages, we are not aware of any pub- RMs use these instead of a quantitative range.) For example,
lished scientific studies demonstrating that RMs improve risk-man- probabilities from 10 to 20% might be labeled as “seldom” and
agement decisions.5 However, several studies indicate the opposite: assigned a score of 4. Probabilities greater than 40% might be
that RMs are conceptually and fundamentally flawed. For example, termed “likely” and given a score of 6. Consequences from USD
Cox et al. (2005) derived and discussed several fundamental flaws 5 to 20 million might be termed “severe” and given a score of 5;
introduced through the qualitative scoring system that is often used losses greater than USD 20 million might be labeled as “cata-
in RMs. Cox (2008) provided further examples of these flaws and strophic” and given a score of 6.
presented a set of rules that RMs must obey if they are to be logi- It is interesting and concerning that such an RM would treat
cally consistent. Hubbard (2009) provided compelling arguments losses of USD 50 billion (on the scale of BP’s losses stemming
for why, in most cases, the use of RMs results in unclear informa- from the Macondo blowout) or USD 20 million in the same way,
tion flow and suboptimal risk-management decisions. despite the three-orders-of-magnitude difference. Because there is
no scientific method of designing the ranges used in an RM, many
practitioners simply use the ranges specified in their company’s
Copyright V
C 2014 Society of Petroleum Engineers
best-practice documents. In fact, as we will show, differently
This paper (SPE 166269) was accepted for presentation at the SPE Annual Technical shaped regions can alter risk rankings.
Conference and Exhibition, New Orleans, 30 September–2 October 2013, and revised for
publication. Original manuscript received for review 16 July 2013. Revised manuscript The cells in RMs are generally colored green, yellow, and red.
received for review 25 November 2013. Paper peer approved 11 December 2013. Green means “acceptable,” yellow stands for “monitor, reduce if
1
Sometimes called probability-impact matrices (PIMs) possible,” and red is “unacceptable, mitigation required.” Previous
2
International Organization for Standardization (ISO), the world’s largest developer of vol- work has detailed the way in which the colors must be assigned if
untary international standards one seeks consistency in the ranking of risks. Most of the SPE
3
American Petroleum Institute (API), which establishes standards for petroleum-industry papers we examined failed to assign colors in a logically consistent
activities in the US
4
way. For example, some of the cells designated as red were “less
NORSOK—produces standards for petroleum-industry activities in Norway
5
risky” than some of the cells that were designated as yellow.
The use of RMs to analyze and manage risks may be better than doing nothing. Indeed,
any approach that generates some discussion of the risks in a particular activity will be The problem context presented in Pritchard et al. (2010) is the
helpful. loss of fluid during drilling in a particular section of a well. There

56 April 2014 SPE Economics & Management

 Probability P - Rating P - Indices
> 40% 6 Likely
20% < p <= 40% 5 Occasional Severe Losses
10% < p <= 20% 4 Seldom
5% < p < = 10% 3 Unlikely Well Control
1% < p < = 5% 2 Remote Blowout
<= 1% 1 Rare
Consequence Rating 1 2 3 4 5 6
Consequence Indices Incidental Minor Moderate Major Severe Catastrophic
Consequence Cost <= USD 100K USD 100–250K USD 250K–1MM USD 1–5MM USD 5–20MM > USD 20MM

Fig. 1—RMs modified from Pritchard et al. (2010).

TABLE 1—DRILLING CASE EXAMPLE TABLE 2—RISK-RANKING RESULTS

Outcome Consequence (USD Million) Probability Outcome Risk Score Rank

Severe Losses 1 to 5 40% Severe Losses 5"4 ¼ 20 1

Well Control 5 to 20 10% Well Control 3"5 ¼ 15 2
Blowout >20 5% Blowout 2"6 ¼ 12 3
Event: Fluid losses occur in hole section (12 to 14 in.)

come of, for example, USD 3 million. Similarly, there was an

is a need to identify the possible outcomes and consequences aris- overlap in their probability ranges of 0 to 1% and 0 to 5%, which
ing from this event and to prioritize these risks. Three possible means that the ranges were not mutually exclusive.
downside outcomes were identified: severe losses of drilling fluid,
well-control issues, and blowout.6 Once the possible outcomes Current Practices and Standards
were defined, Pritchard et al. (2010) specified their probabilities
RMs are considered to be versatile enough to be used to analyze
and the range of possible consequences, both of which are given
and prioritize risks in many settings. A number of international
in Table 1.7 Once the assessment of consequence and probability8
standards support the role of RMs in risk assessment, and many
was complete, the outcome was plotted in the RM (Fig. 1) to
companies consider RMs to be a “best practice.” In this section,
determine whether the risk of an outcome fell into a green, yel-
we illustrate a common RM-analysis approach. We then summa-
low, or red region. Thus, well control and blowout fell in the yel-
rize how some central risk-management standards view the use of
low region, whereas severe losses fell in the red region. Hence, in
RMs.
the parlance of RMs, the possibility of severe losses is “riskier”
than either well control or blowout and should therefore be priori-
tized over these other two concerns. Common Industry Practices. To use the RM for risk prioritiza-
Fig. 1 indicates the score associated with each range. Pritchard tion and communication, several steps must be carried out. Clare
et al. (2010) assumed that cells along a diagonal with slope of –1 and Armstrong (2006) presented a common risk-evaluation pro-
have the same risk. Thus, they considered blowout and well con- cess for the O&G industry, in which they used RMs as a risk-eval-
trol to have the same degree of risk. Poedjono et al. (2009) and uation tool. The work process they used is shown in Fig. 2.
Dethlefs and Chastain (2012) also documented the use of RMs in Step 1: Define Risk Criteria. This step determines the size of
a drilling context, but they used the more common practice of the RM and its number of colors. Although there is no technical
multiplying the probability and consequence scores to obtain a reason for it, RMs are generally square. The most common size is
“risk score” for each outcome. Table 2 shows the results of apply- five rows by five columns (i.e., a 5"5 matrix), but some compa-
ing this procedure to the Pritchard et al. (2010) example. There nies use a 3"3 matrix and others use an 8"8 matrix. Some com-
appears to be no mathematical theory that would allow the multi- panies choose to include more colors than the standard red,
plication of scores, a practice that seems to be an attempt to yellow, and green in their RMs.
mimic the calculation of expected loss, in which case monetary Step 2: Define Risk Events. This step identifies the risk
consequence would be multiplied, or “risked,” by the likelihood events. For example, drilling a particular section of a hole is the
of its occurrence. On the basis of these results, actions to mitigate event for which we are going to identify all the possible downside
severe losses will be prioritized whereas blowout will be add- outcomes.
ressed only after the other two possible outcomes have been Step 3: Consequence Estimation and Probability Assessment.
addressed. This step estimates the consequence range of each outcome iden-
Before concluding this section, we explain how and why we tified in Step 2 and assigns probabilities to each outcome. For
slightly modified the RM used by Pritchard et al. (2010). First, example, the outcome of severe losses is registered, and the
they used a decreasing score scale rather than the increasing scale expected financial consequence is estimated to be from USD 1 to
that is more commonly used. As we will show later, the choice 5 million. The chance of this occurring is estimated to be 40%.
between an ascending or descending scale in our analysis can alter By use of the RM in Fig. 1, this equates to a probability score of 5
the prioritization. Second, they did not use mutually exclusive cat- (“occasional”) and a consequence score of 4 (“major”).
egories. Specifically, they used categories of USD 1 to 5 million Step 4: Risk Profile. This step positions each identified down-
and USD 2 to 20 million. This is clearly problematic for an out- side outcome in a cell in the RM.
Step 5: Rank and Prioritize. This step ranks and prioritizes
the outcomes according to their risk score. Most companies use a
6
The outcomes are assumed to be independent, which might not be correct. For example, risk-management policy in which all outcomes in the red area are
a blowout implies loss of well control. “unacceptable” and thus must be mitigated.
7
The probabilities in this case example are taken from Pritchard et al. (2010), and the con- The results of Steps 2 through 5 are often collectively called a
sequences come from reconversion of the consequence scores into their definition as pre-
sented in Pritchard et al. (2010). “risk register,” and the information required is usually collected
8
The probabilities need not sum to unity because the events are assumed to be mutually
in a joint meeting with the key stakeholders from the operating
exclusive, but not collectively exhaustive. company, service companies, partners, and others.

April 2014 SPE Economics & Management 57

 Step 3a.
Probability Assessment

Step 4. Step 5.
Step 1. Step 2.
Plot in Risk Matrix Risk Prioritization
Define Risk Cricteria Define Risk Events
“Risk Profile” & Mitigation plan

Step 3b.
Consequence Estimation

Fig. 2—Common workflow for analyzing risks by use of RMs.

Standards. Among the standards that are commonly used in the matrix” and “risk matrices.” This returned 527 papers. Then, we
O&G industry are API, NORSOK, and ISO. All of these standards removed 120 papers published before the year 2000, to make sure
recommend RMs as an element of risk management. This section our study is focused upon current practice. We next reviewed the
summarizes how each of these standards supports RMs. remaining 407 papers and selected those that promote the use of
API. API RP 581(2008) recommends RMs customarily for its RMs as a “best practice” and actually demonstrate RMs in the pa-
risk-based-inspection (RBI) technology. RBI is a method to opti- per; leaving 68 papers. We further eliminated papers that pre-
mize inspection planning by generating a risk ranking for equip- sented the same example. In total, we considered a set of 30
ment and processes and, thus, prioritization for inspection of the papers covering a variety of practice areas (e.g., HSE, hazard
right equipment at the right time. API RP 581 specifies how to analysis, and inspection). We believe that this sampling of papers
calculate the likelihoods and consequences to be used in the RMs. presents the current RM practice in the O&G industry. We did not
The specification is a function of the equipment that is being ana- find any SPE papers documenting the known pitfalls of the use of
lyzed. The probability and consequence of a failure are calculated RMs. The 30 papers we consider in this paper are given in Appen-
by use of several factors. API RP 581 asserts that “Presenting the dix A.
results in a risk matrix is an effective way of showing the distribu-
tion of risks for different components in a process unit without nu-
merical values.” Known Deficiencies of RMs
NORSOK. The NORSOK (2002) standards were developed by Several deficiencies of RMs have been identified by other authors.
the Norwegian petroleum industry to “ensure adequate safety, value
adding and cost effectiveness for petroleum industry developments Risk-Acceptance Inconsistency. RMs are used to identify, rank,
and operations. Furthermore, NORSOK standards are as far as pos- and prioritize possible outcomes so that scarce resources can be
sible intended to replace oil company specifications and serve as directed toward the most-beneficial areas. Thus, RMs must reli-
references in the authority’s regulations.” NORSOK recommends ably categorize the possible outcomes into green, yellow, and red
the use of RMs for most of their risk-analysis illustrations. The regions. Cox (2008) suggested we should conform to three axioms
RMs used by NORSOK are less rigid than those of API RBI and one rule when designing RMs to ensure that the EL in the
because the NORSOK RMs can be customized for many problem green region is consistently smaller than the EL in the red region.
contexts (the RM template is not standardized). NORSOK S-012, Cox (2008) also clarifies that the main purpose of the yellow
an HSE document related to the construction of petroleum infra- region is to separate the green region and red region in the RMs,
structure, uses an RM that has three consequence axes—occupa- not to categorize the outcomes. He argues that the RM is inconsis-
tional injury, environment, and material/production cost—with a tent if the EL in the yellow region can be larger than in any of the
single probability axis for all three consequence axes. red cells or smaller than in any of the green cells. Nevertheless,
ISO. ISO standards ISO 31000 (2009) and ISO/IEC 31010 the practice in O&G is to use the yellow region to denote an out-
(2009) influence risk-management practices not only in the O&G come with a medium risk. Every SPE paper we reviewed imple-
industry but in many others. In ISO 31000, the RM is known as a ments this practice and also violates at least one of the axioms or
probability/consequence matrix. In ISO/IEC 31010, there is a ta- the rule proposed by Cox (2008), leading to inconsistencies in the
ble that summarizes the applicability of tools used for risk assess- RMs. inconsistency in yellow region is large
ment. ISO claims that the RM is a “strongly applicable” tool for Fig. 3 shows an example RM with many outcomes. This
risk identification and risk analysis and is “applicable” for risk example shows that there are two groups of outcomes. The first
evaluation. As with the NORSOK standard, ISO does not stand- group is the outcome with medium-high probability and medium-
ardize the number of colors, the coloring scheme (risk-acceptance high consequence (e.g., severe losses, well-control issues) and the
determination), or the size of range for each category. ISO praises second group is the outcome with the low probability but very
RMs for their convenience, ease of use, and quick results. How- high consequence (e.g., blowout). In Fig. 3, the first group of out-
ever, ISO also lists limitations of RMs, including some of their comes is illustrated in the red cells whereas the second group is in
inconsistencies, to which we now turn. the yellow cell. The numbers shown in some of the cells represent
the probability, consequence, and EL, respectively, where EL is
calculated as probability multiplied by consequence. This exam-
Deficiencies of RMs ple shows the inconsistency between EL and color practice in
Several flaws are inherent to RMs. Some of them can be corrected, RMs where all outcomes in the red cells have a lower EL com-
whereas others seem more problematic. For example, we will show pared with the outcome in the yellow cell. Assuming that we wish
that the ranking produced by a RM depends upon arbitrary choices to rank outcomes on the basis of expected loss, we would priori-
regarding its design, such as whether one chooses to use an increas- tize the outcome in the yellow cell compared with the outcomes
ing or decreasing scale for the scores. As we discuss these flaws, in the red cells, which is the opposite of the ranking provided by
we also survey the SPE literature to identify the extent to which the color regions in the RM. Clearly, the use of the RM would in
these flaws are being made in practical applications. this case lead us to focus our risk-mitigation actions on the out-
To locate SPE papers that address or demonstrate the use of come that does not have the highest EL. This type of structure is
RMs, we searched the OnePetro database using the terms “risk evident in eight of the papers we reviewed.

58 April 2014 SPE Economics & Management

 Probability P - Rating P - Indices
> 40% 6 Likely (45%,1,0.45) (45%,3,1.35) (45%,15,6.75) (45%,25,11.25)
20% < p <= 40% 5 Occasional (25%,3,0.75) (25%,15,3.75) (25%,25,6.25)
10% < p <= 20% 4 Seldom (15%,15,2.25) (15%,25,3.75)
5% < p <= 10% 3 Unlikely (10%, 25, 2.5)
1% < p <= 5% 2 Remote (5%, 250, 12.5)
<=1% 1 Rare
Consequence Rating 1 2 3 4 5 6
Consequence Indices Incidental Minor Moderate Major Severe Catastrophic
Consequence cost <= USD 100K USD 100–250K USD 250K–1MM USD 1–5MM USD 5–20MM > USD 20MM

Fig. 3—Risk acceptance inconsistency in RMs.

45% risks. The use of the scoring mechanism embedded in RMs com-
presses the range of outcomes and, thus, miscommunicates the
40% Severe Losses
relative magnitude of both consequences and probabilities. The
35% failure of the RM to convey this distinction seems to undermine
30% its commonly stated benefit of improved communication. This
Range compression problem, that 25M loss is the same as 250M loss example demonstrates the range compression inherent in RMs,
25%
Probability

which necessarily affected all the surveyed SPE papers. The next
20% section will introduce the “lie factor” (LF) that we use to quantify
15% the degree of range compression.
10% Well Control
Blowout
Centering Bias. Centering bias refers to the tendency of people
5%
to avoid extreme values or statements when presented with a
0% choice. For example, if a score range is from 1 to 5, most people
0 20 40 60 80 100
will select a value from 2 to 4. Hubbard (2009) analyzed this in
Consequence Millions, USD the case of information-technology projects. He found that 75%
of the chosen scores were either 3 or 4. This further compacts the
Fig. 4—Plot of probabilities and consequences value of the out- scale of RMs, exacerbating range compression. Smith et al.
comes in the case example.
(2009) came to the same conclusions from investigating risk man-
agement in the airline industry.
Range Compression. Cox (2008) described range compression Is this bias also affecting risk-management decisions in the
in RMs as a flaw that “assigns identical ratings to quantitatively O&G industry? Unfortunately, there is no open-source O&G
very different risk.” Hubbard (2009) also focused extensively on database that can be used to address this question. However, six
this problem. of the reviewed SPE papers presented their data in sufficient
Range compression is unavoidable when consequences and detail to investigate whether the centering bias seems to be occur-
probabilities are converted into scores. The distance between risks ring. Each of the six papers uses an RM with more than 15 out-
in the RM using scores (mimicking expected-loss calculation) comes. Fig. 5 shows the percentage of the outcomes that fell into
does not reflect the actual distance between risks (specifically, the the middle consequence and probability scores. For example, pa-
difference in their expected loss). per SPE 142854 used a 5"5 RM; hence, the probability ratings
In our case example shown in Fig. 1, blowout and well control ranged from 1 to 5. Paper SPE 142854 has 24 outcomes, out of
are considered to have the same risk (both are yellow). However, which 18 have a probability rating of 2, 3, or 4 (which we will
this occurs only because of the ranges that were used and the arbi- denote as “centered”), and the remaining six outcomes have a
trary decision to have the “catastrophic” category include all con- probability rating of 5. Hence, 75% of the probability scores were
sequences greater than USD 20 million. Fig. 4 more accurately centered.
represents these outcomes. A blowout could be many orders of For the six papers combined, 83% of the probability scores
magnitude worse than a loss of well control. Yet, the RM does not were centered, which confirms Hubbard (2009). However, only
emphasize this in a way that we think is likely to lead to high- 52% of the consequence scores were centered, which is less than
quality risk-mitigation actions. To the contrary, the sense that we that found in Hubbard (2009). A closer inspection shows that in
get from Fig. 1 is that a blowout is not significantly different (if four out of the six papers, 90% of either probability or conse-
any different) from a loss in well control—they are both “yellow” quence scores were centered.

100%
90% Average of centered
80% probability scores
70% 75% as documented
Percentage

60% in Hubbard (2009)

50%
Average of centered
40% consequence scores
30%
20%
10%
0%
SPE 142824 SPE 146845 SPE 74080 OTC 18912 SPE 162500 SPE 73897

Paper Number

Probability Consequence

Fig. 5—Centering-bias evidence in SPE papers.

April 2014 SPE Economics & Management 59

 TABLE 3—CATEGORY-DEFINITION-BIAS EVIDENCES IN SPE PAPERS

Paper Index Index Definition Quantitative Measures

SPE 146845 Frequent Several times a year in one location Occurrence > 1/year
SPE 127254 Frequent Expected to occur several times during lifespan of a unit Occurrence > 1/year
SPE 162500 Frequent Happens several times per year in same location or operation Occurrence > 0.1/year
SPE 123457 Frequent Has occurred in the organization in the last 12 months –
SPE 61149 Frequent Possibility of repeated incidents –
SPE 146845 Probable Several times per year in a company 1/year > Occurrence > 0.1/year
SPE 127254 Probable Expected to occur more than once during lifespan of a unit 1/year > Occurrence > 0.03/year
SPE 162500 Probable Happens several times per year in specific group company 0.1/year > Occurrence > 0.01/year
SPE 123457 Probable Has occurred in the organization in the last 5 years or has –
occurred in the industry in the last 2 years
SPE 158115 Probable Not certain, but additional factor(s) likely result in incident –
SPE 61149 Probable Possibility of isolated incident –

Probability P - Rating Descending P - Rating Ascending P - Indices

> 40% 1 6 Likely
20% < p <= 40% 2 5 Occasional Severe Losses
10% < p <= 20% 3 4 Seldom
5% < p <= 10% 4 3 Unlikely Well Control
1% < p <= 5% 5 2 Remote Blowout
<=1% 6 1 Rare
Consequence Rating Ascending 1 2 3 4 5 6
Consequence Rating Descending 6 5 4 3 2 1
Consequence Indices Incidental Minor Moderate Major Severe Catastrophic
Consequence Cost <= USD 100K USD 100–250K USD 250K–1MM USD 1–5MM USD 5–20MM > USD 20MM

Fig. 6—Two different scoring systems for an RM.

Category-Definition Bias. Budescu et al. (2009) concluded that than 1 occurrence in 10 years.” This clearly shows inconsistency
providing guidelines on probability values and phrases is not suf- between members of the same industry. Table 3 summarizes the
ficient to obtain quality probability assessments. For example, variations in definitions within the same indices in some of the
when guidelines specified that “very likely” should indicate a SPE papers surveyed.
probability greater than 0.9, study participants still assigned prob- Given these gross inconsistencies, how can we accept the
abilities in the 0.43 to 0.99 range when they encountered the claim that RMs improve communication? As we show here, RMs
phrase “very likely.” He argued that this creates the “illusion of that are actually being used in the industry are likely to foster mis-
communication” rather than real communication. If a specific def- communication and misunderstanding, rather than improve com-
inition of scores or categories is not effective in helping experts to munication. This miscommunication will result in misallocation
be consistent in their communication, then the use of only qualita- of resources and the acceptance of suboptimal levels of risk.
tive definitions would likely result in even more confusion. Wind- Why EGPC matrix probalility&concequence is not same direction?
schitl and Weber (1999) showed that the interpretation of phrases
Identification of Previously Unrecognized
conveying a probability depends on context and personal prefer-
ences (e.g., perception of the consequence value). Although most Deficiencies
research on this topic has focused on probability-related words, This section discusses three RM flaws that had not been previously
consequence-related words such as “severe,” “major,” or “cata- identified. We demonstrate that these flaws cannot be overcome
strophic” would also seem likely to foster confusion and and that RMs will likely produce arbitrary recommendations.
miscommunication.
We reviewed the 30 SPE papers on the scoring method used. Ranking is Arbitrary. Ranking Reversal. Lacking standards for
The papers were then classified into qualitative, semiqualitative, how to use scores in RMs, two common practices have evolved:
and quantitative categories.9 Most of the scores (97%) were qualita- ascending scores or descending scores. The example in Fig. 1
tive or semiqualitative. However, these papers included no discus- uses ascending scores, in which a higher score indicates a higher
sion indicating that the authors are aware of category-definition probability or more serious consequence. Using descending
bias or any suggestions for how it might be counteracted. scores, a lower score indicates a higher probability or more seri-
Category-definition bias is also clearly seen between papers. ous consequence. These practices are contrasted in Fig. 6.
For example, paper SPE 142854 considered “improbable” as A glance at Fig. 6 might give the impression that ascending or
“virtually improbable and unrealistic.” In contrast, paper SPE descending scores would produce the same risk ranking of out-
158114 defined “improbable” as “would require a rare combina- comes. However, Table 4 shows for each ordering the resulting
tion of factors to cause an incident.” These definitions clearly risk scores and ranking of the outcomes shown in Fig. 6. With the
have different meanings, which will lead to inconsistent risk use of ascending scores, severe losses will be prioritized for risk
assessments. This bias is also seen in the quantitative RMs. Paper mitigation. However, with the use of the descending scores, blow-
SPE 127254 categorized “frequent” as “more than 1 occurrence out will be prioritized for risk mitigation.
per year,” but paper SPE 162500 categorized “frequent” as “more The typical industry RM given in Pritchard et al. (2010) used
descending ordering. However, both ascending and descending
9
Qualitative refers to RMs in which none of the definitions of probability and consequence scoring systems have been cited in the SPE literature and there is
categories provide numerical values. Semiqualitative refers to RMs in which some of the no scientific basis for either method. In the 30 SPE papers sur-
definitions of probability and consequence categories provide numerical values. Quantita-
tive refers to RMs in which definitions of all probability and consequence categories provide veyed, five use the descending scoring system, and the rest use
numerical values. the ascending scoring system. This behavior demonstrates that

60 April 2014 SPE Economics & Management

 TABLE 4—RISK PRIORITIZATION BASED ON ASCENDING AND DESCENDING SCORES

Ascending Descending

Outcome Risk Score Rank Outcome Risk Score Rank

Severe Losses 5"4 ¼ 20 1 Severe Losses 2"3 ¼ 6 2

Well Control 3"5 ¼ 15 2 Well Control 4"2 ¼ 8 3
Blowout 2"6 ¼ 12 3 Blowout 5"1 ¼ 5 1

TABLE 5—PROBABILITY RANGES FOR TWO VALUES OF THE MULTIPLIER n

n¼ 2 n¼ 3

Equation Score Probability Score Probability

0.01#n4 < p $ 1 6 0.16 < p < ¼ 1 6 0.81 < p < ¼ 1

0.01#n3 < p $ 0.01#n4 5 0.08 < p < ¼ 0.16 5 0.27 < p < ¼ 0.81
0.01#n2 < p $ 0.01#n3 4 0.04 < p < ¼ 0.08 4 0.09 < p < ¼ 0.27
0.01#n < p $ 0.01#n2 3 0.02 < p < ¼ 0.04 3 0.03 < p < ¼ 0.09
0.01 < p $ 0.01#n 2 0.01 < p < ¼ 0.02 2 0.01 < p < ¼ 0.03
p $ 0.01 1 < ¼ 0.01 1 <¼ 0.01

TABLE 6—CONSEQUENCE RANGES FOR TWO VALUES OF THE MULTIPLIER n

n¼ 2 n¼ 3

Equation Score Consequence (USD Million) Score Consequence (USD Million)

4
100#n < Consequence 6 1.6 < Consequence 6 8.1 < Consequence
100#n3 < Consequence $ 100#n4 5 0.8 < Consequence < ¼ 1.6 5 2.7 < Consequence < ¼ 8.1
100#n2 < Consequence $ 100#n3 4 0.4 < Consequence < ¼ 0.8 4 0.9 < Consequence < ¼ 2.7
100#n < Consequence $ 100#n2 3 0.2 < Consequence < ¼ 0.4 3 0.3 < Consequence < ¼ 0.9
100 < Consequence $ 100#n 2 0.1 < Consequence < ¼ 0.2 2 0.1 < Consequence < ¼ 0.3
Consequence $ 100 1 < ¼ 0.1 1 <¼ 0.1

RM rankings are arbitrary; whether something is ranked first or sis, we introduced a multiplier n that determines the range for each
last, for example, depends on whether or not one creates an category. We retained ranges for the first category for both conse-
increasing or a decreasing scale. How can a methodology that quence and probability. For the categories that are not at the end-
exhibits such a gross deficiency be considered an industry best points of the axes, n will determine the start value and end value of
practice? Would such a method stand up to scrutiny in a court of the range. For example, with n ¼ 2, the second probability category
law? Imagine an engineer defending their risk-management plan in Fig. 1 has a value range from 0.01 to 0.02 (0.01 to 0.01 " n). For
by noting it was developed by use of an RM, when the lawyer the category at the end of the axis, n will affect only the start value
points out that simply changing the scale would have resulted in a of the range, which must not exceed unity (n ¼ 3.15) for the proba-
different plan. What other engineering best practices produce dif- bility axis and must not exceed USD 20 million (n ¼ 3.6) for the
ferent designs simply by changing the scale or the units? consequence axis. Tables 5 and 6 show the probability and conse-
Instability Because of Categorization. RMs categorize conse- quence ranges, respectively, for n ¼ 2 or n ¼ 3.
quence and probability values, but there are no well-established We vary the multiplier and observe the effect on risk ranking
rules for how to do the categorization. Morgan et al. (2000) rec- for both ascending and descending scores. While varying the mul-
ommended testing different categories because no single category tiplier for one axis, the ranges in the other axis are kept in a
breakdown is suitable for every consequence variable and proba- default value (Fig. 3) and constant. Because Table 1 gives the
bility within a given situation. consequence value in ranges, we use the midpoint10 consequence
Following this recommendation, we tried to find the best cate- value within the range for each outcome, as shown in Table 7.
gories for the RM in Fig. 1 by examining the sensitivity of the Given a single consequence value for each outcome, the categori-
risk ranking to changes in category definitions. To ease this analy- zation instability analysis can be performed. Figs. 7 and 8 show
how the risk ranking is affected by change in n.
Figs. 7 and 8 indicate that except where consequence is in
ascending order, the risk prioritization is a function of n. This is
TABLE 7—CASE FOR CATEGORIZATION INSTABILITY problematic because the resulting risk ranking is unstable in the
ANALYSIS sense that a small change in the choice of ranges, which is again
Outcome Consequence (USD Million) Probability
10
Severe Losses 3 40% For the practicality of the analysis, we assume that for blowout consequence, the ratio of
the range’s high value to low value is the same as for Category 5 (high value ¼ 4 " low
Well Control 12.5 10% value). Thus, the range is USD 20 to 80 million, and the middle value is USD 50 million. No
Blowout 50 5% matter which value is chosen to represent the high-end consequence, the instability
remains and is equally severe.

April 2014 SPE Economics & Management 61

 Probability Sensitivity Analysis Probability Sensitivity Analysis
(Scale 6-5-4-3-2-1) (Scale 1-2-3-4-5-6)
Multiplier, n Multiplier, n
2 2.2 2.4 2.6 2.8 3 2 2.2 2.4 2.6 2.8 3
1 1
Risk Ranking

Risk Ranking
2 2

3 3

Severe Losses (SL) Well Control (WC) Blowout (BO) Severe Losses (SL) Well Control (WC) Blowout (BO)

Rank for each outcome: Rank for each outcome:

n = 2.50 SL = 1| WC = 3 | BO = 2 n = 2.15 SL = 2 or 3| WC = 1 | BO = 2 or 3

n = 2.20 SL = 1 or 2| WC = 3 | BO = 1 or 2
n = 2.55 SL = 2 or 3| WC = 2 or 3 | BO = 1
n = 2.25 SL = 1| WC = 2 | BO = 3

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Probability Probability

Fig. 7—Sensitivity of risk prioritization to probability categorization.

arbitrary, can lead to a large change in risk prioritization. Thus, we we surveyed, Category 3 on the likelihood axis spans 0.9 to 18%
again see that the guidance provided by RMs is arbitrary, being of the total range.
determined by arbitrary design choices that have no scientific basis.
For each SPE paper that used at least one quantitative scale,
Table 8 shows percentage of the domain for Categories 1 through Relative Distance is Distorted. Lie Factor. According to Table
4, with Category 5 being excluded because it was often unbounded. 7, the consequence of a blowout is four times that of well control
The left-hand table is for the frequency and the right-hand table is (50/12.5). However, the ratio of their scores in the RM is only 1.2
for the consequence. For example, the probability categories for (6/5). The difference in how risk is portrayed in the RM vs. the
paper SPE 142854, in ascending order, cover 0.001, 0.1, 0.9, and expected values can be quantified by use of the LF.
99% of the domain. The consequence categories for paper SPE The LF was coined by Tufte and Graves-Morris (1983) to
142854, in ascending order, cover 0.1, 0.9, 9, and 90% of the describe graphical representations of data that deviate from the
domain. principle that “the representation of numbers, as physically meas-
That categories cover different amounts of the total range is ured on the surface of the graphic itself, should be directly propor-
clearly a significant distortion. In addition to this, the size of the tional to the quantities represented” (Tufte and Graves-Morris
categories varies widely across papers. For example, in the papers 1983). This maxim seems intuitive, but it is difficult to apply to

Consequence Sensitivity Analysis Consequence Sensitivity Analysis

(Scale 6-5-4-3-2-1) (Scale 1-2-3-4-5-6)
Multiplier, n Multiplier, n
2 2.2 2.4 2.6 2.8 3 3.2 3.4 3.6 2 2.2 2.4 2.6 2.8 3 3.2 3.4 3.6
1 1
Risk Ranking

Risk Ranking

2 2

3 3

Severe Losses (SL) Well Control (WC) Blowout (BO) Severe Losses (SL) Well Control (WC) Blowout (BO)

Rank for each outcome:

n = 3.0 SL = 1 or 2| WC = 1 or 2 | BO = 3

n = 3.2 SL = 3| WC = 1 | BO = 2

n = 3.4 SL = 2| WC = 3 | BO = 1

0 5 10 15 20 80
Consequence (USD Million)

Fig. 8—Sensitivity of risk prioritization to consequence categorization.

62 April 2014 SPE Economics & Management

 TABLE 8—PERCENTAGE OF TOTAL RANGE FOR EACH RATING

Frequency Consequence

Paper Number Rating Percentage of Range Paper Number Rating Percentage of Range

SPE 127254 1 0.95% SPE 142854 1 0.10%

SPE 127254 2 0.02% SPE 142854 2 0.90%
SPE 127254 3 2.36% SPE 142854 3 9.00%
SPE 127254 4 96.67% SPE 142854 4 90.00%
SPE 142854 1 0.001% SPE 98423 1 1.00%
SPE 142854 2 0.10% SPE 98423 2 4.00%
SPE 142854 3 0.90% SPE 98423 3 15.00%
SPE 142854 4 99.00% SPE 98423 4 81.00
SPE 98852 1 0.04%
SPE 98852 2 1.96%
SPE 98852 3 18.00%
SPE 98852 4 80.00%
SPE 162500 1 0.09%
SPE 162500 2 0.90%
SPE 162500 3 9.00%
SPE 162500 4 90.00%

data that follow an exponential relationship, for example. Such definition might be the maximum LF for any category. Table 9
cases often use log plots, in which the same transformation is shows the result of our average LF calculation. All reviewed RMs
applied to all the data. However, RMs can distort the information use infinity as the upper bound on the consequence axes. This
they convey at different rates within the same graphic. gives infinite LFs. However, in summarizing the LF for the
Slightly modifying the Tufte and Graves-Morris (1983) defini- reviewed papers in Table 9, we have chosen to use the second
tion, we define LF as largest category as the upper limit for the consequences. This
obviously understates the actual LFs in the reviewed papers.
DVm;n All nine papers have an LF greater than unity along at least
LFm;n ¼ ; ........................... ð1Þ
DSm;n one axis. Paper SPE 142854, for example, has an LF of 96 on the
consequence axis and 5,935 on the probability axis.
where Many proponents of RMs extol their visual appeal and result-
jVn ' Vm j jSn ' Sm j ing alignment and clarity in understanding and communication.
DVm;n ¼ ; DSm;n ¼ ; and n > m: However, the commonly used scoring system distorts the scales
Vm Sm and removes the proportionality in the input data. How can it be
The LF is thus calculated as the change in value (of probability argued that a method that distorts the information underlying an
or consequence) over the m and n categories divided by the engineering decision in nonuniform and uncontrolled ways is an
change in score over the m and n categories. In calculating the industry best practice? The burden of proof is squarely on the
LF, we use the midpoint across the value and probability ranges shoulders of those who would recommend the use of such meth-
within each category. ods to prove that these obvious inconsistencies do not impair deci-
From Fig. 1, the score of the consequence axis at m ¼ 3 is sion making, much less improve it, as is often claimed.
S ¼ 3 and at n ¼ 4 is S ¼ 4. By use of the midpoint value for each
category, LF3,4 ¼ 11.4 ¼ (j3,000–625j/625)/(j4–3j/3). The inter-
pretation of this is that the increase in the underlying consequence A Consistent Approach to Risk Management
values is 11.4 times larger than an increase in the score. The motivation for writing this paper was to point out the gross
None of the 30 papers reviewed included enough quantitative inconsistencies and arbitrariness embedded in RMs. Given these
information for the LF to be calculated. We define the LF for an problems, it seems clear to us that RMs should not be used for deci-
RM as the average of the LFs for all categories. An alternative sions of any consequence. Our pointing out that RMs produce arbi-
trary rankings does not require us to provide another method in
their place, any more than we would be required to suggest new
medical treatments to argue against the once popular practice of
TABLE 9—LF FOR NINE SPE PAPERS bloodletting. The arbitrariness of RMs is not conditional on whether
or not other alternatives exist. Nevertheless, the question is bound to
Average of Each Category be raised, and thus this section provides a brief set of references to
what we consider to be a consistent approach to risk management.
Paper Number LF of Consequence LF of Probability Risk management is fundamentally about decision-making. The
objective of the risk-management process is to identify, assess,
SPE 142854 96 5,935 rank, and inform management decisions to mitigate risks. Risks can
SPE 86838 30 – only be managed through our decisions, and the risk-management
SPE 98852 745 245 objectives are best achieved with processes and tools that support
SPE 121094 5 – high-quality decision-making in complex and uncertain situations.
SPE 74080 94 – For centuries people have speculated on how to improve deci-
SPE 123861 28 113 sion making, and a formal approach to decision and risk analysis
SPE 162500 85 389 can be traced through the works of Bayes and Price (1763), Lap-
lace (1902), Ramsey (1931), De Finetti (1931, 1937), von Neu-
SPE 98423 16 –
mann and Morgenstern (1944), Bernoulli (1954), and Savage
IPTC 14946 1 3
(1954). Over the last several decades, important supporting fields

April 2014 SPE Economics & Management 63

have been integrated to provide a discipline, decision analysis,11 approach that generates some discussion of the risks in a particu-
with the objective of informing and supporting decision making lar activity will be helpful. The fact that these flaws have not been
in complex and uncertain environments (e.g., such as many of the raised as an issue before is evidence that RMs obscure rather than
risk-management decisions we face in the O&G industry). Good enlighten communication. Instead of RMs, the O&G industry
general references on decision analysis include Howard (2007) should rely on risk- and decision-analytic procedures that rest on
and Clemen and Reilly (2013), whereas Bratvold and Begg more than 250 years of scientific development and understanding.
(2010) provide a recent O&G-oriented introduction.
There are also a number of excellent publications that apply the
fundamental concepts of decision analysis to the types of problems References
to which RMs are commonly applied. A small, but relevant, sample Alkendi, M.Y.M.S. 2006. ADNOC Environmental Impact Severity Ma-
include Paté-Cornell and Fischbeck’s (1994) work on performing a trix, an Innovative Impact Rating Matrix. Presented at the SPE Interna-
probabilistic risk analysis of failure of the exterior surface tiles on tional Health, Safety & Environment Conference, Abu Dhabi, 2–4
the US space shuttle orbiter; Paté-Cornell’s (2002) use of probabil- April. SPE-98852-MS. http://dx.doi.org/10.2118/98852-MS.
istic risk analysis to solve government safety decisions; Chapman Al-Mitin, A.W., Sardesai, V., Al-Harbi, B. et al. 2011. Risk Based Inspec-
and Ward’s (2003) discussion of project risk management; and tion (RBI) of Aboveground Storage Tanks to Improve Asset Integrity.
Hubbard’s (2009) introduction of several alternatives to RMs. Presented at the International Petroleum Technology Conference,
These authors warn that the processes and tools they discuss, illus- Bangkok, Thailand, 15–17 November. IPTC-14434-MS. http://dx.doi.
trate, and recommend are not perfect and should be used in accord- org/10.2523/14434-MS.
ance with sound decision-analysis principles. However, unlike API RP 581, Risk-Based Inspection Technology. 2008. Washington DC:
RMs, the processes and tools drawn from decision analysis are con- API.
sistent, do not carry the inherent flaws of the RMs, and provide Areeniyom, P. 2011. The Use of Risk-Based Inspection for Aging Pipe-
clarity and transparency to the decision-making situation. Our best lines in Sirikit Oilfield. Presented at the International Petroleum Tech-
chance for providing high-quality risk-management decisions is to nology Conference, Bangkok, Thailand, 15–17 November. IPTC-
apply the well-developed and consistent set of processes and tools 14946-MS. http://dx.doi.org/10.2523/14946-MS.
embodied in decision science. Bayes, T. and Price, R. 1763. An Essay Towards Solving a Problem in the
Doctrine of Chances. By the Late Rev. Mr. Bayes, F. R. S. Communi-
cated by Mr. Price, in a Letter to John Canton, A. M. F. R. S. Philosoph-
Discussion and Conclusions ical Transactions 53: 370–418. http://dx.doi.org/10.1098/rstl.1763.0053.
As suggested by Hubbard (2009), for any risk-management method Bensahraoui, M. and Macwan, N. 2012. Risk Management Register in
used in the O&G industry, we should ask: “How do we know it Projects & Operations. Presented at the Abu Dhabi International Petro-
works?” If we cannot answer that question, then our first risk-man- leum Conference and Exhibition, Abu Dhabi, 11–14 November. SPE-
agement priority should be to find and adopt a risk-management 162500-MS. http://dx.doi.org/10.2118/162500-MS.
method that does work. RMs are among the most commonly used Berg, F.R. 2001. The Development and Use of Risk Acceptance Criteria
tools for risk prioritization and management in the O&G industry. for the Construction Phases of the Karsto Development Project in Nor-
The matrices are recommended by several influential standardiza- way. Presented at the SPE/EPA/DOE Exploration and Production
tion bodies, and our literature search found more than 100 papers in Environmental Conference, San Antonio, Texas, 26–28 February.
the OnePetro database that document the application of RMs in a SPE-66516-MS. http://dx.doi.org/10.2118/66516-MS.
risk-management context. However, we are not aware of any pub- Bernoulli, D. 1954. Exposition of a New Theory on the Measurement of
lished empirical evidence showing that they actually help in man- Risk. Econometrica 22 (1): 23–36. http://dx.doi.org/10.2307/1909829.
aging risk or that they improve decision outcomes. Bower-White, G. 2012. Demonstrating Adequate Management of Risks:
In this paper, we have illustrated and discussed inherent flaws The Move from Quantitative to Qualitative Risk Assessments. Pre-
in RMs and their potential impact on risk prioritization and miti- sented at the SPE Asia Pacific Oil and Gas Conference and Exhibition,
gation. Inherent dangers such as risk-acceptance inconsistency, Perth, Australia, 22–24 October 2012. SPE-158114-MS. http://
range compression, centering bias, and category-definition bias dx.doi.org/10.2118/158114-MS.
were introduced and discussed by Cox et al. (2005), Cox (2008), Bratvold, R.B. and Begg, S.H. 2010. Making Good Decisions. Richardson,
Hubbard (2009), and Smith et al. (2009). We have also addressed Texas: Society of Petroleum Engineers.
several previously undocumented RM flaws: ranking reversal, Budescu, D.V., Broomell, S., and Por, H.H. 2009. Improving communica-
instability resulting from categorization differences, and the LF. tion of uncertainty in the reports of the intergovernmental panel on cli-
These flaws cannot be corrected and are inherent to the design mate change. Psychological Science 20 (3): 299–308. http://
and use of RMs. dx.doi.org/10.1111/j.1467-9280.2009.02284.x.
The ranking produced by RMs was shown to be unduly influ- Campbell, N.W., Tate, D.R.D. 2006. Attacking Metropolitan Driving Haz-
enced by their design, which is ultimately arbitrary. No guidance ards with Field-Proven Practices. Presented at the SPE International
exists regarding these design parameters because there is very lit- Health, Safety & Environment Conference, Abu Dhabi, 2–4 April.
tle to say. A tool that produces arbitrary recommendations in an SPE-98566-MS. http://dx.doi.org/10.2118/98566-MS.
area as important as risk management in O&G should not be con- Chapman, C. and Ward, S. 2003. Project Risk Management: Processes,
sidered an industry best practice. Techniques and Insights, 2nd edition. New York: Wiley.
There are undoubtedly O&G professionals who recognize and Clare, J.B. and Armstrong, L.J. 2006. Comprehensive Risk-Evaluation
understand the inherent inaccuracy of RMs and take steps to avoid Approaches for International E&P Operations. SPE Proj Fac & Const
these dangers, to the extent that this is even possible. However, 1 (3): 1-6. SPE-98679-PA. http://dx.doi.org/10.2118/98679-PA.
we suspect that this does not apply to the majority of O&G profes- Clemen, R.T. and Reilly, T. 2013. Making Hard Decisions with Decision-
sionals who develop or use RMs, on the basis of the literature tools, 3rd edition. Cengage Learning.
review and extensive data gathering conducted for this paper. Fur- Coakley, B., Baraka, C., and Shafi, M. 2003. Enhancing Rig Site Risk
thermore, if the initial assessment of risk is not based on meaning- Awareness. Presented at the SPE/IADC Middle East Drilling Technol-
ful measures, the risk-management decisions are likely to address ogy Conference and Exhibition, Abu Dhabi, 20–22 October. SPE-
the wrong problems, resulting in a waste of money and time (at 85299-MS. http://dx.doi.org/10.2118/85299-MS.
best) and in severe HSE issues (at worst). Cox Jr., L.A. 2008. What’s Wrong with Risk Matrices? Risk Analysis 28
It may be true that using RMs to analyze and manage risks is (2): 497–512. http://dx.doi.org/10.1111/j.1539-6924.2008.01030.x.
better than doing nothing [though even that may be debatable, as Cox Jr., L.A., Babayev, D., and Huber, W. 2005. Some limitations of qual-
pointed out by Cox (2008) and Hubbard (2009)]. Indeed, any itative risk rating systems. Risk Analysis 25 (3): 651–662. http:dx.
doi.org/10.1111/j.1539-6924.2005.00615.x.
11
Howard (1988) defined the profession of decision analysis as a result of his work to
Da Silva, E.N., Neto, L.M., and Amaral, S.P. 2010. LOPA as a PHA com-
merge decision theory and systems engineering. plementary tool: a Case Study. Presented at the SPE International

64 April 2014 SPE Economics & Management

 Conference on Health, Safety and Environment in Oil and Gas Explo- ver, 30 October–2 November. SPE-146845-MS. http://dx.doi.org/10.
ration and Production, Rio de Janeiro, 12–14 April. SPE-127254-MS. 2118/146845-MS.
http://dx.doi.org/10.2118/127254-MS. Piper, J.W. and Carlon, J.R. 2000. Application and Integration of Security
De Finetti, B. 1931. Probabilism. Erkenntnis 31 (2–3): 169-223. (Septem- Risk Assessment Methodologies and Technologies into Health, Safety
ber 1989). http://dx.doi.org/10.1007/BF01236563. and Environmental (SHE) Programs. Presented at the SPE Interna-
De Finetti, B. 1937. Foresight: Its Logical Laws, Its Subjective Sources, trans. tional Conference on Health, Safety, and Environment in Oil and Gas
H.E. Kyburg Jr., Vol. 7, 1–68. Paris: Presses Universitaires de France. Exploration and Production, Stavanger, 26–28 June. SPE-61149-MS.
Dethlefs, J. and Chastain, B. 2012. Assessing Well-Integrity Risk: A Qual- http://dx.doi.org/10.2118/61149-MS.
itative Model. SPE Drill & Compl 27 (2): 294–302. SPE-142854-PA. Poedjono, B., Chinh, P.V., Phillips, W.J., and Lombardo, G.J. 2009. Anti-Colli-
http://dx.doi.org/10.2118/142854-PA. sion Risk Management for Real-World Well Placement. Presented at the
Duguay, A., Baccino, B., and Essel, P. 2012. From 360 Deg Health Asia Pacific Health, Safety, Security and Environment Conference, Jakarta,
Safety Environment Initiatives on the Rig Site to Structured HSE 4–6 August. SPE-121094-MS. http://dx.doi.org/10.2118/121094-MS.
Strategy: A Field Case in Abu Al Bukhoosh Field. Presented at the Poedjono, B., Conran, G., Akinniranye, G. et al. 2007. Minimizing the Risk
Abu Dhabi International Petroleum Conference and Exhibition, Abu of Well Collisions in Land and Offshore Drilling. Presented at the SPE/
Dhabi, 11–14 November. SPE-161547-MS. http://dx.doi.org/ IADC Middle East Drilling and Technology Conference, Cairo, 22–24
10.2118/161547-MS. October. SPE-108279-MS. http://dx.doi.org/10.2118/108279-MS.
Howard, R.A. 1988. Decision Analysis: Practice and Promise. Management Pritchard, D., York, P.L., Beattie, S., and Hannegan, D. 2010. Drilling
Science 34 (6): 679–695. http://dx.doi.org/10.1287/mnsc.34.6.679. Hazard Management : The Value of Risk Assessment. World Oil 231
Howard, R.A. 2007. The Foundations of Decision Analysis Revisited. In (10): 43–52. http://www.successful-energy.com/wp-content/uploads/
Advances in Decision Analysis: From Foundations to Applications, 2011/02/WO1010_Series_2_Final.pdf.
Chap. 3, 32–56. Cambridge University Press. http://dx.doi.org/10. Ramsey, F.P. 1931. Truth and Probability. In The Foundations of Mathe-
1017/CBO9780511611308.004. matics and other Logical Essays, ed. R.B. Braithwaite, Chap. 7,
Hubbard, D.W. 2009. The Failure of Risk Management: Why It’s Broken 156–198. Routledge and Kegan Paul Ltd. (repr. Routledge, 2013).
and How to Fix It. Hoboken, New Jersey: John Wiley & Sons, Inc. Reynolds, J.T. 2000. Risk Based Inspection—Where Are We Today? Pre-
ISO 31000:2009, Risk Management—Principles and Guidelines. 2009. sented at CORROSION 2000, Orlando, Florida, 26–31 March. NACE-
Washington DC: American National Standards Institute. 00690.
ISO/IEC 31010:2009, Risk Management—Risk Assessment Techniques. Samad, S.A., Al Sawadi, O.S., Afzal, M., and Khan, N. 2010. Risk Register
2009. Washington DC: American National Standards Institute. and Risk Ranking of Non-Integral Wells. Presented at the Abu Dhabi
Jones, D.W. and Bruney, J.M. 2008. Meeting the Challenge of Technology International Petroleum Exhibition and Conference, Abu Dhabi, 1–4
Advancement—Innovative Strategies for Health, Environment and November. SPE-137630-MS. http://dx.doi.org/10.2118/137630-MS.
Safety Risk Management. Presented at the SPE International Confer- Samad, S.A., Tarmoom, I.O., Binthabet, H.A. et al. 2007. A Comprehen-
ence on Health, Safety, and Environment in Oil and Gas Exploration sive Approach to Well Integrity Management. Presented at the SPE
and Production, Nice, France, 15–17 April. SPE-111769-MS. http:// Middle East Oil and Gas Show and Conference, Kingdom of Bahrain,
dx.doi.org/10.2118/111769-MS. 11–14 March. SPE-105319-MS. http://dx.doi.org/10.2118/105319-MS.
Kinsella, K.G., Kinn, S.J., Thomassen, O. et al. 2008. Development of a Savage L.J. 1954. The Foundations of Statistics. New York: John Wiley &
Software Tool, EPRA, for Early Phase Risk Assessment. Presented at Sons (repr. Dover Publications, 1972).
the SPE International Conference on Health, Safety, and Environment Smith, E.D., Siefert, W.T., and Drain, D. 2009. Risk matrix input data
in Oil and Gas Exploration and Production, Nice, France, 15–17 April. biases. Systems Engineering 12 (4): 344–360. http://dx.doi.org/
SPE-111549-MS. http://dx.doi.org/10.2118/111549-MS. 10.1002/sys.20126.
Laplace, P.S. 1902. A Philosophical Essay on Probabilities, first edition. Smith, N., BuTuwaibeh, O.I., Cruz, I.C., and Gahtani, M.S. 2002. Risk-
New York: John Wiley & Sons. Based Assessment (RBA) of a Gas/Oil Separation Plant. Presented at
Lee, N.M. 2009. Safety Cultures—Pushing the Boundaries of Risk Assess- the SPE International Conference on Health, Safety and Environment
ment. Presented at the Asia Pacific Health, Safety, Security and Envi- in Oil and Gas Exploration and Production, Kuala Lumpur, 20–22
ronment Conference, Jakarta, 4–6 August. SPE-123457-MS. http:// March. SPE-73897-MS. http://dx.doi.org/10.2118/73897-MS.
dx.doi.org/10.2118/123457-MS. Theriau, R., Rispler, K., and Redpath, S. 2004. Controlling Hazards
Leistad, G.H. and Bradley, A. 2009. Is the Focus too Low on Issues That through Risk Management - A Structured Approach. Presented at the
Have a Potential to Lead to a Major Incident? Presented at Offshore SPE International Conference on Health, Safety, and Environment in
Europe, Aberdeen, 8–11 September. SPE-123861-MS. http://dx.doi. Oil and Gas Exploration and Production, Calgary, 29–31 March. SPE-
org/10.2118/123861-MS. 86838-MS. http://dx.doi.org/10.2118/86838-MS.
McCulloch, B.R. 2002. A Practical Approach to SH&E Risk Assessments Truchon, M., Rouhan, A., and Goyet, J. 2007. Risk Based Inspection
within Exploration & Production Operations. Presented at the SPE Approach for Topside Structural Components. Presented at the Off-
International Conference on Health, Safety and Environment in Oil shore Technology Conference, Houston, 30 April–3 May. OTC-
and Gas Exploration and Production, Kuala Lumpur, 20–22 March. 18912-MS. http://dx.doi.org/10.4043/18912-MS.
SPE-73892-MS. http://dx.doi.org/10.2118/73892-MS. Tufte, E.R. and Graves-Morris, P.R. 1983. The Visual Display of Quantita-
McDermott, M.S. 2007. Risk Assessment (Hazard Management) Process tive Information, Vol. 31. Chesire, Connecticut: Graphics Press.
is a Continual Process, Not a One Off. Presented at the SPE Asia Pa- Valeur, J.R. and Clowers, M. 2006. Structure and Functioning of the ISO
cific Health, Safety, and Security Environment Conference and Exhibi- 14001 and OHSAS 18001 Certified HSE Management System of the
tion, Bangkok, Thailand, 10–12 September. SPE-108853-MS. http:// Offshore Installation South Arne. Presented at the SPE International
dx.doi.org/10.2118/108853-MS. Health, Safety & Environment Conference, Abu Dhabi, 2–4 April.
NORSOK Standard S-012, Health, Safety and Environment (HSE) in con- SPE-98423-MS. http://dx.doi.org/10.2118/98423-MS.
struction-related activities. 2002. Rev. 2, August. Oslo, Norway: Nor- von Neumann, J. and Morgenstern, O. 1944. Theory of Games and Eco-
wegian Technology Centre (NTS). nomic Behavior. Princeton, New Jersey: Princeton University Press.
Paté-Cornell, M.-E. and Fischbeck, P.S. 1994. Risk Management for the Windschitl, P.D. and Weber, E.U. 1999. The interpretation Of “likely”
Tiles of the Space Shuttle. Interfaces 24 (1): 64–86. http://dx.doi.org/ depends on the context, but “70%” is 70%—right? The influence of
10.1287/inte.24.1.64. associative processes on perceived certainty. J Exp Psychol: Learn
Paté-Cornell, E. 2002. Risk and Uncertainty Analysis in Government Mem Cogn 25 (6): 1514–1533.
Safety Decisions. Risk Analysis 22 (3): 633–646. http://dx.doi.org/ Zainuddin, Z.M., Samad, A.H., Hasyim, I.B. et al. 2002. Conducting Pub-
10.1111/0272-4332.00043. lic Health Risk Assessment in a Remote Drilling Site in Indonesia: An
Petrone, A., Scataglini, L., and Cherubin, P. 2011. B.A.R.T (Baseline Risk Experience. Presented at the SPE International Conference on Health,
Assessment Tool): A Step Change in Traditional Risk Assessment Safety and Environment in Oil and Gas Exploration and Production,
Techniques for Process Safety and Asset Integrity Management. Pre- Kuala Lumpur, 20–22 March 2002. SPE-74080-MS. http://dx.doi.org/
sented at the SPE Annual Technical Conference and Exhibition, Den- 10.2118/74080-MS.

April 2014 SPE Economics & Management 65

 A p p e n d i x 3 0 Se l e c t e d SPE Pa p e r s a n d T h e i r Fl a w s

TABLE A-1—30 SPE PAPERS AND (SOME OF) THEIR INHERENT FLAWS

Risk-Acceptance Category-Definition
Paper Year Author(s) Inconsistency Bias Centering Bias Scoring System

Corrosion 2000 2000 Reynolds, J.T. Yes Yes Not available Ascending
SPE 61149 2000 Piper and Carlon Yes Yes Not available Descending
SPE 66516 2001 Berg, F.R. Yes Yes Not available Ascending
SPE 73892 2002 McCulloch Yes Yes Not available –
SPE 73897 2002 Smith et al. Yes Yes Yes Ascending
SPE 74080 2002 Zainuddin et al. Yes Yes Yes Descending
SPE 85299 2003 Coakley et al. Yes Yes Not available Ascending
SPE 86838 2004 Theriau et al. Yes Yes Not available Descending
SPE 98566 2006 Campbell and Tate Yes Yes Not available Ascending
SPE 98852 2006 Alkendi Yes Yes Not available Ascending
SPE 98679 2006 Clare and Armstrong Yes Yes Not available Ascending
SPE 98423 2006 Valeur and Clowers Yes Yes Not available Ascending
SPE 108279 2007 Poedjono et al. Yes Yes Not available Ascending
SPE 108853 2007 McDermott Yes Yes Not available Ascending
SPE 105319 2007 Samad et al. Yes Yes Not available Ascending
OTC 18912 2007 Truchon et al. Yes Yes Yes Descending
SPE 111549 2008 Kinsella et al. Yes Yes Not available Ascending
SPE 121094 2009 Poedjono et al. Yes Yes Not available Ascending
SPE 123457 2009 Lee Yes Yes Not available Ascending
SPE 123861 2009 Leistad and Bradley Yes No Not available Ascending
SPE 111769 2009 Jones and Bruney Yes Yes Not available Descending
SPE 137630 2010 Samad et al. Yes Yes Not available Ascending
SPE 127254 2010 Da Silva et al. Yes Yes Not available Ascending
IPTC 14434 2011 Al-Mitin et al. Yes Yes Not available Ascending
IPTC 14946 2011 Areeniyom Yes Yes Not available Ascending
SPE 146845 2011 Petrone et al. Yes Yes Yes Ascending
SPE 158114 2012 Bower-White Yes Yes Not available Ascending
SPE 162500 2012 Bensahraoui and Macwan Yes Yes Yes Ascending
SPE 142854 2012 Dethlefs and Chastain Yes Yes Yes Ascending
SPE 161547 2012 Duguay et al. Yes Yes Not available Ascending

Philip Thomas is a PhD candidate in petroleum investment ber in the Society of Decision Professionals and was made a
and decision analysis at the University of Stavanger and is member of the Norwegian Academy of Technological Scien-
advised by R.B. Bratvold. He is interested in the applications of ces for his work in petroleum investment and decision analysis.
decision analysis and real-options analysis in the O&G industry. Bratvold holds a PhD degree in petroleum engineering and a
Thomas holds a master’s degree in petroleum engineering master’s degree in mathematics, both from Stanford Univer-
from the University of Stavanger and a bachelor’s degree in sity, and obtained business and management-science edu-
petroleum engineering from Bandung Institute of Technology, cation from INSEAD and Stanford University.
Indonesia. J. Eric Bickel is an assistant professor in both the Graduate Pro-
Reidar B. Bratvold is a professor of petroleum investment and gram in Operations Research/Industrial Engineering (Depart-
decision analysis at the University of Stavanger and at the Nor- ment of Mechanical Engineering) and the Department of
wegian University of Science and Technology in Trondheim, Petroleum and Geosystems Engineering at the University of
Norway. His research interests include decision analysis, valua- Texas at Austin. In addition, he is a fellow with the Center for
tion of risky projects, portfolio analysis, real-option valuation, Petroleum Asset Risk Management. Bickel’s research interests
and behavioral challenges in decision making. Before enter- include the theory and practice of decision analysis and its
ing academia, Bratvold spent 15 years in the industry in various application in the O&G industry. Before returning to aca-
technical and management roles. He is a coauthor of the SPE demia, he was a Senior Engagement Manager for Strategic
Primer Ma king Go o d Dec isio ns. Bratvold is an associate editor Decisions Group. Bickel holds a master’s degree and a PhD
for SPE Ec o nom ic s & Ma na g em ent and has twice served as degree from the Department of Engineering-Economic Sys-
an SPE Distinguished Lecturer. He is a fellow and board mem- tems at Stanford University.

66 April 2014 SPE Economics & Management

View publication stats