Safety and Health at Work 9 (2018) 265e276

Contents lists available at ScienceDirect

Safety and Health at Work

journal homepage: www.e-shaw.org

Original Article

An Analytic Framework to Assess Organizational Resilience

Riccardo Patriarca 1, *, Giulio Di Gravio 1, Francesco Costantino 1, Andrea Falegnami 1,
Federico Bilotta 2
1
Department of Mechanical and Aerospace Engineering, Sapienza University of Rome, Rome, Italy
2
Department of Anesthesiology, Critical Care and Pain Medicine, Sapienza University of Rome, Rome, Italy

a r t i c l e i n f o a b s t r a c t

Article history: Background: Resilience engineering is a paradigm for safety management that focuses on coping with
Received 14 April 2017 complexity to achieve success, even considering several conflicting goals. Modern sociotechnical systems
Received in revised form have to be resilient to comply with the variability of everyday activities, the tight-coupled and under-
5 September 2017
specified nature of work, and the nonlinear interactions among agents. At organizational level, resilience
Accepted 24 October 2017
can be described as a combination of four cornerstones: monitoring, responding, learning, and
Available online 2 November 2017
anticipating.
Methods: Starting from these four categories, this article aims at defining a semiquantitative analytic
Keywords:
Complex system framework to measure organizational resilience in complex sociotechnical systems, combining the
Resilience resilience analysis grid and the analytic hierarchy process.
Resilience engineering Results: This article presents an approach for defining resilience abilities of an organization, creating a
Safety management structured domain-dependent framework to define a resilience profile at different levels of abstraction,
Sociotechnical system and identifying weaknesses and strengths of the system and potential actions to increase system’s
adaptive capacity. An illustrative example in an anesthesia department clarifies the outcomes of the
approach.
Conclusion: The outcome of the resilience analysis grid, i.e., a weighed set of probing questions, can be
used in different domains, as a support tool in a wider Safety-II oriented managerial action to bring safety
management into the core business of the organization.
Ó 2017 Occupational Safety and Health Research Institute, Published by Elsevier Korea LLC. This is an
open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction contractors, etc.) and the relationships among them (behaviors, ac-
tivities, communication, etc.) [3,4], even in terms of work attitudes,
Several changes are occurring in the business and socioeconomic skill levels, and occupational roles. The technical subsystem consists
environment, contributing to increase the complexity of work of the artefacts used to convert inputs to outputs, including mate-
practices in sociotechnical systems. A sociotechnical system is a rials, machines, territory, and techniques used to accomplish the
purposeful structure consisting of interrelated and interdependent tasks of the organization itself [5], through work processes, roles, and
social and technical elements influencing one another, directly or procedures strictly related to with the social subsystem. Examples of
indirectly, to maintain their activity and the existence of the system sociotechnical systems are the air traffic management system, a
itself to purse its goal [1,2]. Sociotechnical systems are made up of production plant, maritime operations, and healthcare practices.
people who produce products or services using some technology, For these systems, following the technological advances in the last
affecting the operation and the appropriateness of the technology as decades, procedures and organizational activities have become
well as the actions of the people operating it. Their interactions partly increasingly more sophisticated, leading to increase system’s
comprise linear causeeeffect relationships and partly nonlinear, complexity. Individuals have to thus cope with a challenging envi-
complex, even unpredictable ones. In a sociotechnical system, two ronment, whose complexity inherently emerges in everyday activ-
sub-systems are tightly interrelated: the social and the technical ities, as well as in design, implementation, and maintenance.
systems. The social subsystem comprises of the people working Traditional approaches in safety management focus on adverse
within the organization at different levels (employees, managers, outcomes (accidents, incidents) trying to reduce their numbers

* Corresponding author. Department of Mechanical and Aerospace Engineering, Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, Italy.
E-mail address: [email protected] (R. Patriarca).

2093-7911/$ e see front matter Ó 2017 Occupational Safety and Health Research Institute, Published by Elsevier Korea LLC. This is an open access article under the CC BY-NC-
ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
https://doi.org/10.1016/j.shaw.2017.10.005
266 Saf Health Work 2018;9:265e276

and/or limiting their effects. Following a linear approach, also functional properties of the system and be used as a decision
labeled Newtonian reasoning [6], traditional safety management support tool to address criticalities, in a safety management
aims at reducing the variability of everyday work through stricter perspective?
regulations and procedures to constrain work activities. This
approach relies on the causality credo, i.e., the possibility to find a Starting from the traditional formulation of RAG, this article
specific causeeeffect link for any event, decomposing the system proposes an innovative analytic hierarchy process (AHP) frame-
in its constituent parts, and generally thinking about humans as a work to represent the effects of each resilience ability (responding,
source of failures, as summarized by the concept of Safety-I [7]. monitoring, learning, and anticipating) to organizational resilience,
Nevertheless, Safety-I assumptions are effective only if system’s by user-friendly indicators that can be analyzed at different levels
complexity is limited. In complex operating scenarios, such as of abstraction. This framework allows dealing with the complexity
nowadays sociotechnical systems, there are many strong in- of the system itself and identifies weakness and strengths of work
teractions among human, technical, organizational, procedural, activities, which remains a puzzling activity for managing system’s
and regulative aspects that lead to lose an effective linear under- resilience.
standing of the whole system functioning. The acknowledgment of The remainder of the article is structured as follows. Section 2
the inevitable complex nature of sociotechnical systems pushed describes materials and methods of this article, presenting the
safety management research toward focusing on a different relevance of the four cornerstones of resilience and detailing their
perspective, i.e., aiming at understanding everyday activities. In meaning in different work domains. Section 3 discusses the tradi-
everyday activities, work-as-done is different from work-as-imag- tional RAG structure and its enhanced approach based on AHP,
ined, and this difference requires humans’ adaptation to cope with including an illustrative case study in an anesthesia department.
variable operating scenarios and safely ensure system’s produc- The Section 4 offers critical reflections on the outcomes of the
tivity safely. This adaptation is embodied in the concept of study, clarifying the potential for further research. Lastly, the Sec-
resilience. tion 5 summarizes the outcomes of the study.
Resiliencedat largedcan be defined as a system property that
confers the ability to remain intact and functional despite the 2. Materials and methods
presence of threats. Thus, resilience engineering is the ability to
engineer systems that are resistant to disturbances. As proved by Resilience is defined as the system’s ability to retain or recover
successful applications in several sociotechnical domains, mainly a rapidly stable condition, enabling pursuing its normal func-
humans’ resilience acquires a crucial role to maintain system’s tioning during, and following, any expected or unexpected dis-
effectiveness and positive capabilities. Cognitive tasks (e.g., decision- ruptions. Resilience reflects the ability of trading-off between
making, inference, reasoning, learning, etc.) become central for safety and production objectives, managing conflicting goals’
maintaining safe and efficient operations. pressures [15e17].
There are several examples detailing the relevance of resilience
in sociotechnical systems, for example, adaptive capacity of air 2.1. Resilience in four cornerstones
traffic controllers and pilots has been modeled to understand how
variability is managed to prevent runway incursions during normal In the field of resilience engineering, several research efforts
runway operations [8]. Similarly, in the rail traffic domain, the aimed at linking resilience to some clearly describable features. One
crucial role of traffic controllers in coping with increasingly com- of the most diffused approaches led to decomposing resilience into
plex automation and ensuring system resilience has been four cornerstones, acknowledging that a resilient system needs to
acknowledged [9]. Resilience becomes necessary to deal with the be conveniently balanced among them [14]. Following this view,
synchronic and diachronic couplings of system functioning, as the system must be able to respond to any type of events
proved by a resilience engineering perspective used to analyze the (addressing the actual), to monitor underway evolutions (address-
Fukushima disaster [10]. Note that resilience is not by itself the ing the critical), to anticipate future threats and opportunities
solution for improving system performance, but by adopting a (addressing the potential), and to learn from past failures and
systemic approach it will be possible to understand where resil- successes (addressing the factual). The validity of these four cor-
ience is necessary and where it is not, as theoretically discussed in nerstones has been widely recognized for representing successfully
the healthcare domain [11]. how people feel comfortable with unexpected and unforeseen
For this purpose, one of the principal efforts in the discipline of events in everyday work [18] and to promote proactive strategies
resilience engineering consists of anchoring the concept of resil- for managing daily operations [19]. The same cornerstones have
ience to some clearly describable features to develop operational been used to define a dedicated framework, which considers legal,
means for its assessment [12,13]. institutional, organizational, and procurement aspects of societal
Resilience engineering suggests four abilities to be considered resilience [20]. Even in the air traffic management context, the
for achieving resilient performance, i.e., responding, monitoring, SCALES framework [21] and the ADAPTER questionnaire [22], both
learning, and anticipating. The resilience analysis grid (RAG) aims starting from the four cornerstones, have been used to retrieve
at measuring how resiliently an organization performs in everyday resilience indicators and patterns to identify criticalities and the
work based on assessing how the organization responds, monitors, focus of more detailed approaches.
learns and anticipates in everyday activities. Therefore, the RAG is
developed as a questionnaire to produce a resilience profile of the 2.2. Responding: dealing with the actual
organization, in terms of the four abilities of resilience, the so-
called four cornerstones [14]. Responding to external and internal disturbances or, more in
With the purpose of measuring organizational resilience in a general, to any input or signal is a crucial need for every system. A
complex system following a systematic and structured approach, system has to be able to distinguish between what is urgent and
this article aims at answering the following research question: what is important and provide effective and on-time responses to
maintain productivity and ensure safety. This distinction is
- How should a method for measuring organizational resilience necessary to ensure that the real-time competences that needed to
in a sociotechnical system be designed to consider the cope with unanticipated or extreme events at the sharp-end are
 R. Patriarca et al / An Analytic Framework to Assess Organizational Resilience 267

not eroded in the continuous attempt to predetermine corre- 2.5. Learning: dealing with the factual
sponding responses to all the possible events [23]. Responding is
linked to sharp-end decision-making: managing potential vari- Traditional safety management bases learning on adverse
ability of patient status and defining the cognitive threshold over events, i.e., accidents, incidents, near misses. Normally, in safe
which requiring additional resources is generally a difficult task systems, this assumption clashes with the principles of effective
[24]. Several examples of prompt responding to system disruption learning: the presence of reasonable opportunities to learn (a
can be retrieved in the health-care literature, especially in the reasonable number of situations); the similarity of situations to
emergency departments, which demands for strong responding avoid having only event-specific conclusions (the events should be
capabilities due to the inevitable variability of everyday practices comparable in some sense); the opportunity to verify whether the
[25e27]. lessons have been learned [35]. Therefore, it becomes necessary to
As a summary, the analysis of this cornerstone should aim at integrate the learning from accidents, or even more extreme events
providing material for understanding what capacities are required such as emergencies and catastrophes, with learning from minor-
to properly respond to everyday situations. consequence events or even from normal functioning, namely
everyday activities [14]. It is from lessons learned that an individ-
2.3. Monitoring: dealing with the critical ual, or even an organization, increases his/her ability to manage
threats, adopting or adapting (refer to Weick’s sensemaking theory
The ability of responding is linked to the capability of under- [36,37]) his knowledge creatively to manage unexpected situations
standing actual threats timely and precisely. An effective [38,39]. At the same time, an effective learning should lead to
responding has thus to be linked to an effective monitoring of choose worthy synthetic indicators and give means to anticipate
system’s status and of its operational environment, i.e., acquiring potential future threats and opportunities.
signals related to both positive and negative situations.
In traditional safety management [7], monitoring consists of
2.6. RAG in literature
adopting indicators such as the number of adverse events: mainly
accidents [28] and also other less-consequence events, such as in-
In accordance with the need to understand and monitor resil-
cidents and near misses [29], are used generally as feedback signals
ience [40], intended as a combination of the four cornerstones,
of system safety. On the contrary, for resilience engineering, it has
Hollnagel originally developed a questionnaire-based tool in 2011,
been claimed the need to gather data from intermediate status,
the RAG [41]. Hollnagel himself suggests that the RAG is not an off-
monitoring how the system actually performs during its normal
the-shelf tool, but it is rather intended as a basis to develop ques-
functioning [30]. This target is generally acquired by leading in-
tions, which must be specific for the system under analysis.
dicators, i.e., precursors to events that are not happened yet, and/or
One of the first applications of RAG is discussed in the domain of
lagging indicators, i.e., observations of events already happened.
rail traffic management, aimed at identifying a set of potential
Note that a lagging indicator related to the short-term staffing
vectors for improvement, with particular reference to the need of
change (e.g., numbers of workers suffering from fatigue-related
integrating trade-offs and making easier to assess and control nu-
injuries) might be considered a leading indicator for the need of
ances of the abstract cornerstones [42].
systemic changes [31].
For the purpose of integrated planning of maintenance and
The analysis of this cornerstone should aim at defining relevant
operations in an offshore oil and gas company, 16 critical functions
indicators to gain knowledge of current working conditions.
have been assessed by a list of RAG-based questions. The study
points out the benefits of adopting RAG after a system change to
2.4. Anticipating: dealing with the potential
understand the effects of the change on the organization at least
once a year [43]. A case study for the air traffic management system
In a traditional sense, the ability of anticipating is related to
details the application of RAG to measure organizational resilience,
forecasting future events based on historic data [7]. Even if these
in collaboration with several domain experts. The study confirms
representations offer an overall understanding of safety levels [32],
the benefits of repeating the RAG several times during a long period
they might fail at anticipating hidden and newdnever happened
of time [44]. In the health-care domain, the RAG has been used to
beforedthreats or even at determining the need for buffers and
derive probes for each cornerstone, creating an interview script
reserves to face unexpected pressures.
iteratively revised based on responses to pilot interviews as well as
What makes anticipating different from monitoring is generally
feedback by human factors and resilience engineering experts [45].
the different time scale of observations and the related point of
In addition, a recent study based on RAG in Polish enterprises of
view. The ability of anticipating generally extends the focus of
different sizes and activities shows how the implementation of
monitoring’s leading indicators to cope with long-term changes,
occupational safety and health management system does not
threats, opportunities, and environmental potential status. At
actually affect safety and resilience levels [46]. For the purpose of
different organizational levels, monitoring and anticipating might
measuring team resilience, the RAG has been even updated by two
become complementary. For sharp-end operators, coping with
additional dimensions, i.e., leadership and cooperation, suggesting
near-term issues is an operational everyday activity (monitoring);
its use for contextualizing actual events and having it filled after
at the blunt-end, more attention is paid generally to strategic
specific incidents [22]. This recent interest in the RAG formulation
decision-making, relying on long-term analysis (anticipating). The
motivates the need to explore its building process to increase its
distinction is not a dichotomy because strategic decisions affect and
potential benefits for sociotechnical systems.
are affected by operational behaviors and vice versa [33]. Typical
anticipating features are related to understand if, and how, future
events like threats and opportunities are modeled (e.g., qualita- 2.7. RAG’s building process
tively, quantitatively, etc.), understanding which efforts in terms of
expertise and funding are employed for this purpose [34]. The RAG consists of four phases [41]:
The analysis of the ability to anticipate should look at detecting
upcoming threats and opportunities timely and efficiently and in- - RAG 1. Define and describe system’s structure, boundaries, time
crease system’s preparedness. horizon, people, and resources involved.
268 Saf Health Work 2018;9:265e276

This preliminary phase is necessary to restrict the field of variable, i.e., defining the effects on organizational resilience of each
application of the analysis, contextualizing the set of questions to category and question [50]. To adopt the AHP, it is assumed that data
ask. collection will take advantage of a structured survey. In terms of the
four traditional phases of the RAG, the AHP aims at structure of the
- RAG 2. Select relevant questions for corresponding relevant items second phase of the RAG, i.e., RAG 2, through a systematic frame-
of the studied system. work to determine and weigh the final set of questions. The AHP’s
outcomes constitute an input data for the traditional third phase,
This phase requires large efforts to define a proper set of ques- RAG 3 and obviously inherently affect RAG 4.
tions which would be reliable enough to describe the system, not The AHP steps, contextualized for the RAG application, can be
being so large to require unmanageable processing times. Even if described as follows:
there are several abstract commonalities across different high-risk
domains of what the cornerstones mean [47,48], it is hardly - AHP 1. Define the problem and determine the kind of knowledge
possible to define general questions valid for each domain. Since sought.
resilience is strongly related to the system’s purpose for which it
has to be assessed, the four cornerstones have to be detailed in a The purpose of this phase consists of determining the factors
domain-specific context. This phase is usually a recursive activity to that influence organizational resilience. This step thus corresponds
formulate the more appropriate questions, involving several sub- to the preliminary activities performed for the second phase of
jectematter experts (SMEs). RAG, i.e., RAG 2.

- RAG 3. Rate the questions for each cornerstone. - AHP 2. Structure the decision hierarchy.

Once the questions are finalized, it is necessary to identify a pool This step allows defining the hierarchy of the problem, from the
of people working in the system who could answer them. In this top with the goal of the process, through intermediate levels
sense, techniques to collect data are unconstrained; it is thus (criteria on which subsequent elements depend) to the lowest level
possible to develop informal phone or face-to face interviews [46], (which usually is the set of alternatives but here represents the
open questions in focus groups [42], narratives [45] or surveys. The different safety events). In term of RAG, this phase consists of
surveys represent the most frequent technique [22,43,44] as they defining a hierarchy of resilience abilities: the overall organiza-
allow faster processing times, even if they do not allow dynamic tional resilience lies at the top level, while the four cornerstones lie
interactions in data collection: for the interviewee to recall expe- at the intermediate levels, and at the lowest levels, there are the so-
rience and for the interviewer to fill eventual gaps and in- called categories and, eventually, sub-categories. The lowest level
consistencies in the judgments. shall include the final questions. The need to develop questions in a
hierarchical structure affects the traditional second phase of RAG,
- RAG 4. Combining the ratings. implying a systematic reorganization of the RAG questions. Even if
this task may generate additional efforts, it actually offers an
In this phase, data are usually presented in a star plot where inherent support in developing the questions themselves and
each axis corresponds to the variables used to rate each corner- guiding the analysts to follow a structured approach by the support
stone. It might be possible to obtain an additional four spoke star of a multilevel reasoning [51]. Note that this step has to take into
chart combining the individual cornerstones’ charts, to gain a account the experts’ judgments and consists generally of an itera-
description of the organizational resilience. tive procedure (see Fig. 1 details of a real scenario, as discussed in
Note that the star plot is not a measure of resilience per se, but it the case study).
represents in a compact way how the abilities are rated in a specific
time moment: it represents a snapshot of organizational resilience - AHP 3. Elaborate a set of pairwise comparison matrices.
under specific conditions. Therefore, it might be performed multi-
ple times to follow and monitor performance developments. The Each element in an upper level compares to the elements in the
RAG can be used firstly to determine where the system is; then to level immediately below. Pairwise comparisons express a relative
spot where the system should be; finally to understand how the judgment between two elements in a 9-degree scale of importance
system may reach a target status. (1 ¼ equal, 3 ¼ moderate, 5 ¼ strong, 7 ¼ very strong, 9 ¼ extreme)
Nevertheless, the traditional outcomes of the RAG fail to and the reciprocal value is assigned to the inverse comparison. In
represent the relative importance of each category and questions the elements of the pairwise comparison matrix A, aij represent the
and their impact on the organizational resilience (the corner of a importance of i-th criterion over j-th one. Since the elements satisfy
star plot do not offer any insights on these aspects). Although these the constraint aij  aji ¼ 1 and aii ¼ 1, only n (n 1)/2 comparisons
open issues have been widely discussed according to a theoretical are necessary. At this step, the consistency index IC has to be
perspective acknowledging the need to understand the domain- evaluated based on the eigenvalues l of the matrix A [49].
specific balance among the abilities [41], there is no operational If IC < 0.1, the matrix is consistent, and the judgments can be
solution in literature. This research aims to fill this gap, through an considered not contradictory. For the purpose of applying the RAG,
innovative analytic structured formulation for the RAG based on the comparison at the intermediate level, i.e., the four cornerstones,
the AHP. might be too abstract for evaluation and thus their weights might
be considered equals. Lower levels of abstractions might be instead
2.8. The AHP for the RAG evaluated following traditional AHP theory.

The AHP, introduced by Saaty [49], is a multicriteria decision- - AHP 4. Use the priorities coming from the comparisons to
making technique, which aims at reducing complex decisions to a weigh the priorities in the level immediately below.
series of pairwise comparisons, in a user-friendly formulation. In
terms of the RAG, the AHP might be used to combine objectively This weighing process has to be repeated for each element, also
subjective judgments and determine the relative weight of each including the weight of the upper category to gain an overall
 R. Patriarca et al / An Analytic Framework to Assess Organizational Resilience 269

indication of their relative importance. The hierarchy composition been restricted to the Level 2 of the framework. The comparisons
principle has to be applied to determine the importance of every at Level 1 would have been too abstract to obtain meaningful and
element for organizational resilience, which represents the main unambiguous responses; and the comparison at Level 3 would
goal in the AHP language. Therefore, starting from the lowest level, have been of limited added value, considering the reduced size, i.e.,
the local weight of every element has to be multiplied for local two or three questions for each Level 2 category (also considering
weight of the related parent element, transforming them in global the extra resources necessary in this additional weighing process).
weights. Consequently, the weights of Level 1 and Level 2 have been thus
In terms of the RAG, once the AHP is applied and the elements’ assigned automatically as equally important, respectively 0,25 for
weights are identified dboth local and globaldat each level of the each cornerstone, and 0,50 or 0,33 for each questions, in case the
hierarchy, it is possible to continue with the traditional survey, as category included respectively two or three questions.
discussed in the third phase of RAG, i.e., RAG 3. The AHP’s structure
will inherently modify and enhance the data management in the
fourth phase of RAG, allowing detailed comparisons among ques-
tions and their effects on the system, as clarified by the illustrative
example in Section 3.

3. Results

This section details an illustrative application in the health-care

domain of the innovative RAG analytic framework based on the
AHP. Mainly due to the instrumental, procedural, and organiza-
tional continuous evolution, new challenges constantly affected
health-care system. The human interaction with these factors
became steadily more complex, requiring resilience as a means to
manage the variability of everyday performance [52,53]. Under-
standing how resilient is a health-care organization might repre-
sent a valuable contribution to determine how effectively the
organization manages the complexity of everyday work, defining
weaknesses and strengths.

3.1. RAG phase 1

This illustrative application focuses on an anesthesia depart-

ment of a hospital located in Central Europe. The aim of the study
consists of assessing the organizational resilience of neuro-
anesthetists working in the same department, by a RAG-based
weighted survey. The target of the study consists of assessing
how the individuals working in the department contribute to
organizational resilience in terms of the four cornerstones, appro-
priately contextualized for the need of a neuroanesthesia domain.

3.2. RAG phase 2

The AHP guides the development of this phase. Starting from a

literature review on resilience engineering, specifically focused on
previous applications of the RAG, an initial set of categories (Level 2 Fig. 1. Process flow for phase 2 of RAG based on the AHP.
of hierarchy) and questions (Level 3) has been created. In the hi- AHP, analytic hierarchy process; RAG, resilience analysis grid.
erarchy, formally the four cornerstones occupy Level 1, while the
overall organizational resilience represents Level 0.
A focus group (one staff and two resident neuroanesthetists and Afterward, based on the final structure of Level 2 (see Table 1),
two researchers working in the field of resilience engineering) seven neuroanesthetists (the ones involved in the focus group and
tuned this initial framework to adapt it properly for the needs of the in the preliminary test) help defining the weights for the cate-
specific neuroanesthesia department. gories, individually filling an online survey. This allows pairwise
After this phase, a preliminary version of the survey has been comparison among the categories, based on a 9-point scale (A and
tested with four physicians: two staff and two resident neuro- B equally important, A slightly more important than B, A mildly
anesthetists. This test led to modify the framework slightly, more important than B, A much more important than B, A abso-
reducing the size of questions and rephrasing some of them to lutely more important than B, and vice versa). It has been chosen
make them clearer and avoid misunderstanding. to ask for individual answers, rather than collective ones (e.g., in a
Fig. 1 summarizes the process and Table 1 details the final list of focus group) to avoid that individual attitude to team working, or
categories (Level 2 of the hierarchy). Note that the staff doctors organizational role might have influenced judgments. Then, a
answering the preliminary survey were not the same involved in grouping strategy for the judgments has been applied relying on
the focus group. Theoretically, it would have been necessary to geometric mean [54]. These surveys were not anonymous, in order
develop pairwise comparisons at each level of the framework. to ease the verification of judgments’ coherence following an
However, for operational reasons, the AHP weighing process has iterative approach to verify IC < 0.1 for each judgment, as described
270 Saf Health Work 2018;9:265e276

in [54]. The outcome of this phase represented the final set of anesthetist needs support to further investigate the patient’s clinical
weighed questions, as summarized in Fig. 2 and more specifically conditions, but he/she does not receive a proper support, mainly
in Fig. 3. due to delays and problems for interacting with other departments.

Table 1
List of RAG categories for each cornerstone (AHP Level 2)

Responding Monitoring Anticipating Learning

R1. International guidelines M1. List of questions for preoperative A1. Improvement projects L1. Discussion of clinical cases
examination
R2. Internal protocols M2. Preoperative specialist support A2. Research activities L2. The register of clinical cases
R3. Resources availability in expected situations M3. Intraoperative risk analysis A3. Introduction of new equipment L3. Event analysis
R4. Resources availability in M4. Identification of intraoperative A4. Participation in updating plans and L4. Practical activities
unexpected situations complications procedures
R5. Experience M5. Instrumental support A5. Planning of training activities L5. Theoretical teaching
R6. Discretionary power M6. Postoperative check A6. Surgeries scheduling L6. Mentoring
R7. Teamwork d A7. Communication L7. Learning check
R8. Roles division d d L8. Experiences in other institutes
AHP, analytic hierarchy process; RAG, resilience analysis grid.

3.3. RAG phase 3 In terms of Anticipating, the interviewees show a strong confi-
dence in communication with surgeons and nurses. Communica-
At this step, it has been necessary to rate each element at Level 3 tion’s high score shows how the verbal interactions among
of the hierarchy, i.e., the probing questions, by a specific survey. The different practitioners allows anticipating future threats and miti-
survey was anonymous, gathering few profile data on the inter- gating emerging risks in patient safety, especially in case of shift
viewee (sex, age, role in the hospital, and years of experience). The rotation over the same intervention.
questionnaire included a description of the meaning of each cate- Furthermore, additional observations can be drawn from the
gory and a 5-point Likert scale, linguistically adapted for each results of the RAG, through a cluster analysis. Thinking of resilience
question, as shown in Fig. 3. The survey has been submitted to 12 as a multilevel property of a system, the purpose of this cluster
neuroanesthetists (six staff and six resident) of different ages and analysis consists of identifying how the individuals contribute to
experience, working in the same department. organizational resilience, assuming that the organization provides
the context for individuals or group of them to make local and
3.4. RAG phase 4 global system properties emerge [55]. On this path, even if the
sample is limited, it allows interesting observations about how
This phase summarizes the answers gathered from the 12 resident and staff neuroanesthetists differently contribute to the
neuroanesthetists involved in the third phase of RAG, suggesting an overall organizational resilience. This cluster analysis is intended to
interpretation of the results. As discussed in literature, one of the tailor the definition of criticalities and the development of potential
main advantages of using the RAG consists of monitoring the mitigating actions to manage system’s resilience.
relative evolution of organizational resilience (e.g.,) repeating the More specifically, at Level 1, one can observe that resident
assessment over time [45]. However, for this illustrative applica- neuroanesthetists offer a lower contribution to organizational
tion, a single preliminary assessment will be discussed. Neverthe- resilience (see Fig. 6), intuitively related to their experience in the
less, other potential ways to interpret RAG data are presented, field. This abstract observation can be broken down taking advan-
taking advantage of the AHP structure. At Level 1 of the hierarchy, it tage of the AHP hierarchy to gain more detailed and helpful
is possible to represent how the organization performs respectively insights.
for each cornerstone (see Fig. 4). As a preliminary observation, one Fig. 7 offers a two-dimensional representation to give an over-
can note how the system appears to be much more able to respond view of the categories where the staff neuroanesthetists’ contri-
and monitor, rather than to anticipate and learn. This abstract bution to overall resilience prevails on the residents’ one (the area
conclusion, obtained by combining all the judgments at each level, under the dashed line), and vice versa (the area above the dashed
can be discussed further in terms of the categories for each line). This analysis shows that in some specific categories the
cornerstone, as detailed in Fig. 5. This analysis allows detailed ob- resident neuroanesthetists are more resilient than the staff ones,
servations, detecting which categories mainly contribute to overall i.e., L.4 (practical teaching activities); R.1 (international guidelines);
resilience. For example, in terms of learning, anesthetists feel quite M.3 (intraoperative risk analysis), M.6 (postoperative check); A.2
comfortable about event analysis, i.e., reactive and proactive in- (research activities), A.6 (surgeries scheduling).
vestigations undertaken following an accident or a risky event to A focus group with three doctors (the same involved in the focus
detect contributing factors. About the ability of responding, expe- group during phase two of RAG, see Section 3.2) allowed drawing
rience has a crucial role, as expected. In addition, the study shows interesting conclusions. For example, about Research
that resources availability cannot be considered the most critical activities (A.2), the focus group confirms that resident neuro-
aspect, regardless if considering expected or unexpected events. anesthetists generally play a major sharp-end role in trials and
About Monitoring, even if real-time observations of complica- experimental projects. These activities force them to keep abreast
tions during intraoperative phase seem to be effective, the analysis of anesthesiology scientific evolution, increasing their anticipating
shows some drawbacks about the preoperative specialist support. skills, i.e., their resilience. This factor has been perceived as positive
This conclusion considers those frequent scenarios where the for the benefit of the organization, indicating the need at a
 R. Patriarca et al / An Analytic Framework to Assess Organizational Resilience 271

Fig. 2. AHP structure and weights for each cornerstone, category, and question.
AHP, analytic hierarchy process; RAG, resilience analysis grid.

managerial level to promote an active involvement of staff anes- results of the RAG motivate a managerial action to encourage the
thetists in research activities. involvement of people with fresh eyes, i.e., resident doctors, in
On the contrary, about the aspect Participation in updating medium and long-term planning. The proposal of these two
plans and procedures (A.4), staff neuroanesthetists prevail on managerial actions aim at fostering multiskilled anesthetists with
resident ones, since the latter are not usually involved in the interchangeable roles and distributed workloads, in line with
planning processes. Staff doctors, whose expert opinion is 1950s seminal observations in mining systems [1], and as recently
generally predominant, feel more comfortable to manage pro- confirmed in the health-care domain (see [56] for an example in
cedures and plans in which they have been directly involved. The operating room and [57] for information technology
272 Saf Health Work 2018;9:265e276

Fig. 3. Details of nomenclature and example of questions.

AHP, analytic hierarchy process.

management). At Level 3, further observations can be drawn, action to underline the need of an accurate and detailed preop-
based on scores of specific probing questions. For example, local erative visit, regardless of the patients’ conditions, has been
improvement actions might consist of increasing the number of strongly encouraged.
teaching activities with mannequins (L.4.1) and stressing the
relevance of using a complete list of questions during the pre- 4. Discussion
operative visit (M.1.1). For this latter aspect, the results of the RAG
motivated a detailed investigation of everyday work, which Measuring resilience is a challenging task, mainly because it re-
confirmed that the preoperative visit is frequently underrated for quires an empirical, context-dependent measurement tool. On this
apparently healthy patients, generating potential complications path, since resilience refers to something that a system does, rather
during the intraoperative phase for underestimated risk factors. than something it has, the RAG in combination with the AHP defines
At management level, the development of a dedicated training a resilience profile that may offer insights to understand when and
 R. Patriarca et al / An Analytic Framework to Assess Organizational Resilience 273

of respondents and/or to assess resilience profiles for different

groups of respondents (as discussed in the case study).
As emerged in the case study, there are no predetermined set of
indicators, but they have to be specific to the context and the risks
that exist in a specific scenario. For this purpose, the involvement of
SMEs working within the organization (with a job related to the
target of the assessment) acquires a crucial role, as understandable
through the lessons learned from the case study presented in the
article. It is important to gain a strong consensus among SMEs in
the process of defining the final set of questions, since this will
clearly affect the definition of the resilience profile and the iden-
tification of the mitigating actions. Sharp-end operators have to be
involved among the SMEs in the development of the RAG, in order
to get a meaningful and reliable set of questions which can define a
resilience profile that is able to describe how the system cope with
the complexity of everyday activities. While developing the RAG in
combination with the AHP, the analyst has to limit the efforts of the
Fig. 4. The RAG score for the four cornerstones. respondents, trying to define a reasonable number of questions. As
RAG, resilience analysis grid.
emerged from the case study presented in this article, the most
where it is possible to intervene to manage work activities in a way time-consuming phase is the weighing process of each category
capable of sustaining the required operations. through specific pairwise comparison matrices. For this purpose,
When assessing an inherent property of a system, e.g. its resil- we developed an online user-friendly survey taking advantage of
ience, it is important to avoid making important what is measured, Google Survey, which allowed respondents to answer questions
rather than measuring what is important, defining relevant proxy directly from their computer, tablet, or smartphones, limiting the
measures as for the proposed approach. Therefore, it is useful to perceived efforts in the task. The format of the tool was extremely
remember that the RAG profile, even in combination with the AHP, appreciated by the respondents, who were willing to contribute
does not offer an absolute rating of system’s abilities but a relative even in the iterative procedure to maintain the coherence of the
assessment support tool to compare different RAG profiles. The pairwise comparison judgments, as explained in Section 3.2.
RAG can be used to assess how the resilience profile of an organi- Similarly, another survey to rate each probing question has been
zation changes over time, using it repeatedly with the same group developed in Google Survey, as needed for the third phase of RAG.

Fig. 5. The RAG score for each category at Level 2 of the hierarchy.
RAG, resilience analysis grid.
274 Saf Health Work 2018;9:265e276

a renormalization of the other weights, generating a potential

reduction of the number of questions to be answered.
In line with the traditional RAG approach, multiple represen-
tations may be helpful to describe the relative status of the
system or the effects of a systemic change in the resilience profile.
In this context, a limitation of the illustrative case study presented
in the article is the limited size of the sample, which has to be
increased in future research. Furthermore, a larger sample would
have allowed multiple cluster analyses to compare different groups
of respondents and gain multiperspective insights. Similarly,
although the illustrative case study presents a single assessment of
the resilience profile, capable of generating valuable insights, a
systemic analysis would largely benefit of multiple assessments
Fig. 6. The RAG score for each category at Level 1 of the hierarchy and comparison over time to assess the evolution of the resilience profile of the
between the resilience profiles of staff and resident neuroanesthetists. organization (as a whole or for different clusters).
RAG, resilience analysis grid. In terms of future research, for potential adaptation of the
framework presented in this article to different domains, it is
The framework and the structure of the online surveys can be
helpful to recall that the analyst might even not limit the RAG to the
adopted in different departments or organizations, but requiring
four cornerstones, extending or modifying the categories, if other
the contextualization of the domain-dependent categories and
abilities are specific to a work domain.
variables.
Future research might even explore the possibility to extend the
Even if the AHP does not limit the number of hierarchical levels,
analysis by a network structure, based on the analytic network
lessons learned from the case studies suggest that a three-level
process, allowing assessment of interaction and dependence of
structure adequately represent the need of a sociotechnical
higher level elements in a hierarchy on lower elements. However,
assessment. More specifically, Level 2 is a reasonable level of
in this case, the analytic structure will be more complicated,
abstraction to weight the factors, with the possibility to avoid too
requiring more time-consuming pairwise comparisons. In terms of
much abstract weighting process (at Level 1, for the four corner-
methodological formulation, other approaches to gather data could
stones), or too detailed ones (at Level 3, for the questions). Also note
be evaluated as well, i.e., interviewing people to get open answer
that the AHP could be used to rank factors and exclude from the
and ensure the interviewees completely understand the questions.
analysis those categories that receive substantially lower weights
Even in this case, more resources will be requested for gathering
than others, under the assumption that their effect on organiza-
data, in addition to the need of experienced interviewers.
tional resilience is limited. This assumption will eventually require

Fig. 7. The RAG score for each category at Level 2 of the hierarchy and comparison between the resilience profiles of staff and resident neuroanesthetists.
RAG, resilience analysis grid.
 R. Patriarca et al / An Analytic Framework to Assess Organizational Resilience 275

Lastly, future research may focus on the development of a fuzzy technologies in health in the context of wicked problems and sociotechnical
theory. J Am Med Inform Assoc 2007;14:746e55. https://doi.org/10.1197/
AHP, to deal with the problem arising when making static and
jamia.M2462.
deterministic quantitative assessment. Owing to the linguistic [5] Pasmore W, Francis C, Haldeman J. Sociotechnical systems: a North American
values and variables necessary for the application of the RAG, the reflection on empirical studies of the seventies. Hum Relat 1982;35:1179e
decision of taking the fuzziness into account might provide 204. https://doi.org/10.1177/001872678203501207.
[6] Dekker S. Patient safety e a human factor approach. Boca Raton (FL): CRC
potentially less biased and more realistic assessments. Press e Taylor & Francis Group; 2011.
[7] Hollnagel E. Safety-I and safety-II (the past and future of safety management).
5. Conclusions Farnham (UK): Ashgate; 2014.
[8] Patriarca R, Di Gravio G, Costantino F. A Monte Carlo evolution of the Func-
tional Resonance Analysis Method (FRAM) to assess performance variability in
Within the scope of resilience engineering for safety manage- complex systems. Saf Sci 2017;91:49e60. https://doi.org/10.1016/j.ssci.
ment, this article suggests an approach to measure resilience by 2016.07.016.
[9] Ferreira PNP, Balfe N. Lect Notes Comput Sci (Including Subser Lect Notes Artif
means of the RAG, a questionnaire-based tool based on the four Intell Lect Notes Bioinformatics). The contribution of automation to resilience
cornerstones of resilience. Since resilience is a property of the in rail traffic control, 8532 LNAI; 2014. p. 458e69. https://doi.org/10.1007/
system and is strictly dependent on the features of the system itself, 978-3-319-07515-0_46.
[10] Hollnagel E, Fujita Y. The Fukushima disaster-systemic failures as the lack of
the theory of resilience engineering does not prescribe a standard resilience. Nucl Eng Technol 2013;45:13e20. https://doi.org/10.5516/
meaning of the four cornerstones or even a certain desired balance NET.03.2011.078.
or proportion among them. Adopting the AHP, this article develops [11] Amalberti R. Resilience and safety in health care: marriage or divorce? In:
Hollnagel E, Braithwaite J, Wears RL, editors. Resilient Heal. Care 2013. p. 27e38.
a structured approach to determine an in-depth meaning of resil-
[12] McDonald N. Organisational resilience and industrial risk. In: Hollnagel E,
ience abilities, creating a hierarchy based on different level of ab- Woods DD, Leveson N, editors. Resil. Eng. Concepts Precepts. Ashgate Pub-
stractions. The RAG’s survey data, systematically weighed following lishing, Ltd; 2006. p. 155e80.
[13] Patterson M, Deutsch ES. Safety-I, Safety-II and resilience engineering. Curr
the AHP, represent a resilience profile that can be used to assess the
Probl Pediatr Adolesc Health Care 2015;45:382e9. https://doi.org/10.1016/
system status and to addressdif necessarydpriorities for inter- j.cppeds.2015.10.001.
vention (both for safety and performance enhancement), as [14] Hollnagel E. Prologue: the scope of resilience engineering. In: Hollnagel E,
confirmed by the outcomes of the illustrative case study presented Pariès J, Woods DD, Wreathall J, editors. Resil. Eng. Pract. A Guideb., MINES
ParisTech. France: Ashgate Publishing, Ltd; 2011 p. xxixexxxix.
in this article. It is important to observe that due to its inherent [15] Woods DD. Essential characteristics of resilience. In: Hollnagel E, Woods DD,
questionnaire-based nature, the RAG relies on truthful answers Leveson N, editors. Resil. Eng. Concepts Precepts. Ashgate Publishing, Ltd;
and, consequently, on an interviewee who wants to exaggerate in 2006. p. 21e34.
[16] Dekker S. Resilience engineering: chronicling the emergence of confused
one sense or the other his/her work conditions, which can seriously consensus. In: Hollnagel E, Woods DD, Leveson N, editors. Resil. Eng. Concepts
affect its value. For this purpose, even if increasing the number of Precepts. Ashgate Publishing, Ltd; 2006. p. 77e94.
participants may reduce considerably the risk of facing this bias, [17] Nemeth CP. Resilience engineering: the birth of a notion. In: Hollnagel E,
Nemeth CP, Dekker S, editors. Resil. Eng. Perspect. Vol. 1-Remain. Sensitive to
there is the need to develop the RAG as part of a wider Safety-II Possibility Fail. Ashgate Publishing, Ltd; 2008. p. 3e10.
oriented managerial action with focus on sharp-end activities. In [18] Rankin A, Lundberg J, Woltjer R. A framework for learning from adaptive
this local analysis, the outcome of RAG could even motivate performance. In: Nemeth CP, Hollnagel E, editors. Becom. Resilient. Ashgate
Publishing, Ltd; 2014. p. 79e96.
detailed work-as-done research, using methods to analyze systemic [19] Praetorius G, Hollnagel E. Control and resilience within the maritime traffic
interactions and functional properties, e.g., through the Functional management domain. J Cogn Eng Decis Mak 2014;8:303e17. https://doi.org/
Resonance Analysis Method (FRAM) [58] and/or the Systems- 10.1177/1555343414560022.
[20] Becker P, Abrahamsson M, Tehler H. An emergent means to assurgent ends:
Theoretic Accident Modeling and Processes/Systems Theoretic
societal resilience for safety and sustainability. In: Nemeth CP, Hollnagel E,
Process Analysis (STAMP/STPA) [59]. On this path, the RAG may editors. Becom. Resilient. Ashgate Publishing, Ltd; 2014. p. 1e12.
represent a tool to bring safety management into the core business [21] Herrera IA, Pasquini A, Ragosta M, Vennesland A. The SCALES framework for
of the organization to support the development of a healthy and identifying and extracting resilience related indicators: preliminary findings
of a go-around case study. SIDs 2014-Proc. SESAR Innov. Days, EURO-
efficient work environment. CONTROL; 2014.
[22] Van Der Beek D, Schraagen JM. ADAPTER: analysing and developing adapt-
Conflicts of interest ability and performance in teams to enhance resilience. Reliab Eng Syst Saf
2015:141. https://doi.org/10.1016/j.ress.2015.03.019.
[23] Pariès J. Lessons from the hudson. In: Hollnagel E, Pariès J, Woods DD,
All authors have no conflict of interest to declare. Wreathall J, editors. Resil. Eng. Pract. A Guideb. Ashgate Publishing, Ltd; 2011.
p. 9e28.
[24] Cuvelier L, Falzon P. Coping with uncertainty. Resilient Decis Anaesth 2011.
Acknowledgments [25] Wears RL, Perry S, McFauls A. “ Free Fall ” e a case study of resilience, its
degradation, and recovery in an emergency department. In: Proc Second Resil
The authors thank Giulia Nanti for her her support in the early Eng Symp Novemb 2006 Sophia-Antip [Online] 2006. p. 8e10.
[26] Nemeth C, Wears R, Woods D, Hollnagel E, Cook R. Minding the gaps: creating
development of a previous version of the case study and both the resilience in health care. In: Henriksen K, Battles JB, Keyes MA, Grady ML,
staff- and resident neuro-anesthetists, who voluntarily participated editors. Advances in Patient Safety: New Directions and Alternative Ap-
in the case study discussed in the paper. proaches (Vol. 3: Performance and Tools). Rockville (MD): Agency for
Healthcare Research and Quality (US); 2008.
[27] Wachs P, Saurin TA, Righi AW, Wears RL. Resilience skills as emergent phe-
References nomena: a study of emergency departments in Brazil and the United States.
Appl Ergon 2016. https://doi.org/10.1016/j.apergo.2016.02.012.
[1] Trist EL, Bamforth KW. Some social and psychological consequences of the [28] Valdez Banda OA, Goerlandt F, Montewka J, Kujala P. A risk analysis of winter
Longwall method of coal-getting: an examination of the psychological situa- navigation in Finnish sea areas. Accid Anal Prev 2015;79:100e16. https://
tion and defences of a work group in relation to the social structure and doi.org/10.1016/j.aap.2015.03.024.
technological content of the work system. Hum Relat 1951;4:3e38. https:// [29] Di Gravio G, Mancini M, Patriarca R, Costantino F. Overall safety perfor-
doi.org/10.1177/001872675100400101. mance of the air traffic management system: indicators and analysis. J Air
[2] Walker G. Come back sociotechnical systems theory, all is forgiven. Civ Transp Manag 2015;44e45:65e9. https://doi.org/10.1016/j.jairtraman.2015.
Eng Environ Syst 2015;32:170e9. https://doi.org/10.1080/10286608.2015. 02.005.
1024112. [30] Øien K, Massaiu S, Tinmannsvik RK, Størseth F. Development of early warning
[3] Pincetl S, Chester M, Eisenman D. Urban heat stress vulnerability in the U.S. indicators based on Resilience Engineering. In: 10th Int. Conf. Probabilistic Saf.
Southwest: the role of sociotechnical systems. Sustainability 2016;8:1e13. Assess. Manag. 2010, PSAM 2010, vol. 3; 2010.
https://doi.org/10.3390/su8090842. [31] Wreathall J. Monitoring e a critical ability in resilience engineering. In: Resil
[4] Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E, Eng Pract A Guideb., Safety and Security in Work and Its Environment, Brazil
Iedema R. Multimethod evaluation of information and communication 2011. p. 61e8.
276 Saf Health Work 2018;9:265e276

[32] Di Gravio G, Mancini M, Patriarca R, Costantino F. Overall safety performance [46] Peciłło M. The resilience engineering concept in enterprises with and without
of air traffic management system: forecasting and monitoring. Saf Sci occupational safety and health management systems. Saf Sci 2016:82. https://
2015;72:351e62. https://doi.org/10.1016/j.ssci.2014.10.003. doi.org/10.1016/j.ssci.2015.09.017.
[33] Tjørhom B, Aase K. The art of balance: using upward resilience traits to deal [47] Branlat M, Woods DD. How do systems manage their adaptive capacity to
with conflicting goals. In: Hollnagel E, Pariès J, Woods DD, Wreathall J, editors. successfully handle disruptions? A resilience engineering perspective. In:
Resil. Eng. Pract. A Guideb. Ashgate Publishing, Ltd; 2011. p. 157e70. AAAI Fall Symp. e Tech. Rep., vol. FS-10e03 2010.
[34] Wilson JR, Ryan B, Schock A, Ferreira P, Smith S, Pitsopoulos J. Understanding [48] Grote G. Safety management in different high-risk domains e all the same?
safety and production risks in rail engineering planning and protection. Er- Saf Sci 2012:50. https://doi.org/10.1016/j.ssci.2011.07.017.
gonomics 2009:52. https://doi.org/10.1080/00140130802642211. [49] Saaty TL. Decision making d the analytic hierarchy and network processes
[35] Hollnagel E. To learn or not to learn, that is the question. In: Hollnagel E, (AHP/ANP). J Syst Sci Syst Eng 2004;13:1e35. https://doi.org/10.1007/s11518-
Paries J, Woods DD, Wreathall J, editors. Resil. Eng. Pract. A Guideb. Ashgate 006-0151-5.
Publishing, Ltd; 2011. p. 193e8. [50] De Felice F, Petrillo A. Proposal of a structured methodology for the measure
[36] Weick KE. Educational organization as loosely couples systems. Adm Sci Q of intangible criteria and for decision making. Int J Simul Process Model
1976;21:1e19. 2014;9:157. https://doi.org/10.1504/IJSPM.2014.064392.
[37] Weick KE, Sutcliffe KM. Managing the unexpected: resilient performance in an [51] Patriarca R, Di Gravio G, Mancini M, Costantino F. Change management in the
age of uncertainty. John Wiley & Sons; 2011. ATM system: integrating information in the preliminary system safety
[38] Herrera IA, Nordskag AO, Myhre G, Halvorsen K. Aviation safety and main- assessment. Int J Appl Decis Sci 2016;9:121e38. https://doi.org/10.1504/
tenance under major organizational changes, investigating non-existing ac- IJADS.2016.080123.
cidents. Accid Anal Prev 2009;41:1155e63. https://doi.org/10.1016/ [52] Braithwaite J, Wears RL, Hollnagel E. Resilient health care: turning patient
j.aap.2008.06.007. safety on its head. Int J Qual Heal Care 2015;27:418e20. https://doi.org/
[39] Woods DD, Sarter NB. Learning from automation surprises and “going sour” ac- 10.1093/intqhc/mzv063.
cidents: progress on human-centered automation. In: Sarter NB, Amalberti R, [53] Hernan AL, Giles SJ, Fuller J, Johnson JK, Walker C, Dunbar JA. Patient and carer
editors. Cogn. Eng. Aviat. Domain. Hillsdale (NJ): Erlbaum; 2000. p. 327e54. identified factors which contribute to safety incidents in primary care: a
[40] Wreathall J. Measuring resilience. In: Nemeth CP, Hollnagel E, Dekker S, edi- qualitative study. BMJ Qual Saf 2015;24:583e93. https://doi.org/10.1136/
tors. Resil. Eng. Perspect. Vol. 2 Prep. Restor. Ashgate Publishing, Ltd; 2009. bmjqs-2015-004049.
p. 95e114. [54] Di Gravio G, Patriarca R, Mancini M, Costantino F. Overall safety performance
[41] Hollnagel E. Epilogue: RAG e the resilience analysis grid. In: Hollnagel E, of the air traffic management system: the Italian ANSP’s experience on APF.
Pariès J, Woods DD, Wreathall J, editors. Resil. Eng. Pract. A Guideb. Ashgate Res Transp Bus Manag 2016;20:3e12. https://doi.org/10.1016/j.rtbm.2016.
Publishing, Ltd; 2011. p. 275e96. 03.001.
[42] Rigaud E, Martin C. Considering trade-offs when assessing resilience. In: Resil. [55] van der Vorm J, van der Beek D, Bos E, Steijger N, Gallis R. Images of resilience:
Eng. Assoc. 4th Int. Symp. Soesterberg (The Netherlands): REA; 2013. the resilience analysis grid applicable at several organizational levels. In: 4th
[43] Apneseth K, Wahl AM, Hollnagel E. Measuring resilience in integrated plan- Resil. Eng. Assoc. Symp. 2011. p. 263e8.
ning. In: Albrechtsen E, Besnard D, editors. Oil Gas, Technol. Humans Assess. [56] Bonzo SM, Mclain D, Avnet MS. Process modeling in the operating room: a
Hum. Factors Technol. Chang. Norway: Health, Safety and Environment (HSE) socio-technical systems perspective. Syst Eng 2016;19:267e77. https://
Competence Centre, Statoil; 2013. doi.org/10.1002/sys.
[44] Ljungberg D, Lundh V. Resilience engineering within ATM e development, [57] Lawler EK, Hedge A, Pavlovic-Veselinovic S. International journal of industrial
adaption, and application of the resilience analysis grid (RAG). Norrköping, ergonomics cognitive ergonomics, socio-technical systems, and the impact of
Sweden; 2013. healthcare information technologies. Int J Ind Ergon 2011;41:336e44. https://
[45] Hegde S, Hettinger AZ, Fairbanks RJ, Wreathall J, Wears RL, Bisantz AM. doi.org/10.1016/j.ergon.2011.02.006.
Knowledge elicitation for resilience engineering in health care. In: Proc. Hum. [58] Hollnagel E. FRAM: the functional resonance analysis method e modelling
Factors Ergon. Soc., vol. 2015eJanua. Buffalo (NY), United States: Department complex socio-technical systems. Ashgate; 2012.
of Industrial and Systems Engineering, University at Buffalo-SUNY; 2015. [59] Leveson N. A new accident model for engineering safer systems. Saf Sci
p. 175e9. https://doi.org/10.1177/1541931215591036. 2004;42:237e70. https://doi.org/10.1016/S0925-7535(03)00047-X.