Buléon 

et al. Advances in Simulation (2022) 7:42

https://doi.org/10.1186/s41077-022-00238-9

ADVANCING SIMULATION PRACTICE Open Access

Simulation‑based summative assessment

in healthcare: an overview of key principles
for practice
Clément Buléon1,2,3*   , Laurent Mattatia4, Rebecca D. Minehart3,5,6, Jenny W. Rudolph3,5,6, Fernande J. Lois7,
Erwan Guillouet1,2, Anne‑Laure Philippon8, Olivier Brissaud9, Antoine Lefevre‑Scelles10, Dan Benhamou11,
François Lecomte12, the SoFraSimS Assessment with simulation group, Anne Bellot, Isabelle Crublé,
Guillaume Philippot, Thierry Vanderlinden, Sébastien Batrancourt, Claire Boithias‑Guerot, Jean Bréaud,
Philine de Vries, Louis Sibert, Thierry Sécheresse, Virginie Boulant, Louis Delamarre, Laurent Grillet,
Marianne Jund, Christophe Mathurin, Jacques Berthod, Blaise Debien, Olivier Gacia, Guillaume Der Sahakian,
Sylvain Boet, Denis Oriot and Jean‑Michel Chabot 

Abstract 
Background:  Healthcare curricula need summative assessments relevant to and representative of clinical situations
to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its util‑
ity for training in a formative context. Advancing to the next step, “the use of simulation for summative assessment”
requires rigorous and evidence-based development because any summative assessment is high stakes for partici‑
pants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.
Methods:  First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the
why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was
explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to
identify further references. Our goal was to identify current knowledge and potential recommendations for future
directions. Results were cross-checked among groups and reviewed by an independent expert committee.
Results:  Seven topics were selected by the task force: “What can be assessed in simulation?”, “Assessment tools for
SBSA”, “Consequences of undergoing the SBSA process”, “Scenarios for SBSA”, “Debriefing, video, and research for
SBSA”, “Trainers for SBSA”, and “Implementation of SBSA in healthcare”. Together, these seven explorations provide an
overview of what is known and can be done with relative certainty, and what is unknown and probably needs further
investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related con‑
clusions, the remaining important problems and questions, and their consequences for participants and institutions
of how SBSA is conducted.
Conclusion:  Our results identified among the seven topics one area with robust evidence in the literature (“What can
be assessed in simulation?”), three areas with evidence that require guidance by expert opinion (“Assessment tools for

*Correspondence: [email protected]
1
Department of Anesthesiology, Intensive Care and Perioperative Medicine,
Caen Normandy University Hospital, 6th Floor, Caen, France
Full list of author information is available at the end of the article

© The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​
mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Buléon et al. Advances in Simulation (2022) 7:42 Page 2 of 21

SBSA”, “Scenarios for SBSA”, “Implementation of SBSA in healthcare”), and three areas with weak or emerging evidence
(“Consequences of undergoing the SBSA process”, “Debriefing for SBSA”, “Trainers for SBSA”). Using SBSA holds much
promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously con‑
ducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation.
We believe this baseline can direct future investigation and the development of guidelines.
Keywords:  Medical education, Summative, Assessment, Simulation, Education, Competency-based education

Background what, when, and who. We aim at identifying areas (i) with
There is a critical need for summative assessment in robust evidence in the literature, (ii) with evidence that
healthcare education [1]. Summative assessment is high requires guidance by expert opinion, and (iii) with weak
stakes, both for graduation certification and for recerti- or emerging evidence. This may serve as a basis for future
fication in continuing medical education [2–5]. Knowing research and guideline development for the safe and
the consequences, the decision to validate or not validate effective use of SBSA (Fig. 1).
the competencies must be reliable, based on rigorous
processes, and supported by data [6]. Current methods Methods
of summative assessment such as written or oral exams First, we performed a modified Nominal Group Tech-
are imperfect and need to be improved to better benefit nique (NGT) to define the further questions to be
programs, learners, and ultimately patients [7]. A good explored in order to have the most comprehensive under-
summative assessment should sufficiently reflect clinical standing of SBSA. We followed recommendations on
practice to provide a meaningful assessment of compe- NGT for conducting and reporting this research [14].
tencies [1, 8]. While some could argue that oral exams Second, we conducted state-of-the-art literature reviews
are a form of verbal simulation, hands-on simulation can to assess the current knowledge on the questions/top-
be seen as a solution to complement current summa- ics identified by the modified NGT. This work did not
tive assessments and enhance their accuracy by bringing require Institutional Review Board involvement.
these tools closer to assessing the necessary competen-
cies [1, 2]. Context
Simulation is now well established in the healthcare A discussion on the use of SBSA was led by execu-
curriculum as part of a modern, comprehensive approach tive committee members of the Société Francophone de
to medical education (e.g., competency-based medical Simulation en Santé (SoFraSimS) in a plenary session
education) [9–11]. Rich in various modalities, simula- and involved congress participants in May 2018 at the
tion provides training in a wide range of technical and SoFraSimS annual meeting in Strasbourg, France. Key
non-technical skills across all disciplines. Simulation points addressed during this meeting were the growing
adds value to the educational training process particu- interest in using SBSA, its informal uses, and its inclu-
larly with feedback and formative assessment [9]. With sion in some formal healthcare curricula. The discus-
the widespread use of simulation in the formative setting, sion identified that these important topics lacked current
the next logical step is using simulation for summative guidelines. To reduce knowledge gaps, the SoFraSimS
assessment. executive committee assigned one of its members (FL,
The shift from formative to summative assessment one of the authors) to lead and act as a NGT facilita-
using simulation in healthcare must be thoughtful, evi- tor for a task force on SBSA. The task force’s mission
dence-based, and rigorous. Program directors and edu- was to map the current landscape of SBSA, the current
cators may find it challenging to move from formative knowledge and gaps; and potentially to identify experts’
to summative use of simulation. There are currently lim- recommendations.
ited experiences (e.g., OSCE [12, 13]) but not established
guidelines on how to proceed. The evidence needed Task force characteristics
for the feasibility, the validity, and the definition of the The task force panelists were recruited among volunteer
requirement for simulation-based summative assessment simulation healthcare trainers in French-speaking coun-
(SBSA) in healthcare education has not yet been formally tries after a call for application by SoFraSimS in May
gathered. With this evidence, we can hope to build a rig- 2019. Recruiting criteria were a minimum of 5  years of
orous and fair pathway to SBSA. experience in simulation and a direct involvement in
The purpose of this work is to review current knowl- simulation programs development or currently running.
edge for SBSA by clarifying the guidance on why, how, There were 34 (12 women and 22 men) from 3 countries
 Buléon et al. Advances in Simulation (2022) 7:42 Page 3 of 21

Fig. 1  Study question and topic level of evidence

(Belgium, France, Switzerland) included. Twenty-three geographic dispersion of the panelists across multiple
were physicians and 11 were nurses, while 12 total had countries and the context of the COVID-19 pandemic.
academic positions. All were experienced trainers in The first two steps of the NGT (generation of ideas
simulation for more than 7  years and were involved or and round robin) facilitated by the task force leader
responsible for initial training or continuing education (FL) were conducted online simultaneously and asyn-
programs with simulation. The task force leader (FL) chronously via email exchanges and online surveys
was responsible for recruiting panelists, organizing, and over a 6-week period. For the initiation of the first step
coordinating the modified NGT, synthesizing responses, (generation of ideas), the task force leader (FL) sent
and writing the final report. A facilitator (CB) assisted an initial non-exhaustive literature review of 95 arti-
the task force leader with the modified NGT, the synthe- cles and proposed the initial following items for reflec-
sis of responses, and the writing of the final report. Both tion: definition of assessment, educational principles
NGT facilitators (FL and CB) had more than 14 years of of simulation, place of summative assessment and its
experience in simulation, had experience in research in implementation, assessment of technical and non-tech-
simulation, and were responsive to simulation programs nical skills in initial training, continuing education, and
development and running. interprofessional training. The task force leader (FL)
asked the panelists to formulate topics or questions to
First part: initial question and modified nominal group propose for exploration in Part 2 based on their knowl-
technique (NGT) edge and the literature provided Panelists indepen-
To answer the challenging question of “What do we dently elaborated proposals and sent them back to the
need to know for a safe and effective SBSA practice?”, task force leader (FL) who regularly synthesized them
following the French Haute Autorité de Santé guidelines and sent the status of the questions/topics to the whole
[15], we applied a modified nominal group technique task force while preserving the anonymity of the con-
(NGT) approach [16] between September and October tributors and asking them to check the accuracy of the
2019. The goal of our modified NGT was to define the synthesized elements (second step, as a “round robin”).
further questions to be explored to have the most com- The third step of the NGT (clarification) was carried
prehensive understanding of the current SBSA (Fig. 2). out during a 2-h video conference session. All panelists
The modifications to NGT included interactions that were able to discuss the proposed ideas, group the
were not in-person and were asynchronous for some. ideas into topics, and make the necessary clarifications.
Those modifications were introduced as a result of the As a result of this step, 24 preliminary questions were
Buléon et al. Advances in Simulation (2022) 7:42 Page 4 of 21

Fig. 2  Study flowchart

defined for the fourth step (Supplemental Digital Con- a snowballing literature review technique [18] based on
tent 1). the articles’ references. The selected literature search
The fourth step of the NGT (vote) consisted of four dis- performed by each group was inserted into the task
tinct asynchronous and anonymous online vote rounds force’s common library on SBSA in healthcare as it was
that led to a final set of topics with related sub-questions conducted.
(Supplemental Digital content 2). Panelists were asked For references, we searched electronic databases
to vote to regroup, separate, keep, or discard questions/ (MEDLINE), gray literature databases (including digi-
topics. All vote rounds followed similar validation rules. tal theses), simulation societies and centers’ websites,
We [NGT facilitators (FL and CB)] kept items (either generic web searches (e.g., Google Scholar), and refer-
questions or topics) with more than 70% approval rat- ence lists from articles. We selected publications related
ings by panelists. We reworded and resubmitted in the to simulation in healthcare with keywords “summative
next round all items with 30–70% approval. We dis- assessment,” “summative evaluation,” and also specific
carded items with less than 30% approval. The task force keywords related to topics. The search was iterative to
discussed discrepancies and achieved final ratings with seek all available data until saturation was achieved.
a complete agreement for all items. For each round, we Ninety-five references were initially provided to the task
sent reminders to reach a minimum participation rate of force by the NGT facilitator leader (FL). At the end of the
80% of the panelists. Then, we split the task force into 7 work, the task force common library contained a total of
groups, one for each of the 7 topics defined at the end of 261 references.
the vote (step 4).
Techniques to enhance trustworthiness from primary
Second part: literature review reports to the final report
From November 2019 to October 2020, the groups The groups’ primary reports were reviewed and critiqued
each identified existing literature containing the current by other groups. After group cross-reviewing, primary
knowledge, and potential recommendations on the topic reports were compiled by NGT facilitators (FL and CB)
they were to address. This identification was done based in a single report. This report, responding to the 7 top-
on a non-systematic review of the existing literature. To ics, was drafted in December 2020 and submitted as a
identify existing literature, the groups conducted state- single report to an external review committee composed
of-the-art reviews [17] and expanded their reviews with of 4 senior experts in education, training, and research
 Buléon et al. Advances in Simulation (2022) 7:42 Page 5 of 21

from 3 countries (Belgium, Canada, France) with at least consider when defining the assessable competencies. Not
15 years of experience in simulation. NGT facilitators (FL all simulation methods are equivalent to assessing spe-
and CB) responded directly to reviewers when possible cific competencies [22].
and sought assistance from the groups when necessary. Most healthcare competencies can be assessed with
The final version of the report was approved by the SoF- simulation, throughout at curriculum, if certain con-
raSimS executive committee in January 2021. ditions are met. First, the competency being assessed
summatively must have already been assessed forma-
Results tively with simulation [23, 24]. Second, validated assess-
First part: modified nominal group technique (NGT) ment tools must be available to conduct this summative
The first two steps of the NGT by their nature (genera- assessment [25, 26]. These tools must be reliable, objec-
tion of ideas and “round robin”) did not provide results. tive, reproducible, acceptable, and practical [27–30]. The
The third step (clarification phase), identified 24 pre- small number of currently validated tools limits the use
liminary questions (Supplemental digital content 1) to of simulation for competency certification [31]. Third, it
be submitted to the fourth step (vote). The 4 rounds of is not necessary or desirable to certify all competencies
voting (step 4) resulted in 7 topics with sub-questions [32]. The situations chosen must be sufficiently frequent
(Supplemental Digital content 2): (1) “What can be in the student’s future professional practice (or poten-
assessed in simulation?” (2) “Assessment tools for SBSA,” tially impactful for the patient) and must be hard or
(3) “Consequences of undergoing the SBSA process,” (4) impossible to assess and validate in other circumstances
“Simulation scenarios for SBSA,” (5) “Debriefing, video, (e.g., clinical internships) [2]. Fourth, simulation can be
research and SBSA strategies,” (6) Trainers for SBSA,” (7) used for certification throughout the curriculum [33–35].
“Implementation of SBSA in healthcare”. These 7 topics Finally, limitations for the use of simulation throughout
and their sub-questions were the starting point for the the curriculum may be a lack of logistical resources [36].
state-of-the-art literature reviews of each group for the Based on our findings in the literature, we have sum-
second part. marized in Table  1 the educational consideration when
implementing a SBSA.
Second part: literature review
For each of the 7 topics, the groups highlighted what Assessment tools for simulation‑based summative
appears to be validated in the literature, the remain- assessment
ing important problems and questions, and their conse- One of the challenges of assessing competency lies in
quences for participants and institutions of how SBSA is the quality of the measurement tools [31]. A tool that
conducted. Results in this section present the major ideas allows the raters to collect data must also allow them to
and principles from the literature review, including their give meaning to their assessment, while securing that it is
nuances where necessary. really measuring what it aims to [25, 37]. A tool must be
valid and, capable of measuring the assessed competency
What can be assessed in simulation? with fidelity and, reliability while providing reproducible
Healthcare faculty and institutions must ensure that each data [38]. Since a competency is not directly measurable,
graduate is practice ready. Readiness to practice implies it will be analyzed on the basis of learning expectations,
mastering certain competencies, which is dependent on the most “concrete” and observable form of a compe-
learning them appropriately. The competency approach tency [19]. Several authors have described definitions of
involves explicit definitions of the acquired core compe- the concept of validity and the steps to achieve it [38–41].
tencies necessary to be a “good professional.” Professional Despite different validation approaches, the objectives
competency could be defined as the ability of a profes- are similar: to ensure that the tool is valid, the scoring
sional to use judgment, knowledge, skills, and attitudes items reflect the assessed competency, and the contents
associated with their profession to solve complex prob- are appropriated for the targeted learners and raters [20,
lems [19–21]. Competency is a complex “knowing how to 39, 42, 43]. A tool should have psychometric characteris-
act” based on the effective mobilization and combination tics that allow users to be confident of its reproducibility,
of a variety of internal and external resources in a range discriminatory nature, reliability, and external consist-
of situations [19]. Competency is not directly observable; ency [44]. A way to ensure that a tool has acceptable
it is the performance in a situation that can be observed validity is to compare it to existing and validated tools
[19]. Performance can vary depending on human factors that assess the same skills for the same learners. Finally,
such as stress, fatigue, etc.… During simulation, compe- it is important to consider the consequences of the test to
tencies can be assessed by observing “key” actions using determine whether it best discriminates competent stu-
assessment tools [22]. Simulation’s limitations must dents from others [38, 43].
Table 1  Considerations for implementing a summative assessment with simulation
Considerations Elements Items Example adapted to cardiopulmonary resuscitation (CPR)
for an emergency physician

Competency to be assessed Clear definition of competency Know how to act in a professional situation The practitioner is able to handle an in-hospital cardiac arrest
Identify internal resources: knowledge, skills, behavior, and (CA)
reasoning ACLS algorithm, airway management, leadership, manage‑
Identify external resources: equipment, written or electronic ment according to the type of CA (e.g., asystole, pulseless
resources), colleagues, and so on to mobilize electrical activity, ventricular fibrillation)
e.g., defibrillator, cognitive aids (a chart, a checklist, …), ECMO
team, …
Buléon et al. Advances in Simulation

Number of competencies Consider the possibility of assessing one or more competen‑ In-hospital CA alone, or CA in adult patient and/or in specific
cies simultaneously conditions (e.g., child, pregnant, …)
Measurements Consider measuring performance in representative and CA in a young polytrauma patient, in an elderly diabetic
diverse situations patient, in a pregnant woman or in a child out-of-hospital
Assessment Context authenticity Complex problems e.g., CA due to hyperkaliemia in a patient with renal failure
(2022) 7:42

Adapt the complexity to the training level Complexity may be tuned for an expert with patient’s chronic
Ensure context relevance to future or current professional use of beta-blockers
practice CA occurs in an ambulance or in an emergency room or in
Interprofessional situations (vs uniprofessional) OR or in ICU
Prefer a situation where the learner is not alone such as a
member of an emergency team and not as a first responder
in the street
Standardization Tasks and requirements known before by the participants Send to the learner the assessment template prior to the
Direct observation associated with a phase of student inter‑ assessment
action (questioning) The simulation is followed by a debriefing (feedback)
Rate with a checklist or a rubric
Correction criteria Multiple sources and/or iteration (e.g., repeated perfor‑ e.g., time from the start of VF to the first external electric
mances of the same scenario) shock and/or compliance with ACLS steps and/or quality of
Clear and specific objectives external cardiac massage (visual and/or via sensors)
Adjusted to the assessed knowledge or to the simulation Only items that have been previously decided are assessed
Integration of self-assessment (see above)
Consider only important errors It is not possible to assess the use of the defibrillator if the
Strategies (cognitive and metacognitive) assessed during situation is pulseless electrical activity
the interaction phase 6 instead of 5 min between 2 doses of adrenaline (minor
Prior consensus on rating and definition regarding expected error) versus no recognition of a shockable rhythm (major
level of development error)
Ask questions during feedback phase: “Can you remind me of
the administration schedule for epinephrine in CA?” (cogni‑
tion). “I have observed that you administered it every minute,
but as you have just said and as I think it is every 3 to 5 min,
could you explain why in the situation you administered it
every minute?” (metacognition)
Identify minor and major errors together (all instructors
involved in the assessment of this competency). Define the
number of acceptable minor and/or major errors to validate
the acquisition or not of the competency at this level of
development
Page 6 of 21
Table 1  (continued)
Considerations Elements Items Example adapted to cardiopulmonary resuscitation (CPR)
for an emergency physician

Scenarios Development Developing scenarios only after defining the skills and or e.g., if we want to evaluate the use of the defibrillator, we
competences to be assessed need to construct a scenario where the patient has VF or VT
Ensuring the scenario reflects professional reality e.g., use a hyperkalemia CA scenario after a burial extraction
Incorporating the targeted skills into a scenario representing but not when releasing a tourniquet after a knee replacement
professional practice, rather than a task trainer, for example for an emergency physician
Prefer to use a scenario with a clinical history of CA to assess
Buléon et al. Advances in Simulation

CPR skills rather than performing CPR in a skill station

Multiple skills Several stations with short scenarios (e.g., 5–6 min) each are Ensure that all steps can be assessed. E.g., the use of ECMO
preferable to long scenarios (e.g., > 20 min) is reserved for refractory CA and cannot be considered if the
Critical situation scenario lasts for 5 min and begins with the recognition of the
arrest. In this case, a scenario with a CA that has already been
(2022) 7:42

under management for 15 min should be used

Test prior to use Validity, reliability, reproducibility The scenarios used should be pre-tested by the teaching
team including using the assessment forms
Simulators (High and low-Technology) Use and difficulty level validated e.g., if intubation is expected during the scenario, the chosen
manikin should allow it
Assessment test standardization Facilitator’s role and intervention specified in advance What can the facilitator do? E.g., can he/she guide on 4H-4 T if
(Fairness) Only one candidate per station the learner does not think about it?
Practical conditions Minimum number of scenarios (8 to 15) [157] Scenarios in different circumstances (in and out-of-hospital),
Incentive to verbalize after action different causes (4H-4 T), different ages (child to elderly adult)
(Reasoning, what is done or not done) To be recalled in the pre-briefing
Raters At least, two raters e.g., clinical supervisor, ACLS instructor, simulation instructor
Ideally, a rater should be involved in the formative assess‑ who has supervised the learner during the formative sessions,
ment program …
Page 7 of 21
Buléon et al. Advances in Simulation (2022) 7:42 Page 8 of 21

Like a diagnostic score, a relevant assessment tool must framework for a technical task [38, 45, 46]. An alternative
be specific [30, 39, 41]. It is not good or bad, but valid framework using three sources of validity for teamwork’s
through a rigorous validation process [39, 41, 42]. This non-technical skills are presented in Table 3.
validation process determines whether the tool measures A tool is validated in a language. Theoretically, this tool
what it is supposed to measure and whether this meas- can only be used in this language, given the nuances pre-
urement is reproducible at different times (test–retest) sent with interpretation [49]. In certain circumstances, a
or with 2 observers simultaneously. It also determines if “translated” tool, but not a “translated and validated in a
the tool results are correlated with another measure of specific language” tool, can lead to semantic biases that
the same ability or competency and if the consequences can affect the meaning of the content and its represen-
of the tool results are related to the learners’ actual com- tation [49–55]. For each assessment sequence, validity
petency [38]. criteria consist of using different tools in different assess-
Following Messick’s framework, which aimed to gather ment situations and integrating them into a comprehen-
different sources of validity in one global concept (uni- sive program which considers all aspects of competency.
fied validity), Downing describes five sources of validity, The rating made with a validated tool for one situation
which must be assessed with the validation process [38, must be combined with other assessment situations,
45, 46]. Table  2 presents an illustration of the develop- since there is no “ideal” tool [28, 56] A given tool can
ment used in SBSA according to the unified validity be used with different professions or with participants

Table 2  Example of the development of a tool to assess technical skill achievement in a simulated situation, based on work by Oriot
et al., Downing, and Messick’s framework [38, 46, 47]
Source of validity Method Judgment criteria Results

content 1. Description of the checklist develop‑ Relevance of items Obtaining a list of 12 items (after the initial
ment by 2 experts Adapted illustration of the skill proposal of 20 items)
2. Review by 2 outside experts Conditions of skill achievement
3. Definitive Checklist
Response process Pilot study, search for error sources Interrater reproducibility Fusion/removal of redundant items
Adapting items Item content (redundant, inaccurate) Minutes, degrees, centimeters
Defining units of measurement Controlling the sources of measurement justification
errors
Weighing items
Internal structure Internal coherence Cronbach Alpha Coefficient, Cronbach result
Reproducibility interrater: Cohen Kappa, ICC Correlation between 2 raters
Discrimination of learners
Comparison with Score vs success or failure of the proce‑ Correlation between procedure success Time for success, score for success and
other variables dure or theoretical assessment and score with rating
Score vs theoretical assessment the tool
Score vs previous experience/level of
expertise
Consequences Minimum passing score Pass-fail score with procedure success 14/20

Table 3  Example of the development of an assessment tool for the observation of teamwork in simulation [48]
Source of validity Method Judgment criteria Results

Content 1. Description of the Clinical Team- Literature review 15 items

work Scale (CRM scale) Develop‑ Scale already used in another field (aeronaut‑ 5 categories
ment ics) 1 overall skill score
Response process 1. Relevance of items 1. Precise description of each item 1. Ratings aid table
2. weighting items 2. Quantitative criteria 2. 0 to 10 or 0/1
3. Raters’ training (moderate) 3. Qualitative criteria Descriptive levels: not relevant/unacceptable/
4. CRM principles poor/average/good/perfect
Internal structure 1. Built-in validity 1. Distribution of scores from the preset level 1. Score tailored to each level
2. Scale usability 2. Number of items filled in full 2. Easy-to-use scale (little loss of information)
3. Reproducibility 3. interrater concordance, the correlation 3. correlation between raters
between overall score and categories (Kappa, 4. Variation in scores between scenarios
Kendall, Pearsons, ICC) sources of error
4. Variance of each category
 Buléon et al. Advances in Simulation (2022) 7:42 Page 9 of 21

at different levels of expertise or in different languages Second, the transparency on the objectives and methods
if it is validated for these situations [57, 58]. In a sum- of assessment limits detrimental psychological impact
mative context, a tool must have demonstrated a high- [77, 78]. Finally, detrimental psychological impacts are
level of validity to be used because of the high stake for increased by abnormally high physiological or emotional
the participants [56]. Finally, the use or creation of an stress such as fatigue, and stressful events in the 36 h pre-
assessment tool requires trainers to question its various ceding the assessment, such that students with a history
aspects, from how it was created to its reproducibility of post-traumatic stress disorder or psychological disor-
and the meaning of the results generated [59, 60]. der may be strongly and negatively impacted by the simu-
Two types of assessment tools should be distinguished: lation [76, 79–81].
tools that can be adapted to different situations and tools It is necessary to optimize SBSA implementation to
that are specific to a situation [61]. Thus, technical skills limit its pedagogical and psychological negative impacts.
may have a dedicated assessment tool (e.g., intraosseous) Ideally, during the summative assessment, it has been
[47] or an assessment grid generated from a list of pre- proposed to take into account the formative assessment
established and validated items (e.g., TAPAS scale) [62]. that has already been carried out [1, 20, 21]. Similarly in
Non-technical skills can be observed using scales that are continuing education, the professional context of the per-
not situation-specific (e.g., ANTS, NOTECHS) [63, 64] son assessed should be considered. In the event of failure,
or that are situation-specific (e.g., TEAM scale for resus- it will be necessary to ensure sympathetic feedback and
citation) [57, 65]. Assessment tools should be provided to propose a new assessment if necessary [21].
to participants and should be included in the scenario
framework, at least as a reference [66–69]. In the sum- Scenarios for simulation‑based summative assessment
mative assessment of a procedure, structured assessment Some authors argue that there are differences between
tools should probably be used, using a structured objec- summative and formative assessment scenarios [76, 79–
tive assessment form for technical skills [70]. The use of a 81]. The development of a SBSA scenario begins with
scale, in the context of the assessment of a technical ges- the choice of a theme, which is most often agreed upon
ture, seems essential. As with other tools, any scale must by experts at the local level [66]. The themes are most
be validated beforehand [47, 70–72]. often chosen based on the participants’ competencies to
be assessed and included in the competencies require-
Consequences of undergoing the simulation‑based ment for the initial [82] and continuing education [35,
summative assessment process 83]. A literature review even suggests the need to choose
Summative assessment has two notable consequences on themes covering all the competences to be assessed [41].
learning strategies. First, it may drive the learner’s behav- These choices of themes and objectives also depend on
ior during the assessment, while it is essential to assess the simulation tools technically available: “The themes
the competencies targeted, not the ability of the par- were chosen if and only if the simulation tools were capa-
ticipant to adapt to the assessment tool [6]. Second, the ble of reproducing “a realistic simulation” of the case.”
pedagogy key concept of “pedagogical alignment” must [84].
be respected [23, 73]. It means that assessment meth- The main quality criterion for SBSA is that the cases
ods must be coherent with the pedagogical activities and selected and developed are guided by the assessment
objectives. For this to happen, participants must have objectives [85]. It is necessary to be clear about the
formative simulation training focusing on the assessed assessment objectives of each scenario to select the right
competencies prior to the SBSA [24]. assessment tool [86]. Scenarios should meet four main
Participants have been reported as experiencing com- principles: predictability, programmability, standardiz-
monly mild (e.g., appearing slightly upset, distracted, ability, and reproducibility [25]. Scenario writing should
teary-eyed, quiet, or resistant to participating in the include a specific script, cues, timing, and events to
debriefing) or moderate (e.g., crying, making loud, and practice and assess the targeted competencies [87]. The
frustrated comments) psychological events in the simu- implementation of variable scenarios remains a challenge
lation [74]. While voluntary recruitment for formative [88]. Indeed, most authors develop only one scenario per
simulation is commonplace, all students are required to topic and skill to be assessed [85]. There are no recom-
take summative assessments in training. This required mendations for setting a predictable duration for a sce-
participation in high-stake assessment may have a more nario [89]. Based on our findings we suggest some key
consequential psychological impact [26, 75]. This impact elements for structuring a SBSA scenario in Table 4. For
can be modulated by training and assessment conditions technical skill assessment, scenarios will be short and the
[75]. First, the repetition of formative simulations reduces assessment is based on an analytical score [82, 89]. For
the psychological impact of SBSA on participants [76]. non-technical skill assessment, scenarios will be longer
Buléon et al. Advances in Simulation (2022) 7:42 Page 10 of 21

Table 4 Key element structuring a summative assessment (Table  5) [95, 98]. In SBSA sessions, video can be used
scenario during the prebriefing to provide participants with stand-
Elements Recommendations ardized and reproducible information [99]. A video can
increase the realism of the situation during the simula-
Duration 10 to 15 min tion with ultrasound loops and laparoscopy footage.
Short for technical skills
Longer for non-technical skills Simulation records can be reviewed either for debriefing
Objectives Accurate list of competencies and skills to be assessed or rating purposes [34, 71, 100, 101]. A video is very use-
Essential items Initial assessment ful for the training raters (e.g., for calibration and recali-
Diagnostic strategy bration) [102]. It enables raters to rate the participants’
Situation management performance offline and to have an external review if
Orientation strategy
necessary [34, 71, 101]. Despite the technical difficulties
Script Computerized (programed if possible)
to be considered [42, 103], it can be expected that video-
Rating scale Checklist, Global Rating Scale
Scale (20 to 30 items) based automated scoring assistance will facilitate assess-
Analytic score for technical skills ments in the future.
Analytic and holistic (e.g., ANTS) for non-technical skills The constraints associated with the use of video rely on
Validation Pilot sessions (scenario testing and rater training) the participants’ agreement, the compliance with local
1 or 2 cases per student during scenario testing
rules, and that the structure in charge of the assessment
Assessment Video rating
Cohen’s Kappa test for differences between raters
with video secures the protection of the rights of individ-
Student’s t test for the ability to discriminate between uals and data safety, both at a national and at the higher
students (e.g., European GDPR) level [104, 105].
In Table 5, we list the main uses of video during simula-
tion sessions found in the literature.
and the assessment based on analytical and holistic Research in SBSA can focus, as in formative assess-
scores [82, 89]. ment, on the optimization of simulation processes (pro-
grams, structures, human resources). Research can also
Debriefing, video, and research for simulation‑based explore the development and validation of summa-
summative assessment tive assessment tools, the automation and assistance of
Studies have shown that debriefings are essential in assessment resources, and the pedagogical and clinical
formative assessment [90, 91]. No such studies are avail- consequences of SBSA.
able for summative assessment. Good practice requires
debriefing in both formative and summative simulation- Trainers for simulation‑based summative assessment
based assessments [92, 93]. In SBSA, debriefing is often Trainers for SBSA probably need specific skills because
brief feedback given at the end of the simulation session, of the high number of potential errors or biases in SBSA,
in groups [85, 94, 95], or individually [83]. Debriefing can despite the quest for objectivity (Table 6) [106]. The dif-
also be done later with a trainer and help of video, or via ficulty in ensuring objectivity is likely the reason why the
written reports [96]. These debriefings make it possible to use of self or peer assessment in the context of SBSA is
assess clinical skills for summative assessment purposes not well documented and the literature does not yet sup-
[97]. Some tools have been developed to facilitate this port it [59, 60, 107, 108].
assessment of clinical reasoning [97]. SBSA requires the development of specific scenar-
Video can be used for four purposes: session prepara- ios, staged in a reproducible way, and the mastery of
tion, simulation improvement, debriefing, and rating

Table 5  Uses of video for simulation-based formative and summative assessment

Formative assessment Summative assessment

Prebriefing Participant information

Simulation Increased scenario realism (e.g., coelioscopy video)
Watching by observers
Immediate visualization after Self-assessment No self-assessment (in the literature)
simulation Debriefing by trainers (selected sequences)
Delayed visualization Learning teamwork or skills for a formative purpose Deferred debriefing
Rater training (calibration and recalibration)
Administrative evidence
 Buléon et al. Advances in Simulation (2022) 7:42 Page 11 of 21

Table 6  Potential errors, effects, and bias in simulation-based summative assessment [109, 110]
Type of error Error description

Error of homogenization Tendency to rate neither too good or too bad, making discrimination more difficult
Halo effect Tendency to see everything right or wrong in the same performance
Time effect Bias related to observations of early or late good or bad performance during sessions
Bias of “clemency” Willingness not to give bad grades
Repository error Judgment based on what the rater would have done and not on the assessment tool
Group effect Evaluation based on the team’s performance rather than the participant’s performance

assessment tools to avoid assessment bias [111–114]. Ful- The development of a competency framework valid
filling those requirements calls for specific abilities to fit for an entire curriculum (e.g., medical studies) satisfies
with the different roles of the trainer. These different roles a fundamental need [7, 120]. This development allows
of trainers would require specific initial and ongoing identifying competencies to be assessed with simulation,
training tailored to their tasks [111, 113]. In the future, those to be assessed by other methods, and those requir-
concepts of the roles and tasks of these trainers should ing triangulation by several assessment methods. This
be integrated into any “training of trainers” in simulation. identification then guides scenarios’ writing and exami-
nation’s development with good content validity. Sce-
Implementation of simulation‑based summative assessment narios and examinations will form a bank of reproducible
in healthcare assessment exercises. The examination quality process,
The use of SBSA has been described by Harden in 1975 including psychometric analyses, is part of the develop-
with Objective and Structured Clinical Examination ment process from the beginning [85].
(OSCE) tests for medical students [115]. The summa- We have summarized in Table  7 the different steps in
tive use of simulation has been introduced in different the implementation of SBSA.
ways depending on the professional field and the coun-
try [116]. There is more literature on certification at the Recertification  Recertification programs for various
undergraduate and graduate levels than on recertification healthcare domains are currently being implemented or
at the postgraduate level. The use of SBSA in re-certifica- planned in many countries (e.g., in the USA [118] and
tion is currently more limited [83, 117]. Participation is France [116]). This is a continuation of the movement
often mandated, and it does not provide a formal assess- to promote the maintenance of competencies. Examples
ment of competency [83]. Some countries are defin- can be cited in France with the creation of an agency for
ing processes for the maintenance of certification in continuing professional development or in the USA with
which simulation is likely to play a role (e.g., in the USA the Maintenance Of Certification [83, 126]. The certifica-
[118] and France [116]). Recommendations regarding tion of health care facilities and even teams is also being
the development of SBSA for OSCE were issued by the studied [116]. Simulation is regularly integrated into
AMEE (Association for Medical Education in Europe) in these processes (e.g., in the USA [118] and France [116]).
2013 [12, 119]. Combined with other recommendations Although we found some commonalities basis between
that address the organization of examinations on other the certification and recertification processes, there are
immersive simulation modalities, in particular, full-scale many differences (Table 8).
sessions using complex mannequins [22, 85], they give us
a solid foundation for the implementation of SBSA. Currently, when simulation-based training is manda-
The overall process to ensure a high-quality examina- tory (e.g., within the American Board of Anesthesiol-
tion by simulation is therefore defined but particularly
or MOCA 2.0® in the US), it is most often a formative
ogy’s “Maintenance Of Certification in Anesthesiology,”
demanding. It mobilizes many material and human
resources (administrative staff, trainers, standardized process [34, 83]. SBSA has a place in the recertification
patients, and healthcare professionals) and requires a process, but there are many pitfalls to avoid. In the short
long development time (several months to years depend- term, we believe that it will be easier to incorporate form-
ing on the stakes) [36]. We believe that the steps to over- ative sessions as a first step. The current consensus seems
come during the implementation of SBSA range from to be that there should be no pass/fail recertification sim-
setting up a coordination team, to supervising the writ- ulation without personalized global professional support,
ers, the raters, and the standardized patients, as well as
taking into account the logistical and practical pitfalls.
Buléon et al. Advances in Simulation (2022) 7:42 Page 12 of 21

Table 7  Implementation of simulation-based summative assessment step by step

Items Goals Modalities

Team Identify the training staff Structure coordination

Size the team: skills, time available, stability (project over
several months/years)
Competencies repository Create the competencies repository to be assessed Expert panels
Define the number and type of examination needed
Must be known to students
Curriculum integrate summative assessment in the curriculum Pedagogical alignment: summative part drives the forma‑
tive part of the curriculum
No summative assessment without pre-simulation expo‑
sure
Intermediate summative assessment could be useful [121]
Examination Define summative assessment modalities through Length and number of scenarios stations [122, 123]
simulation The higher the fidelity of the examination, the harder is it
to set it up, the lower the feasibility
Scenarios Develop a bank of scenarios and rating grids [124] Choose the editors for the scenarios
Write the scenarios
Scenarios’ peer-review and test
Establish/choose assessment tools (Checklist or global
scale)
Set the minimum passing score
The themes of the bank’s scenarios cover the competen‑
cies of the repository
Training raters Limit rating variations for a given performance Choice of raters
Raters’ Training Workshop
Standardized Patients Develop a standardized patient pool Recruitment, selection, training, and standardization [125]
D-Day How the examination take place Logistics: e.g., dates, rooms, standardized patients, rights of
personal portrayal, GDPR
Participants’ path, breaks
Materials to supply, to be brought by students (e.g.,
stethoscope)
Examination-adapted briefings
Problems to anticipate: e.g., maintenance of standardiza‑
tion, failure or breakage of equipment, backup paper sup‑
ports, dedicated staff for support to stressed participants,
immediately after examination Finalize the examination Collect and check assessment grids for early detection of
inconsistencies, rating oversights, missing data
Management of participants’ complaints and plea
Quality process Prepare future examination Identify potential changes to do to some scenarios
Removal of inappropriate scenarios: e.g., too long, mislead‑
ing, source of rating inconsistency,
Changes to standardized patients’ training
Changes in raters’ training

but which is not limited to a binary aptitude/inaptitude What can be assessed in simulation?
approach [21, 116]. SBSA is currently mainly used in initial training in uni-
professional and individual settings via standardized
patients or task trainers (OSCE) [12, 13]. In the future,
Discussion SBSA will also be used in continuing education for pro-
Many important issues and questions remain regarding fessionals who will be assessed throughout their career
the field of SBSA. This discussion will return to our iden- (re-certification) as well as in interprofessional settings
tified 7 topics and highlight these points, their implica- [83]. When certifying competencies, it is important to
tions for the future, and some possible leads for future keep in mind the differences between the desired com-
research and guidelines development for the safe and petencies and the observed performances [128]. Indeed,
effective use of this tool in SBSA. it must be that “what is a competency” is specifically
defined [6, 19, 21]. Competencies are what we wish to
evaluate during the summative assessment to validate
 Buléon et al. Advances in Simulation (2022) 7:42 Page 13 of 21

Table 8  Commonalities and discrepancies between certification and recertification

Items Commonalities Discrepancies

Modalities Multimodal process (course, simulation, etc.) [34, 83, 92] Low percentage of existing recertification [34, 83]
Field follow-up opportunities [35] Level of acceptability and feasibility of recertification
Level of recertification: pursuing individual certification or switching
with team recertification
Organization bodies Accredited centers Can institutions (universities, schools) in charge of certification, provide
(functional specification) [34, 83] recertification?
Same rigor in setting up
Objectives Targeted level of competency Difficulties in the efficient selection of competencies to be assessed
with recertification:
•
Multiple constraints (time/means)
•
Communication/teamwork, performance gaps, frequent adverse
events?
Scenarios and assessment tools adapted for learning objectives [127]
Consequences Possible failure of certification or recertification The impact of a failure to recertification is major for a professional
Mandatory discretion of the recertification process
Opportunity for screening of professionals in difficulty (burn out…) [92,
116]
Funding Funding difficulties Many options of financing in recertification (state, professional insur‑
ance, etc.)

or revalidate a professional for his/her practice. Perfor- Assessment tools for simulation‑based summative
mance is what can be observed during an assessment [20, assessment
21]. In this context, we consider three unresolved issues. Rigor and method in the development and selection
The first issue is that an assessment only gives access to of assessment tools are paramount to the quality of
a performance at a given moment (“Performance is a the summative assessment [136]. The literature shows
snapshot of a competency”), whereas one would like to that is necessary that assessment tools be specific to
assess a competency more generally [128]. The second their intended use that their intrinsic characteristics be
issue is: How does an observed performance—especially described and that they be validated [38, 40, 41, 137].
in simulation—reveal a real competency in real life? These specific characteristics must be respected to
[129] In other words, does the success or failure of a sin- avoid two common issues [1, 6]. The first issue is that of
gle SBSA really reflect actual real-life competency? [130] a poorly designed or constructed assessment tool. This
The third issue is the assessment of a team performance/ tool can only give poor assessments because it will be
competency [131–133]. Until now, SBSA has come from unable to capture performance correctly and therefore
the academic field and has been an individual assessment to approach the skill to be assessed in a satisfactory way
(e.g., OSCE). Future SBSA could involve teams, driven by [56]. The second issue is related to poor or incomplete
governing bodies, institutions, or insurances [134, 135]. tool evaluation or inadequate tool selection. If the tool
The competency of a team is not the sum of the compe- is poorly evaluated, its quality is unknown [56]. The
tencies of the individuals who compose it. How can we scope of the assessment that is done with it is limited by
proceed to assess teams as a specific entity, both com- the imprecision of the tool’s quality. If the tool is poorly
posed of individuals and independent of them? To make selected, it will not accurately capture the performance
progress in answering these three issues, we believe it being assessed. Again, summative assessment will be
is probably necessary to consider the approximation compromised. It is currently difficult to find tools that
between observed and assessed performance and compe- meet all the required quality and validation criteria
tency as acceptable, but only by specifying the scope of [56]. On the one hand, this requires complex and rig-
validity. Research in these areas is needed to define it and orous work; on the other hand, the potential number
answer these questions. of tools required is large. Thus, the overall volume of
The consequence of undergoing SBSA has focused on work to rigorously produce assessment tools is sub-
the psychological aspect and have set aside the more stantial. However, the literature provides the charac-
usual consequences such as achieving (or not) the mini- teristics of validity (content, response process, internal
mum passing score. Future research should embrace structure, comparison with other variables, and con-
more global SBSA consequence field, including how reli- sequences), and the process of developing qualitative
able SBSA is at determining how someone is competent. and reliable assessment tools [38–41, 45]. It therefore
Buléon et al. Advances in Simulation (2022) 7:42 Page 14 of 21

seems important to systematize the use of these guide- learn and progress in mastering this same skill. Although
lines for the selection, development, and validation of there may be a continuum between the two, they remain
assessment tools [137]. Work in this area is needed and distinct. SBSA scenarios must be predictable, program-
network collaboration could be a solution to move for- mable, standardizable, and reproductible [25] to ensure
ward more quickly toward a bank of valid and validated fairly assessed performances among participants. This
assessment tools [39]. is even more crucial when standardized patients are
involved (e.g., OSCE) [119, 145]. In this case, a specific
Consequences of undergoing the simulation‑based script with expectations and training is needed for the
summative assessment process standardized patient. The problem is that currently there
We had focused our discussion on the consequences of are many formative scenarios but few summative sce-
SBSA excluding the determining of the competencies and narios. The rigor and expertise required to develop them
passing rates. Establishing and maintaining psychological is time-consuming and requires expert trainer resources.
safety is mandatory in simulation [138]. Considering the We believe that a goal should be to homogenize the sce-
psychological and physiological consequences of SBSA narios, along with preparing the human resources who
is fundamental to control and limit negative impacts. will implement them (trainers and standardized patients)
Summative assessment has consequences for both the and their application. We believe one solution would
participants and the trainers [139]. These consequences be to develop a methodology for converting forma-
are often ignored or underestimated. However, these tive scenarios into summative ones in order to create a
consequences can have an impact on the conduct or structuring model for summative scenarios. This would
results of the summative assessment. The consequences reinvest the time and expertise already used for develop-
can be positive or negative. The “testing effect” can have ing = formative scenarios.
a positive impact on long-term memory [139]. On the
other hand, negative psychological (e.g., stress or post- Debriefing for simulation‑based summative assessment
traumatic stress disease), and physiological (e.g., sleep) The place of debriefing in SBSA is currently undefined
consequences can occur or degrade a fragile state [139, and raises important questions that need exploration
140]. These negative consequences can lead to ques- [77, 90, 146–148]. Debriefing for formative assessment
tioning the tools used and the assessments made. These promotes knowledge retention and helps to anchor good
consequences must therefore be logically considered behaviors while correcting less ideal ones [149–151]. In
when designing and conducting the SBSA. We believe general, taking an exam promotes learning of the topic
that strategies to mitigate their impact must be put in [139, 152]. Formative assessment without a debriefing
place. The trainers and the participants must be aware has been shown to be detrimental, so it is reasonable to
of these difficulties to better anticipate them. It is a real assume that the same is true in summative assessment
duality for the trainer: he/she has to carry out the assess- [91]. The ideal modalities for debriefing in SBSA are cur-
ment in order to determine a mark and at the same time rently unknown [77, 90, 146–148]. Integrating debriefing
guarantee the psychological safety of the participants. It into SBSA raises a number of organizational, pedagogi-
seems fundamental to us that trainers master all aspects cal, cognitive, and ethical issues that need to be clari-
of SBSA as well as the concept of the safe container [138] fied. From an organizational perspective, we consider
to maximize the chances of a good experience for all. that debriefing is time and human resource-consuming.
We believe that ensuring a fluid pedagogical continuum, The extent of the organizational impact varies according
from training to (re)certification in both initial and con- to whether the feedback is automatized, standardized,
tinuing education using modern pedagogical techniques personalized, and collective or individual. From an edu-
(e.g., mastery learning, rapid cycle deliberate practice) cational perspective, debriefing ensures pedagogical con-
[141–144] could help maximize the psychological and tinuity and continued learning. We believe this notion is
physiological safety of participants. nuanced, depending on whether the debriefing is inte-
grated into the summative assessment or instead follows
Scenarios for simulation‑based summative assessment the assessment while focusing on formative assessment
The structure and use of scenarios in a summative setting elements. We believe that if the debriefing is part of the
are unique and therefore require specific development SBSA, it is no longer a “teaching moment.” This must
and skills [83, 88]. SBSA scenarios differ from formative be factored into the instructional strategy. How should
assessment scenarios by the different educational objec- the trainer prioritize debriefing points between those
tives that guide their development. Summative scenar- established in advance for the summative assessment
ios are designed to assess a skill through observation of and those that would emerge from any individuals’ per-
performance, while formative scenarios are designed to formance? From a cognitive perspective, whether the
 Buléon et al. Advances in Simulation (2022) 7:42 Page 15 of 21

debriefing is integrated into the summative assessment selecting the assessment tool(s), training the trainer-
may alter the interactions between the trainer and the rater(s), and supervising the SBSA sessions. Second,
participants. We believe that if the debriefing is inte- there should be the trainer-operators responsible for run-
grated into the SBSA, the participant will sometimes be ning the simulation conditions that support the assess-
faced with the cognitive dilemma of whether to express ment. Third, there are the trainer-raters who conduct the
his/her “true” opinions or instead attempt to provide the assessment using the assessment tool(s) selected by the
expected answers. The trainer then becomes uncertain trainer-designer(s) for which these trainer-raters have
of what he/she is actually assessing. Finally, from an ethi- been specifically trained. The high-stake nature of SBSA
cal perspective, in the case of a mediocre or substand- requires a high level of rigor and professionalism from
ard clinical performance, there is a question of how the the three levels of trainers, which implies they have a
trainer resolves discrepancies between observed behavior working definition of the skills and the necessary training
and what the participant reveals during the debriefing. to be up to the task.
What weight should be given to the simulation and to the
debriefing for the final rating? We believe there is prob- Implementing simulation‑based summative assessment
ably no single solution to how and when the debriefing in healthcare
is conducted during a summative assessment but rather Implementing SBSA is delicate, requires rigor, respect
promote the idea of adapting debriefing approaches for each step, and must be evidence-based. While
(e.g., group or individualized debriefing) to various con- OSCEs are simulation-based, simulation is not limited
ditions (e.g., success or failure in the summative assess- to OSCEs. Summative assessment with OSCEs has been
ment). These questions need to be explored to provide used and studied for many years [12, 13]. This literature
answers as to how debriefing should be ideally conducted is therefore a valuable source for learning lessons about
in SBSA. We believe a balance must be found that is summative assessment applied to simulation as a whole
ethically and pedagogically satisfactory, does not induce [22, 85, 155]. Knowledge from OSCE summative assess-
a cognitive dilemma for the trainer, and is practically ment needs to be supplemented so that simulation can
manageable. perform summative assessment according to good evi-
dence-based practices. Given the high stakes of SBSA, we
Trainers for simulation‑based summative assessment believe it necessary to rigorously and methodically adapt
The skills and training of trainers required for SBSA are what is already validated during implementation (e.g.,
crucial and must be defined [136, 153]. We consider scenarios, tools) and to proceed with caution for what
that skills and training for SBSA closely mirror skills has not yet been validated. As described above, many
and training needed for formative assessment in simu- steps and prerequisites are necessary for optimal imple-
lation. This continuity is part of the pedagogical align- mentation, including (but not limited to) identifying
ment. These different steps have common characteristics objectives; identifying and validating assessment tools;
(e.g., rules in simulation, scenario flow) and specific ones preparing simulations scenarios, trainers, and raters; and
(e.g., using assessment tools, validating competence). To planning a global strategy beginning from integrating the
ensure pedagogical continuity, the trainers who super- summative assessment in the curriculum to the manag-
vise these courses must be trained in and be masterful in ing the consequences of this assessment. SBSA must be
simulation, adhering to pedagogical theories. We believe conducted within a strict framework for its own sake and
training for SBSA represents new skills and a poten- that of the people involved. Poor implementation would
tially greater cognitive load for the trainers. It is neces- be detrimental to all participants, trainers, and the prac-
sary to provide solutions to both of these issues. For the tice SBSA. This risk is greater for recertification than for
new skills of trainers, we consider it necessary to adapt certification [156], while initial training is able to accom-
or complete the training of trainers by integrating knowl- modate SBSA easily because it is familiar (e.g., train-
edge and skills needed for properly conducting SBSA: ees engage in OSCEs at some point in their education),
good assessment practices, assessment bias mitigation, including SBSA in recertifying practicing professionals is
rater calibration, mastery of assessment tools, etc. [154]. not as obvious and may be context-dependent [157]. We
To optimize the cognitive load induced by the tasks and understand that the consequences of failed recertification
challenges of SBSA, we suggest that it could be helpful are potentially more impactful, both psychologically and
to divide the tasks between the different trainers’ roles. for professional practice. We believe that solutions must
We believe that conducting a SBSA therefore requires be developed, tested, and validated that both fill gaps and
three types of trainers whose training is adapted to their preserve professionals and patients. Implementing SBSA
specific role. First, three are the trainer-designers who therefore must be progressive, rigorous, and evidence-
are responsible for designing the assessment situation, based to be accepted and successful [158].
Buléon et al. Advances in Simulation (2022) 7:42 Page 16 of 21

Limitations debriefing, and trainers) (Fig.  1). We modestly hope that

This work has some limitations that should be empha- this work can help reflection on SBSA for future investiga-
sized. First, this work covers only a limited number of tions and can drive guideline development for SBSA.
issues related to SBSA. The entire topic is possibly not
covered and we may not have explored other questions
Abbreviations
of interest. Nevertheless, the NGT methodology allowed GDPR: General data protection regulation; NGT: Nominal group technique;
this work to focus on those issues that were most relevant OSCE: Objective structured clinical examination; SBSA: Simulation-based sum‑
and challenging to the panel. Second, the literature review mative assessment.
method (state-of-the-art literature reviews expanded Acknowledgements
with a snowball technique) does not guarantee exhaus- The authors thank SoFraSimS Assessment with simulation group members:
tiveness, and publications on the topic may have escaped Anne Bellot, Isabelle Crublé, Guillaume Philippot, Thierry Vanderlinden,
Sébastien Batrancourt, Claire Boithias-Guerot, Jean Bréaud, Philine de Vries,
the screening phase. However, it is likely that we have Louis Sibert, Thierry Sécheresse, Virginie Boulant, Louis Delamarre, Laurent
identified key articles focused on the questions explored. Grillet, Marianne Jund, Christophe Mathurin, Jacques Berthod, Blaise Debien,
Potentially unidentified articles would therefore either and Olivier Gacia who have contributed to this work. The authors thank the
external experts committee members: Guillaume Der Sahakian, Sylvain Boet,
not be important to the topic or would address ques- Denis Oriot and Jean-Michel Chabot; and the SoFraSimS executive Committee
tions not selected by the NGT. Third, this work was done for their review and feedback.
by a French-speaking group, and a Francophone-specific
Author’s contributions
approach to simulation, although not described to our CB helped with the study conception and design, data contribution, data
knowledge, cannot be ruled out. This risk is reduced by analysis, data interpretation, writing, visualization, review, and editing. FL
the fact that the work is based on international literature helped with the study conception and design, data contribution, data analysis,
data interpretation, writing, review, and editing. RDM, JWR, and DB helped
from different specialties in different countries and that with the study writing, and review and editing. JWR and DB helped with the
the panelists and reviewers were from different countries. data interpretation, writing, and review and editing. LM, FJL, EG, ALP, OB, and
Fourth, the analysis and discussion of the consequences of ALS helped with the data contribution, data analysis, data interpretation, and
review. The authors read and approved the final manuscript.
SBSA were focused on psychological consequences. This
does not cover the full range of consequences including Funding
the impact on subsequent curricula or career pathways. This work has been supported by the French Speaking Society for Simulation
in Healthcare (SoFraSimS).
Data in the literature exist on the subject and probably This work is a part of CB PhD which has been support by grants from the
deserve a specific body of work. Despite these limitations, French Society for Anesthesiology and Intensive Care (SFAR), the Arthur
however, we believe this work is valuable because it raises Sachs-Harvard Foundation, the University Hospital of Caen, the North-West
University Hospitals Group (G4), and the Charles Nicolle Foundation. Funding
questions and offers some leads toward solutions. bodies did not have any role in the design of the study, collection, analysis,
and interpretation of the data and in writing the manuscript.

Availability of data and materials

Conclusions All data generated or analyzed during this study are included in this published
The use of SBSA is very promising with a growing demand article.
for its application. Indeed, SBSA is a logical extension of
simulation-based formative assessment and competency- Declarations
based medical education development. It is probably wise
Ethics approval and consent to participate
to anticipate and plan for approaches to SBSA, as many Not applicable.
important moving parts, questions, and consequences are
emerging. Clearly identifying these elements and their Consent for publication
Not applicable.
interactions will aid in developing reliable, accurate, and
reproducible models. All this requires a meticulous and Competing interests
rigorous approach to preparation commensurate with The authors declare that they have no competing interests.

the challenges of certifying or recertifying the skills of Author details

healthcare professionals. We have explored the current 1
 Department of Anesthesiology, Intensive Care and Perioperative Medicine,
knowledge on SBSA and have now shared an initial map- Caen Normandy University Hospital, 6th Floor, Caen, France. 2 Medical School,
University of Caen Normandy, Caen, France. 3 Center for Medical Simulation,
ping of the topic. Among the seven topics investigate, we Boston, MA, USA. 4 Department of Anesthesiology, Intensive Care and Perio‑
have identified (i) areas with robust evidence (what can perative Medicine, Nîmes University Hospital, Nîmes, France. 5 Department
be assessed with simulation?); (ii) areas with limited evi- of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital,
Boston, MA, USA. 6 Harvard Medical School, Boston, MA, USA. 7 Department
dence that can be assisted by expert opinion and research of Anesthesiology, Intensive Care and Perioperative Medicine, Liège Univer‑
(assessment tools, scenarios, and implementation); and sity Hospital, Liège, Belgique. 8 Department of Emergency Medicine, Pitié
(iii) areas with weak or emerging evidence requiring Salpêtrière University Hospital, APHP, Paris, France. 9 Department of Pediatric
Intensive Care, Pellegrin University Hospital, Bordeaux, France. 10 Depart‑
guidance by expert opinion and research (consequences, ment of Emergency Medicine, Rouen University Hospital, Rouen, France.
 Buléon et al. Advances in Simulation (2022) 7:42 Page 17 of 21

11
 Department of Anesthesiology, Intensive Care and Perioperative Medicine, 19. ten Cate O, Scheele F. Competency-based postgraduate training: can
Kremlin Bicêtre University Hospital, APHP, Paris, France. 12 Department of Emer‑ we bridge the gap between theory and clinical practice? Acad Med.
gency Medicine, Cochin University Hospital, APHP, Paris, France. 2007;82:542–7.
20. Miller GE. The assessment of clinical skills/competence/performance.
Received: 2 March 2022 Accepted: 30 November 2022 Acad Med. 1990;65:S63-67.
21. Epstein RM. Assessment in medical education. N Engl J Med.
2007;356:387–96.
22. Boulet JR, Murray DJ. Simulation-based assessment in anesthesiol‑
ogy: requirements for practical implementation. Anesthesiology.
References 2010;112:1041–52.
1. van der Vleuten CPM, Schuwirth LWT. Assessment in the context 23. Bédard D, Béchard JP. L’innovation pédagogique dans le supérieur : un
of problem-based learning. Adv Health Sci Educ Theory Pract. vaste chantier. Innover dans l’enseignement supérieur. Paris: Presses
2019;24:903–14. Universitaires de France; 2009. p. 29–43.
2. Boulet JR. Summative assessment in medicine: the promise of simula‑ 24. Biggs J. Enhancing teaching through constructive alignment. High
tion for high-stakes evaluation. Acad Emerg Med. 2008;15:1017–24. Educ [Internet]. 1996 [cited 2020 Oct 25];32:347–64. Available from:
3. Green M, Tariq R, Green P. Improving patient safety through simula‑ https://​doi.​org/​10.​1007/​BF001​38871.
tion training in anesthesiology: where are we? Anesthesiol Res Pract. 25. Wong AK. Full scale computer simulators in anesthesia training and
2016;2016:4237523. evaluation. Can J Anaesth. 2004;51:455–64.
4. Krage R, Erwteman M. State-of-the-art usage of simulation in anesthe‑ 26. Messick S. Evidence and ethics in the evaluation of tests. Educational
sia: skills and teamwork. Curr Opin Anaesthesiol. 2015;28:727–34. Researcher [Internet]. 1981 [cited 2020 Mar 19];10:9–20. Available from:
5. Askew K, Manthey DE, Potisek NM, Hu Y, Goforth J, McDonough K, et al. http://journals.sagepub.com/doi/https://​doi.​org/​10.​3102/​00131​89X01​
Practical application of assessment principles in the development of an 00090​09.
innovative clinical performance evaluation in the entrustable profes‑ 27. Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in
sional activity era. Med Sci Educ. 2020;30:499–504. anaesthesia. Br J Anaesth. 2009;103:472–83.
6. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical 28. Schuwirth LWT, van der Vleuten CPM. Programmatic assessment and
competence. Lancet. 2001;357:945–9. Kane’s validity perspective. Med Educ. 2012;46:38–48.
7. Boulet JR, Murray D. Review article: assessment in anesthesiology 29. Brailovsky C, Charlin B, Beausoleil S, Coté S, Van der Vleuten C. Measure‑
education. Can J Anaesth. 2012;59:182–92. ment of clinical reflective capacity early in training as a predictor of
8. Bauer D, Lahner F-M, Schmitz FM, Guttormsen S, Huwendiek S. An over‑ clinical reasoning performance at the end of residency: an experimen‑
view of and approach to selecting appropriate patient representations tal study on the script concordance test. Med Educ. 2001;35:430–6.
in teaching and summative assessment in medical education. Swiss 30. van der Vleuten CPM, Schuwirth LWT. Assessing professional compe‑
Med Wkly. 2020;150: w20382. tence: from methods to programmes. Med Educ. 2005;39:309–17.
9. Park CS. Simulation and quality improvement in anesthesiology. Anes‑ 31. Gordon M, Farnan J, Grafton-Clarke C, Ahmed R, Gurbutt D, McLachlan
thesiol Clin. 2011;29:13–28. J, et al. Non-technical skills assessments in undergraduate medical
10. Higham H, Baxendale B. To err is human: use of simulation to education: a focused BEME systematic review: BEME Guide No. 54. Med
enhance training and patient safety in anaesthesia. British Journal of Teach. 2019;41(7):732–45.
Anaesthesia [Internet]. 2017 [cited 2021 Sep 16];119:i106–14. Avail‑ 32. Jouquan J. L’évaluation des apprentissages des étudiants en formation
able from: https://​www.​scien​cedir​ect.​com/​scien​ce/​artic​le/​pii/​S0007​ médicale initiale. Pédagogie Médicale [Internet]. 2002 [cited 2020
09121​75412​15. Feb 2];3:38–52. Available from: http://www.pedagogie-medicale.
11. Mann S, Truelove AH, Beesley T, Howden S, Egan R. Resident percep‑ org/https://​doi.​org/​10.​1051/​pmed:​20020​06.
tions of competency-based medical education. Can Med Educ J. 33. Gale TCE, Roberts MJ, Sice PJ, Langton JA, Patterson FC, Carr AS, et al.
2020;11:e31-43. Predictive validity of a selection centre testing non-technical skills for
12. Khan KZ3, Ramachandran S, Gaunt K, Pushkar P. The objective struc‑ recruitment to training in anaesthesia. Br J Anaesth. 2010;105:603–9.
tured clinical examination (OSCE): AMEE Guide No. 81. Part I: an histori‑ 34. Gallagher CJ, Tan JM. The current status of simulation in the mainte‑
cal and theoretical perspective. Med Teach. 2013;35(9):e1437-1446. nance of certification in anesthesia. Int Anesthesiol Clin. 2010;48:83–99.
13. Daniels VJ, Pugh D. Twelve tips for developing an OSCE that measures 35. S DeMaria Jr ST Samuelson AD Schwartz AJ Sim AI Levine Simulation-
what you want. Med Teach. 2018;40:1208–13. based assessment and retraining for the anesthesiologist seeking
14. Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus reentry to clinical practice: a case series. Anesthesiology [Internet]. 2013
group methods such as Delphi and Nominal Group in medical educa‑ [cited 2021 Sep 6];119:206–17 Available from: https://​doi.​org/​10.​1097/​
tion research. Med Teach. 2017;39:14–9. ALN.​0b013​e3182​9761c8.
15. Haute Autorité de Santé. Recommandations par consensus formalisé 36. Amin Z, Boulet JR, Cook DA, Ellaway R, Fahal A, Kneebone R, et al.
(RCF) [Internet]. Haute Autorité de Santé. 2011 [cited 2020 Oct 29]. Technology-enabled assessment of health professions education:
Available from: https://​www.​has-​sante.​fr/​jcms/c_​272505/​fr/​recom​ consensus statement and recommendations from the Ottawa 2010
manda​tions-​par-​conse​nsus-​forma​lise-​rcf. conference. Medical Teacher [Internet]. 2011 [cited 2021 Jul 7];33:364–9.
16. Humphrey-Murto S, Varpio L, Wood TJ, Gonsalves C, Ufholz L-A, Mascioli Available from: http://www.tandfonline.com/doi/full/https://​doi.​org/​10.​
K, et al. The use of the delphi and other consensus group methods in 3109/​01421​59X.​2011.​565832.
medical education research: a review. Academic Medicine [Internet]. 37. Scallon G. L’évaluation des apprentissages dans une approche par
2017 [cited 2021 Jul 20];92:1491–8. Available from: https://​journ​als.​ compétences. Bruxelles: De Boeck Université-Bruxelles; 2007.
lww.​com/​acade​micme​dicine/​Fullt​ext/​2017/​10000/​The_​Use_​of_​the_​ 38. Downing SM. Validity: on meaningful interpretation of assessment data.
Delphi_​and_​Other_​Conse​nsus_​Group.​38.​aspx. Med Educ. 2003;37:830–7.
17. Booth A, Sutton A, Papaioannou D. Systematic approaches to a success‑ 39. Cook DA, Hatala R. Validation of educational assessments: a primer for
ful literature review [Internet]. Second edition. Los Angeles: Sage; 2016. simulation and beyond. Adv Simul [Internet]. 2016 [cited 2021 Aug
Available from: https://​uk.​sagep​ub.​com/​sites/​defau​lt/​files/​upm-​assets/​ 24];1:31. Available from: http://advancesinsimulation.biomedcentral.
78595_​book_​item_​78595.​pdf. com/articles/https://​doi.​org/​10.​1186/​s41077-​016-​0033-y.
18. Morgan DL. Snowball Sampling. In: Given LM, editor. The Sage ency‑ 40. Kane MT. Validating the interpretations and uses of test scores. Journal
clopedia of qualitative research methods [Internet]. Los Angeles, Calif: of Educational Measurement [Internet]. 2013 [cited 2020 Sep 9];50:1–
Sage Publications; 2008. p. 815–6. Available from: http://​www.​yanch​ 73. Available from: https://onlinelibrary.wiley.com/doi/abs/https://​doi.​
ukvla​dimir.​com/​docs/​Libra​ry/​Sage%​20Enc​yclop​edia%​20of%​20Qua​litat​ org/​10.​1111/​jedm.​12000.
ive%​20Res​earch%​20Met​hods-%​202008.​pdf. 41. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to
validity arguments: a practical guide to Kane’s framework. Med Educ.
2015;49:560–75.
Buléon et al. Advances in Simulation (2022) 7:42 Page 18 of 21

42. DA Cook B Zendejas SJ Hamstra R Hatala R Brydges What counts as 58. Pires S, Monteiro S, Pereira A, Chaló D, Melo E, Rodrigues A. Non-tech‑
validity evidence? Examples and prevalence in a systematic review of nical skills assessment for prelicensure nursing students: an integrative
simulation-based assessment. Adv in Health Sci Educ [Internet]. 2014 review. Nurse Educ Today. 2017;58:19–24.
[cited 2020 Feb 2];19:233–50 Available from: https://​doi.​org/​10.​1007/​ 59. Khan R, Payne MWC, Chahine S. Peer assessment in the objec‑
s10459-​013-​9458-4. tive structured clinical examination: a scoping review. Med Teach.
43. Cook DA, Lineberry M. Consequences validity evidence: evaluating the 2017;39:745–56.
impact of educational assessments. Acad Med [Internet]. 2016 [cited 60. Hegg RM, Ivan KF, Tone J, Morten A. Comparison of peer assessment
2020 Oct 24];91:785–95. Available from: http://​journ​als.​lww.​com/​00001​ and faculty assessment in an interprofessional simulation-based
888-​20160​6000-​00018. team training program. Nurse Educ Pract. 2019;42: 102666.
44. Tavakol M, Dennick R. Post-examination analysis of objective tests. Med 61. Scavone BM, Sproviero MT, McCarthy RJ, Wong CA, Sullivan JT, Siddall
Teach. 2011;33:447–58. VJ, et al. Development of an objective scoring system for measure‑
45. Messick S. The interplay of evidence and consequences in the valida‑ ment of resident performance on the human patient simulator.
tion of performance assessments. Educational Researcher [Internet]. Anesthesiology. 2006;105:260–6.
1994 [cited 2021 Feb 15];23:13–23. Available from: http://journals. 62. Oriot D, Bridier A, Ghazali DA. Development and assessment of an
sagepub.com/doi/https://​doi.​org/​10.​3102/​00131​89X02​30020​13. evaluation tool for team clinical performance: the Team Average
46. Validity MS. Education measurement. 3rd ed. New York: R. L. Linn; 1989. Performance Assessment Scale (TAPAS). Health Care : Current Reviews
p. 13–103. [Internet]. 2016 [cited 2018 Jan 17];4:1–7. Available from: https://​www.​
47. Oriot D, Darrieux E, Boureau-Voultoury A, Ragot S, Scépi M. Validation omics​online.​org/​open-​access/​devel​opment-​and-​asses​sment-​of-​an-​
of a performance assessment scale for simulated intraosseous access. evalu​ation-​tool-​for-​team-​clini​calpe​r form​ance-​the-​team-​avera​ge-​perfo​
Simul Healthc. 2012;7:171–5. rmance-​asses​sment-​scale-​tapas-​2375-​4273-​10001​64.​php?​aid=​72394.
48. Guise J-M, Deering SH, Kanki BG, Osterweil P, Li H, Mori M, et al. 63. Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills.
Validation of a tool to measure and promote clinical teamwork. Simul Br J Anaesth. 2010;105:38–44.
Healthc. 2008;3:217–23. 64. Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS System:
49. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of reliability and validity of a tool for measuring teamwork behaviour
instruments or scales for use in cross-cultural health care research: in the operating theatre. Quality and Safety in Health Care [Internet].
a clear and user-friendly guideline: Validation of instruments or 2009 [cited 2021 Jul 6];18:104–8. Available from: https://qualitysafety.
scales. Journal of Evaluation in Clinical Practice . 2011 [cited 2022 bmj.com/lookup/doi/https://​doi.​org/​10.​1136/​qshc.​2007.​024760.
Jul 22];17:268–74. Available from: https://onlinelibrary.wiley.com/ 65. Cooper S, Cant R, Porter J, Sellick K, Somers G, Kinsman L, et al. Rat‑
doi/https://​doi.​org/​10.​1111/j.​1365-​2753.​2010.​01434.x. ing medical emergency teamwork performance: development of
50. Stoyanova-Piroth G, Milanov I, Stambolieva K. Translation, adaptation the Team Emergency Assessment Measure (TEAM). Resuscitation.
and validation of the Bulgarian version of the King’s Parkinson’s Disease 2010;81:446–52.
Pain Scale. BMC Neurol [Internet]. 2021 [cited 2022 Jul 22];21:357. Avail‑ 66. Adler MD, Trainor JL, Siddall VJ, McGaghie WC. Development and
able from: https://bmcneurol.biomedcentral.com/articles/https://​doi.​ evaluation of high-fidelity simulation case scenarios for pediatric
org/​10.​1186/​s12883-​021-​02392-5. resident education. Ambul Pediatr. 2007;7:182–6.
51. Behari M, Srivastava A, Achtani R, Nandal N, Dutta R. Pain assessment in 67. Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA. Linking simula‑
Indian Parkinson’s disease patients using King’s Parkinson’s disease pain tion-based educational assessments and patient-related outcomes: a
scale. Ann Indian Acad Neurol [Internet]. 2020 [cited 2022 Jul 22];0:0. systematic review and meta-analysis. Acad Med. 2015;90:246–56.
Available from: http://​www.​annal​sofian.​org/​prepr​intar​ticle.​asp?​id=​ 68. Cazzell M, Howe C. Using Objective Structured Clinical Evaluation for
300170;​type=0. Simulation Evaluation: Checklist Considerations for Interrater Reli‑
52. Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of ability. Clinical Simulation In Nursing [Internet]. 2012;8(6):e219–25.
health-related quality of life measures: literature review and proposed [cited 2019 Dec 14] Available from: https://​www.​nursi​ngsim​ulati​on.​
guidelines. Journal of Clinical Epidemiology [Internet]. 1993 [cited 2022 org/​artic​le/​S1876-​1399(11)​00249-0/​abstr​act.
Jul 22];46:1417–32. Available from: https://​linki​nghub.​elsev​ier.​com/​retri​ 69. Maignan M, Viglino D, Collomb Muret R, Vejux N, Wiel E, Jacquin L, et al.
eve/​pii/​08954​35693​90142N. Intensity of care delivered by prehospital emergency medical service
53. Franc JM, Verde M, Gallardo AR, Carenzo L, Ingrassia PL. An Italian ver‑ physicians to patients with deliberate self-poisoning: results from a
sion of the Ottawa crisis resource management global rating scale: a 2-day cross-sectional study in France. Intern Emerg Med. 2019;14:981–8.
reliable and valid tool for assessment of simulation performance. Intern 70. Alcaraz-Mateos E, Jiang X “Sara,” Mohammed AAR, Turic I, Hernández-
Emerg Med. 2017;12:651–6. Sabater L, Caballero-Alemán F, et al. A novel simulator model and
54. Gosselin É, Marceau M, Vincelette C, Daneau C-O, Lavoie S, Ledoux standardized assessment tools for fine needle aspiration cytology train‑
I. French translation and validation of the Mayo High Performance ing. Diagn Cytopathol [Internet]. 2019 [cited 2020 Feb 3];47:297–301.
Teamwork Scale for nursing students in a high-fidelity simulation Available from: http://doi.wiley.com/https://​doi.​org/​10.​1002/​dc.​24105.
context. Clinical Simulation in Nursing [Internet]. 2019 [cited 2022 Jul 71. I Ghaderi M Vaillancourt G Sroka PA Kaneva MC Vassiliou I Choy
25];30:25–33. Available from: https://​linki​nghub.​elsev​ier.​com/​retri​eve/​ Evaluation of surgical performance during laparoscopic incisional
pii/​S1876​13991​83018​90. hernia repair: a multicenter study. Surg Endosc [Internet]. et al 2011
55. Sánchez-Marco M, Escribano S, Cabañero-Martínez M-J, Espinosa- [cited 2020 Feb 2];25:2555–63 Available from: https://​doi.​org/​10.​1007/​
Ramírez S, José Muñoz-Reig M, Juliá-Sanchis R. Cross-cultural adapta‑ s00464-​011-​1586-4.
tion and validation of two crisis resource management scales. Interna‑ 72. IJgosse WM, Leijte E, Ganni S, Luursema J-M, Francis NK, Jakimowicz JJ,
tional Emergency Nursing [Internet]. 2021 [cited 2022 Jul 25];57:101016. et al. Competency assessment tool for laparoscopic suturing: develop‑
Available from: https://​www.​scien​cedir​ect.​com/​scien​ce/​artic​le/​pii/​ ment and reliability evaluation. Surg Endosc. 2020;34(7):2947–53.
S1755​599X2​10005​49. 73. Pelaccia T, Tardif J. In: Comment [mieux] former et évaluer les étudiants
56. Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: from en médecine et en sciences de la santé? 1ère. Louvain-la-Neuve: De
assessment of learning to assessment for learning. Medical Teacher Boeck supérieur; 2016. p. 343–56. (Guides pratiques).
[Internet]. 2011 [cited 2021 Sep 6];33:478–85. Available from: http:// 74. Henricksen JW, Altenburg C, Reeder RW. Operationalizing healthcare
www.tandfonline.com/doi/full/https://​doi.​org/​10.​3109/​01421​59X.​2011.​ simulation psychological safety: a descriptive analysis of an interven‑
565828. tion. Simul Healthc. 2017;12:289–97.
57. Maignan M, Koch F-X, Chaix J, Phellouzat P, Binauld G, Collomb Muret R, 75. Gaba DM. Simulations that are challenging to the psyche of partici‑
et al. Team Emergency Assessment Measure (TEAM) for the assessment pants: how much should we worry and about what? Simulation in
of non-technical skills during resuscitation: validation of the French Healthcare: The Journal of the Society for Simulation in Healthcare
version. Resuscitation [Internet]. 2016 [cited 2019 Mar 12];101:115–20. [Internet]. 2013 [cited 2020 Mar 17];8:4–7. Available from: http://​journ​
Available from: http://​www.​scien​cedir​ect.​com/​scien​ce/​artic​le/​pii/​S0300​ als.​lww.​com/​01266​021-​20130​2000-​00002.
95721​50089​89.
 Buléon et al. Advances in Simulation (2022) 7:42 Page 19 of 21

76. Ghazali DA, Breque C, Sosner P, Lesbordes M, Chavagnat J-J, Ragot 2];9:e25–33. Available from: http://​www.​scien​cedir​ect.​com/​scien​ce/​
S, et al. Stress response in the daily lives of simulation repeaters. A artic​le/​pii/​S1876​13991​10012​77.
randomized controlled trial assessing stress evolution over one year of 96. Frey-Vogel AS, Scott-Vernaglia SE, Carter LP, Huang GC. Simulation for
repetitive immersive simulations. PLoS One. 2019;14(7):e0220111. milestone assessment: use of a longitudinal curriculum for pediatric
77. Rudolph JW, Simon R, Raemer DB, Eppich WJ. Debriefing as formative residents. Simul Healthc. 2016;11:286–92.
assessment: closing performance gaps in medical education. Acad 97. Durning SJ, Artino A, Boulet J, La Rochelle J, Van der Vleuten C, Arze B,
Emerg Med. 2008;15:1010–6. et al. The feasibility, reliability, and validity of a post-encounter form for
78. Kang SJ, Min HY. Psychological safety in nursing simulation. Nurse Educ. evaluating clinical reasoning. Med Teach. 2012;34:30–7.
2019;44:E6-9. 98. Stone J. Moving interprofessional learning forward through formal
79. Howard SK, Gaba DM, Smith BE, Weinger MB, Herndon C, Keshavacha‑ assessment. Medical Education. 2010 [cited 2020 Feb 12];44:396–403.
rya S, et al. Simulation study of rested versus sleep-deprived anesthesi‑ Available from: http://doi.wiley.com/https://​doi.​org/​10.​1111/j.​1365-​
ologists. Anesthesiology. 2003;98(6):1345–55. 2923.​2009.​03607.x.
80. Neuschwander A, Job A, Younes A, Mignon A, Delgoulet C, Cabon P, 99. Manser T, Dieckmann P, Wehner T, Rallf M. Comparison of anaesthetists’
et al. Impact of sleep deprivation on anaesthesia residents’ non-tech‑ activity patterns in the operating room and during simulation. Ergo‑
nical skills: a pilot simulation-based prospective randomized trial. Br J nomics. 2007;50:246–60.
Anaesth. 2017;119:125–31. 100. Perrenoud P. Évaluation formative et évaluation certificative : postures
81. Eastridge BJ, Hamilton EC, O’Keefe GE, Rege RV, Valentine RJ, Jones contradictoires ou complémentaires ? Formation Professionnelle suisse
DJ, et al. Effect of sleep deprivation on the performance of simulated . 2001 [cited 2020 Oct 29];4:25–8. Available from: https://​www.​unige.​
laparoscopic surgical skill. Am J Surg. 2003;186:169–74. ch/​fapse/​SSE/​teach​ers/​perre​noud/​php_​main/​php_​2001/​2001_​13.​
82. Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability html.
and validity of a simulation-based acute care skills assessment for medi‑ 101. Atesok K, Hurwitz S, Anderson DD, Satava R, Thomas GW, Tufescu T,
cal students and residents. Anesthesiology. 2003;99:1270–80. et al. Advancing simulation-based orthopaedic surgical skills train‑
83. Levine AI, Flynn BC, Bryson EO, Demaria S. Simulation-based Main‑ ing: an analysis of the challenges to implementation. Adv Orthop.
tenance of Certification in Anesthesiology (MOCA) course optimi‑ 2019;2019:1–7.
zation: use of multi-modality educational activities. J Clin Anesth. 102. Chiu M, Tarshis J, Antoniou A, Bosma TL, Burjorjee JE, Cowie N, et al.
2012;24:68–74. Simulation-based assessment of anesthesiology residents’ competence:
84. Boulet JR, Murray D, Kras J, Woodhouse J. Setting performance stand‑ development and implementation of the Canadian National Anesthe‑
ards for mannequin-based acute-care scenarios: an examinee-centered siology Simulation Curriculum (CanNASC). Can J Anesth/J Can Anesth.
approach. Simul Healthc. 2008;3:72–81. 2016 [cited 2020 Feb 2];63:1357–63. Available from: https://​doi.​org/​10.​
85. Furman GE, Smee S, Wilson C. Quality assurance best practices for 1007/​s12630-​016-​0733-8.
simulation-based examinations. Simul Healthc. 2010;5:226–31. 103. TC Everett RJ McKinnon E Ng P Kulkarni BCR Borges M Letal Simu‑
86. Kane MT. The assessment of professional competence. Eval Health Prof lation-based assessment in anesthesia: an international multicen‑
[Internet]. 1992 [cited 2022 Jul 22];15:163–82. Available from: http:// tre validation study. Can J Anesth, J Can Anesth. et al 2019 [cited
journals.sagepub.com/doi/https://​doi.​org/​10.​1177/​01632​78792​01500​ 2020 Feb 2];66:1440–9 Available from: https://​doi.​org/​10.​1007/​
203. s12630-​019-​01488-4.
87. Blum RH, Boulet JR, Cooper JB, Muret-Wagstaff SL. Harvard Assessment 104. Regulation (EU) 2016/679 of the European Parliament and of the
of Anesthesia Resident Performance Research Group. Simulation-based Council of 27 April 2016 on the protection of natural persons with
assessment to identify critical gaps in safe anesthesia resident perfor‑ regard to the processing of personal data and on the free movement of
mance. Anesthesiol. 2014;120(1):129–41. such data, and repealing Directive 95/46/EC (General Data Protection
88. Rizzolo MA, Kardong-Edgren S, Oermann MH, Jeffries PR. The national Regulation) (Text with EEA relevance). May 4, 2016. Available from:
league for nursing project to explore the use of simulation for http://​data.​europa.​eu/​eli/​reg/​2016/​679/​2016-​05-​04/​eng.
high-stakes assessment: process, outcomes, and recommendations: 105. Commission Nationale de l’Informatique et des Libertés. RGPD : passer
nursing education perspectives . 2015 [cited 2020 Feb 3];36:299–303. à l’action. 2021 [cited 2021 Jul 8]. Available from: https://​www.​cnil.​fr/​fr/​
Available from: http://​Insig​hts.​ovid.​com/​cross​ref?​an=​00024​776-​20150​ rgpd-​passer-​a-​lacti​on.
9000-​00006. 106. Ten Cate O, Regehr G. The power of subjectivity in the assessment of
89. Mudumbai SC, Gaba DM, Boulet JR, Howard SK, Davies MF. External medical trainees. Acad Med. 2019;94:333–7.
validation of simulation-based assessments with other performance 107. Weller JM, Robinson BJ, Jolly B, Watterson LM, Joseph M, Bajenov S,
measures of third-year anesthesiology residents. Simul Healthc. et al. Psychometric characteristics of simulation-based assessment
2012;7:73–80. in anaesthesia and accuracy of self-assessed scores. Anaesthesia.
90. Fanning RM, Gaba DM. The role of debriefing in simulation-based learn‑ 2005;60:245–50.
ing. Simul Healthc. 2007;2:115–25. 108. Wikander L, Bouchoucha SL. Facilitating peer based learning through
91. Savoldelli GL, Naik VN, Park J, Joo HS, Chow R, Hamstra SJ. Value of summative assessment - an adaptation of the objective structured
debriefing during simulated crisis managementoral versus video- clinical assessment tool for the blended learning environment. Nurse
assisted oral feedback. Anesthesiology . American Society of Anesthe‑ Educ Pract. 2018;28:40–5.
siologists; 2006 [cited 2020 Oct 19];105:279–85. Available from: https://​ 109. Gaugler BB, Rudolph AS. The influence of assessee performance varia‑
pubs.​asahq.​org/​anest​hesio​logy/​artic​le/​105/2/​279/​6669/​Value-​of-​Debri​ tion on assessors’ judgments. Pers Psychol. 1992;45:77–98.
efing-​during-​Simul​ated-​Crisis. 110. Feldman M, Lazzara EH, Vanderbilt AA, DiazGranados D. Rater training
92. Haute Autorité de Santé. Guide de bonnes pratiques en simulation en to support high-stakes simulation-based assessments. J Contin Educ
santé . 2012 [cited 2020 Feb 2]. Available from: https://​www.​has-​sante.​ Health Prof. 2012 [cited 2019 Dec 14];32:279–86. Available from: https://​
fr/​upload/​docs/​appli​cation/​pdf/​2013-​01/​guide_​bonnes_​prati​ques_​ www.​ncbi.​nlm.​nih.​gov/​pmc/​artic​les/​PMC36​46087/.
simul​ation_​sante_​guide.​pdf. 111. Pelgrim E a. M, Kramer AWM, Mokkink HGA, van den Elsen L, Grol RPTM,
93. INACSL Standards Committee. INACSL Standards of best practice: simu‑ van der Vleuten CPM. In-training assessment using direct observation
lation. Simulation design. Clinical Simulation In Nursing . 2016 [cited of single-patient encounters: a literature review. Adv Health Sci Educ
2020 Feb 2];12:S5–12. Available from: https://​www.​nursi​ngsim​ulati​on.​ Theory Pract. 2011;16(1):131–42.
org/​artic​le/​S1876-​1399(16)​30126-8/​abstr​act. 112. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defen‑
94. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Crite‑ sible absolute passing scores on performance examinations in health
ria for good assessment: consensus statement and recommendations professions education. Teach Learn Med. 2006;18:50–7.
from the Ottawa 2010 Conference. Med Teach. 2011;33:206–14. 113. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based
95. Gantt LT. The effect of preparation on anxiety and performance in sum‑ objective structured clinical examination into the Israeli National Board
mative simulations. Clinical Simulation in Nursing. 2013 [cited 2020 Feb Examination in Anesthesiology. Anesth Analg. 2006;102:853–8.
Buléon et al. Advances in Simulation (2022) 7:42 Page 20 of 21

114. Hedge JW, Kavanagh MJ. Improving the accuracy of performance of qualitative literature. JBI Database System Rev Implement Rep.
evaluations: comparison of three methods of performance appraiser 2016;14:96–137.
training. J Appl Psychol. 1988;73:68–73. 133. Leblanc VR. Review article: simulation in anesthesia: state of the sci‑
115. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of ence and looking forward. Can J Anaesth. 2012;59:193–202.
clinical competence using objective structured examination. Br Med J. 134. Hanscom R. Medical simulation from an insurer’s perspective. Acad
1975;1:447–51. Emerg Med. 2008;15:984–7.
116. Uzan S. Mission de recertification des médecins - Exercer une 135. McCarthy J, Cooper JB. Malpractice insurance carrier provides
médecine de qualit | Vie publique.fr. Ministère des Solidarités et de premium incentive for simulation-based training and believes it has
la Santé - Ministère de l’Enseignement supérieur, de la Recherche et made a difference. Anesth Patient Saf Found Newsl. 2007 [cited 2021
de l’Innovation; 2018 Nov. Available from: https://​www.​vie-​publi​que.​ Sep 17];17. Available from: https://​www.​apsf.​org/​artic​le/​malpr​actice-​
fr/​rappo​r t/​37741-​missi​on-​de-​recer​tific​ation-​des-​medec​ins-​exerc​er-​ insur​ance-​carri​er-​provi​des-​premi​um-​incen​tive-​for-​simul​ation-​based-​
une-​medec​ine-​de-​qualit. train​ing-​and-​belie​ves-​it-​has-​made-a-​diffe​rence/.
117. Mann KV, MacDonald AC, Norcini JJ. Reliability of objective structured 136. Edler AA, Fanning RG, Chen MichaelI, Claure R, Almazan D, Struyk B,
clinical examinations: four years of experience in a surgical clerkship. et al. Patient simulation: a literary synthesis of assessment tools in
Teaching and Learning in Medicine. 1990 [cited 2021 May 1];2:219– anesthesiology. J Educ Eval Health Prof. 2009 [cited 2021 Sep 17];6:3.
24. Available from: http://www.tandfonline.com/doi/abs/https://​doi.​ Available from: https://​www.​ncbi.​nlm.​nih.​gov/​pmc/​artic​les/​PMC27​
org/​10.​1080/​10401​33900​95394​64. 96725/.
118. Maintenance Of Certification in Anesthesiology (MOCA) 2.0. [cited 137. Borgersen NJ, Naur TMH, Sørensen SMD, Bjerrum F, Konge L, Subhi
2021 Sep 18]. Available from: https://​theaba.​org/​about%​20moca%​ Y, et al. Gathering validity evidence for surgical simulation: a system‑
202.0.​html. atic review. Annals of Surgery. 2018 [cited 2022 Sep 25];267:1063–8.
119. Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The Objective Available from: https://​journ​als.​lww.​com/​00000​658-​20180​
Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: 6000-​00014.
Organisation & Administration. Med Teach. 2013 [cited 2020 Oct 138. Rudolph JW, Raemer DB, Simon R. Establishing a safe container for
29];35:e1447–63. Available from: http://www.tandfonline.com/doi/ learning in simulation: the role of the presimulation briefing. Simul
full/https://​doi.​org/​10.​3109/​01421​59X.​2013.​818635. Healthc. 2014;9:339–49.
120. Coderre S, Woloschuk W, McLaughlin K. Twelve tips for blueprinting. 139. Cilliers FJ, Schuwirth LW, Adendorff HJ, Herman N, van der Vleuten
Med Teach. 2009;31:322–4. CP. The mechanism of impact of summative assessment on
121. Murray DJ, Boulet JR. Anesthesiology board certification changes: a medical students’ learning. Adv Health Sci Educ Theory Pract.
real-time example of “assessment drives learning.” Anesthesiology. 2010;15:695–715.
2018;128:704–6. 140. Hadi MA, Ali M, Haseeb A, Mohamed MMA, Elrggal ME, Cheema E.
122. Roberts C, Newble D, Jolly B, Reed M, Hampton K. Assuring the qual‑ Impact of test anxiety on pharmacy students’ performance in Objec‑
ity of high-stakes undergraduate assessments of clinical compe‑ tive Structured Clinical Examination: a cross-sectional survey. Int J
tence. Med Teach. 2006;28:535–43. Pharm Pract. 2018;26:191–4.
123. Newble D. Techniques for measuring clinical competence: objective 141. Dunn W, Dong Y, Zendejas B, Ruparel R, Farley D. Simulation, mastery
structured clinical examinations. Med Educ. 2004;38:199–203. learning and healthcare. Am J Med Sci. 2017;353:158–65.
124. Der Sahakian G, Lecomte F, Buléon C, Guevara F, Jaffrelot M, Alinier G. 142. McGaghie WC. Mastery learning: it is time for medical education to
Référentiel sur l’élaboration de scénarios de simulation en immersion join the 21st century. Acad Med. 2015;90:1438–41.
clinique.  Paris: Société Francophone de Simulation en Santé; 2017 p. 143. Ng C, Primiani N, Orchanian-Cheff A. Rapid cycle deliberate
22. Available from: https://​sofra​sims.​org/​wp-​conte​nt/​uploa​ds/​2019/​ practice in healthcare simulation: a scoping review. Med Sci Educ.
10/R%​C3%​A9f%​C3%​A9ren​tiel-​Scena​rio-​Simul​ation-​Sofra​sims.​pdf. 2021;31:2105–20.
125. Lewis KL, Bohnert CA, Gammon WL, Hölzer H, Lyman L, Smith C, et al. 144. Taras J, Everett T. Rapid cycle deliberate practice in medical educa‑
The Association of Standardized Patient Educators (ASPE) Standards tion - a systematic review. Cureus. 2017;9: e1180.
of Best Practice (SOBP). Adv Simul. 2017;2:10. 145. Cleland JA, Abe K, Rethans J-J. The use of simulated patients in medi‑
126. Board of Directors of the American Board of Medical Specialties cal education: AMEE Guide No 42. Med Teach. 2009;31:477–86.
(ABMS). Standards for the ABMS Program for Maintenance of Certifi‑ 146. Garden AL, Le Fevre DM, Waddington HL, Weller JM. Debriefing
cation (MOC). American Board of Medical Specialties; 2014 Jan p. 13. after simulation-based non-technical skill training in healthcare:
Available from: https://​www.​abms.​org/​media/​1109/​stand​ards-​for-​ a systematic review of effective practice. Anaesth Intensive Care.
the-​abms-​progr​am-​for-​moc-​final.​pdf. 2015;43:300–8.
127. Hodges B, McNaughton N, Regehr G, Tiberius R, Hanson M. The chal‑ 147. Sawyer T, Eppich W, Brett-Fleegler M, Grant V, Cheng A. More than
lenge of creating new OSCE measures to capture the characteristics one way to debrief: a critical review of healthcare simulation debrief‑
of expertise. Med Educ. 2002;36:742–8. ing methods. Simul Healthc. 2016;11:209–17.
128. Hays RB, Davies HA, Beard JD, Caldon LJM, Farmer EA, Finucane PM, 148. Rudolph JW, Simon R, Dufresne RL, Raemer DB. There’s no such thing
et al. Selecting performance assessment methods for experienced as “nonjudgmental” debriefing: a theory and method for debriefing
physicians. Med Educ. 2002;36:910–7. with good judgment. Simul Healthc. 2006;1:49–55.
129. Ram P, Grol R, Rethans JJ, Schouten B, van der Vleuten C, Kester A. 149. Levett-Jones T, Lapkin S. A systematic review of the effectiveness of
Assessment of general practitioners by video observation of commu‑ simulation debriefing in health professional education. Nurse Educ
nicative and medical performance in daily practice: issues of validity, Today. 2014;34:e58-63.
reliability and feasibility. Med Educ. 1999;33:447–54. 150. Palaganas JC, Fey M, Simon R. Structured debriefing in simulation-
130. Weersink K, Hall AK, Rich J, Szulewski A, Dagnone JD. Simulation based education. AACN Adv Crit Care. 2016;27:78–85.
versus real-world performance: a direct comparison of emergency 151. Rudolph JW, Foldy EG, Robinson T, Kendall S, Taylor SS, Simon R.
medicine resident resuscitation entrustment scoring. Adv Simul. Helping without harming: the instructor’s feedback dilemma in
2019 [cited 2020 Feb 12];4:9. Available from: https://advancesin‑ debriefing–a case study. Simul Healthc. 2013;8:304–16.
simulation.biomedcentral.com/articles/https://​doi.​org/​10.​1186/​ 152. Larsen DP, Butler AC, Roediger III HL. Test-enhanced learning in medi‑
s41077-​019-​0099-4. cal education. Medical Education. 2008 [cited 2021 Aug 25];42:959–
131. Buljac-Samardzic M, Doekhie KD, van Wijngaarden JDH. Interven‑ 66. Available from: https://onlinelibrary.wiley.com/doi/https://​doi.​
tions to improve team effectiveness within health care: a systematic org/​10.​1111/j.​1365-​2923.​2008.​03124.x.
review of the past decade. Hum Resour Health. 2020;18:2. 153. Koster MA, Soffler M. Navigate the challenges of simulation for
132. Eddy K, Jordan Z, Stephenson M. Health professionals’ experience of assessment: a faculty development workshop. MedEdPORTAL.
teamwork education in acute hospital settings: a systematic review 2021;17:11114.
 Buléon et al. Advances in Simulation (2022) 7:42 Page 21 of 21

154. Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, et al.
Testing the raters: inter-rater reliability of standardized anaesthesia
simulator performance. Can J Anaesth. 1997;44:924–8.
155. Kelly MA, Mitchell ML, Henderson A, Jeffrey CA, Groves M, Nulty DD,
et al. OSCE best practice guidelines—applicability for nursing simula‑
tions. Adv Simul. 2016 [cited 2020 Feb 3];1:10. Available from: http://
advancesinsimulation.biomedcentral.com/articles/https://​doi.​org/​10.​
1186/​s41077-​016-​0014-1.
156. Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J, Cooper
JB, et al. Simulation-based assessment of the management of criti‑
cal events by board-certified anesthesiologists. Anesthesiology.
2017;127:475–89.
157. Sinz E, Banerjee A, Steadman R, Shotwell MS, Slagle J, McIvor WR,
et al. Reliability of simulation-based assessment for practicing physi‑
cians: performance is context-specific. BMC Med Educ. 2021;21:207.
158. Ryall T, Judd BK, Gordon CJ. Simulation-based assessments in health
professional education: a systematic review. J Multidiscip Healthc.
2016;9:69–82.

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in pub‑
lished maps and institutional affiliations.

Ready to submit your research ? Choose BMC and benefit from:

• fast, convenient online submission

• thorough peer review by experienced researchers in your field
• rapid publication on acceptance
• support for research data, including large and complex data types
• gold Open Access which fosters wider collaboration and increased citations
• maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions