Medical Teacher

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/imte20

Developments in medical education in response to

the COVID-19 pandemic: A rapid BEME systematic
review: BEME Guide No. 63

Morris Gordon, Madalena Patricio, Laura Horne, Alexandra Muston,

Sebastian R. Alston, Mohan Pammi, Satid Thammasitboon, Sophie Park,
Teresa Pawlikowska, Eliot L. Rees, Andrea Jane Doyle & Michelle Daniel

To cite this article: Morris Gordon, Madalena Patricio, Laura Horne, Alexandra Muston, Sebastian
R. Alston, Mohan Pammi, Satid Thammasitboon, Sophie Park, Teresa Pawlikowska, Eliot L. Rees,
Andrea Jane Doyle & Michelle Daniel (2020) Developments in medical education in response to
the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63, Medical Teacher,
42:11, 1202-1215, DOI: 10.1080/0142159X.2020.1807484

To link to this article: https://doi.org/10.1080/0142159X.2020.1807484

View supplementary material Published online: 26 Aug 2020.

Submit your article to this journal Article views: 10640

View related articles View Crossmark data

Citing articles: 49 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=imte20
MEDICAL TEACHER
2020, VOL. 42, NO. 11, 1202–1215
https://doi.org/10.1080/0142159X.2020.1807484

BEME GUIDE

Developments in medical education in response to the COVID-19 pandemic: A

rapid BEME systematic review: BEME Guide No. 63
Morris Gordona,b, Madalena Patricioc, Laura Hornea, Alexandra Mustona, Sebastian R. Alstond,
Mohan Pammie, Satid Thammasitboone, Sophie Parkf, Teresa Pawlikowskag, Eliot L. Reesf,h ,
Andrea Jane Doyleg and Michelle Danieli
a
Blackpool Victoria Hospital, Blackpool, UK; bSchool of Medicine, University of Central Lancashire, Preston, UK; cFaculty of Medicine,
University of Lisbon, Lisboa, Portugal; dAlabama College of Osteopathic Medicine, Dothan, AL, USA; eTexas Children’s Hospital and
Baylor College of Medicine, Houston, TX, USA; fUCL Medical School, University College London, London, UK; gRCSI University of
Medicine and Health Sciences, Dublin, Ireland; hSchool of Medicine, Keele University, UK; iMedical School, University of Michigan, USA

ABSTRACT KEYWORDS
Background: The novel coronavirus disease (COVID-19) was declared a pandemic in March 2020. Best evidence medical
This rapid systematic review synthesised published reports of medical educational developments education; undergraduate;
in response to the pandemic, considering descriptions of interventions, evaluation data and lessons postgraduate; medi-
cine; methods
learned.
Methods: The authors systematically searched four online databases and hand searched
MedEdPublish up to 24 May 2020. Two authors independently screened titles, abstracts and full
texts, performed data extraction and assessed risk of bias for included articles. Discrepancies were
resolved by a third author. A descriptive synthesis and outcomes were reported.
Results: Forty-nine articles were included. The majority were from North America, Asia and
Europe. Sixteen studies described Kirkpatrick’s outcomes, with one study describing levels 1–3. A
few papers were of exceptional quality, though the risk of bias framework generally revealed capri-
cious reporting of underpinning theory, resources, setting, educational methods, and content. Key
developments were pivoting educational delivery from classroom-based learning to virtual spaces,
replacing clinical placement based learning with alternate approaches, and supporting direct
patient contact with mitigated risk. Training for treating patients with COVID-19, service reconfigur-
ation, assessment, well-being, faculty development, and admissions were all addressed, with the
latter categories receiving the least attention.
Conclusions: This review highlights several areas of educational response in the immediate after-
math of the COVID-19 pandemic and identifies a few articles of exceptional quality that can serve
as models for future developments and educational reporting. There was often a lack of practical
detail to support the educational community in enactment of novel interventions, as well as lim-
ited evaluation data. However, the range of options deployed offers much guidance for the med-
ical education community moving forward and there was an indication that outcome data and
greater detail will be reported in the future.

Background
Practice points
The novel coronavirus disease (COVID-19) is a highly conta- ! Remote synchronous and asynchronous educa-
gious viral illness caused by the severe acute respiratory tional developments were rapidly deployed and
syndrome coronavirus 2 (SARS-CoV-2). COVID-19 was first will likely persist beyond the pandemic. Learner
reported in Wuhan, Hubei Province, China in December engagement, structure and organization are key.
2019. Within weeks of its emergence, it had spread to sev- ! Maintaining clinical exposure is important for
eral countries. In January 2020, the World Health learners impacted by COVID-19 and can be
Organisation (WHO) declared the outbreak a Public Health achieved using telehealth, PPE, physical distancing.
Emergency of International Concern. By March 2020, ! Quality and detail of reporting educational devel-
COVID-19 had evolved into a pandemic (Bedford et al. opments must improve to promote replication in
2020). According to the dashboard of the Center for different contexts.
Systems Science and Engineering at Johns Hopkins
University (JHU 2020), Baltimore, USA, the disease has now
been reported in 188 countries, affecting over 15,000,000 physically distance and conserve personal protective equip-
people worldwide, resulting in over 600,000 deaths. ment (PPE) has resulted in the suspension of in-person
The impact of COVID-19 on healthcare systems and learning in classrooms, and even the workplace. The effects
medical education has been unprecedented. Huge num- of COVID-19 have been felt across the medical education
bers of campuses have gone into lockdown. The need to continuum, necessitating a myriad of changes.

CONTACT Morris Gordon [email protected] School of Medicine, University of Central Lancashire, Harrington Building HA340, Preston, UK
Supplemental data for this article can be accessed here.
! 2020 Informa UK Limited, trading as Taylor & Francis Group
 MEDICAL TEACHER 1203

The educational community has rapidly adjusted their 18 May 2020 to check the appropriateness of the search
approach to meet these challenges, and a number of edu- strategy. This led to the addition of further terms, as the
cational developments to support learning and educational search was producing too many potential papers (roughly
progress have been reported. 1 paper meeting the inclusion criteria for every 10 titles)
Journals have expedited peer review to ensure COVID suggesting the search was too narrow. The final search was
related innovations and adaptations reach educators in a performed on 24 May 2020 using the following terms: (cor-
timely manner. This has resulted in a large number of onavirus OR covid19 OR covid-19 OR SARS-Cov-2 OR 2019-
articles of varying quality being published in a very short nCoV) AND (Medical education OR undergraduate medical
timeframe. Busy educators trying to adapt their practices OR medical student OR medical school OR training OR con-
to the continually evolving pandemic need an up-to-date tinuing medical education OR postgraduate medical educa-
collated resource that discusses and evaluates these tion OR assessment OR teaching OR evaluation OR
developments. interview OR recruitment OR distance learning OR examina-
Dedeilia et al. (2020) previously conducted a systematic
tions OR OSCE OR PPE OR clinical skills). To identify add-
review of educational developments in response to COVID-
itional relevant articles, we conducted a hand search of
19. At the time of their review, they noted a ‘scarcity of
MedEdPublish. Due to the short timeframe between the
available sources’ and thus decided to include letters to
advent of COVID-19 and this review being performed, for-
the editor, commentaries, editorials and perspectives. Their
wards and backwards citation searching was not performed
search ended 18 April 2020 and there has been a signifi-
as this was not considered likely to identify any further
cant increase in the quantity and quality of articles since
that time. relevant papers.
The aim of the current systematic review is to identify
the evidence concerning teaching, assessment or other
educational developments in response to the COVID-19 Inclusion and exclusion criteria
pandemic within medical education. Our review will
address three main questions: The following inclusion criteria were used:

! What developments or changes in medical education ! Studies describing developments in medical education
have been deployed? (i.e. description or ‘what was explicitly deployed in response to COVID-19.
done’ (Cook et al. 2008)). ! Studies in undergraduate, graduate or continuing med-
! What is the impact of these developments or changes? ical education.
(i.e. evaluation or ‘did it work?’). ! Studies published after 1 December 2019, when COVID-
! What lessons to be applied in the future have been 19 was first identified.
learned by the teams who deployed these develop- ! Studies in any language.
ments or changes? (i.e. implications or ‘what’s next?’).
The following exclusion criteria were applied:

Methods ! Opinion pieces, commentaries, editorials, perspectives,

calls for change, needs assessments and other
This review was conducted as a ‘rapid’ review, meaning
studies where no actual development had
that the timeframe from inception to completion was <
4 weeks. It is vital to note that the speed with which the been deployed.
review was conducted in no way impacted the methodo- ! Studies that have Health Care Professionals but no med-
logical rigour of the approach. We embraced systematicity ical students, residents, fellows, or physicians.
throughout, from the search strategy to the synthesis ! Studies that describe the development as a minor part
(Gordon, Daniel, et al. 2019). We aligned with both positiv- of a larger package of planned measures.
ism (applying the principles of systematic reviewing) and
constructivism (utilizing qualitative synthesis methods). A The original protocol stipulated that Kirkpatrick’s out-
study protocol (Gordon et al. 2020) was completed a priori comes needed to be reported for inclusion (Kirkpatrick and
and uploaded into the study repository on the Best Kirkpatrick 2016). However, during the pilot phase, we
Evidence in Medical Education (BEME) website. We identified several interventions of interest that had been
reported our findings in alignment with the STORIES executed, but not evaluated due to insufficient time prior
(STructured apprOach to the Reporting In healthcare edu- to publication Consequently, we decided to amend the
cation of Evidence Synthesis) statement (Gordon and Gibbs protocol and include such studies.
2014) and BEME guidance (Hammick et al. 2010). The titles and abstracts of all papers identified
through the search were reviewed independently by two
authors against the above inclusion and exclusion criteria.
Search strategy Inter-rater reliability was calculated using Cohen’s Kappa.
We conducted an electronic search of four databases The full papers of all studies included after abstract
(MEDLINE, EMBASE, CINAHL and PsychINFO). We selected screening were retrieved and again reviewed against our
these 4 databases as they contain almost all the journals inclusion and exclusion criteria by two authors. Disputes
that publish on medical education and they are most com- at either stage were resolved through discussion, includ-
monly used in BEME reviews. We utilized 22 search terms ing a third author where necessary, until consensus
and their Boolean combinations. The search was piloted on was reached.
1204 M. GORDON ET AL.

Table 1. Quality assessment/risk of bias of the interventions presented.

Bias source High quality Unclear quality Low Quality
Underpinning bias (U) Clear and relevant description of Some limited discussion of No mention of underpinning
theoretical models or conceptual underpinning, with minimal
frameworks that underpin the interpretation in the context of
development the study
Resource bias (R) Clear description of the cost / time Some limited description No mention of resources
/ resources needed for the of resources
development
Setting bias (S) Clear details of the educational Some description, but not No details of learner characteristics
context and learner significant as to support or setting
characteristics of the study dissemination
Educational bias (E) Clear description of relevant Some educational methods No details of educational methods
educational methods employed mentioned but limited detail as
to support delivery to how applied
Content bias (C) Provision of detailed materials (or Some elements of materials No educational content presented
details of access) presented or summary
information

Data extraction Gordon, Farnan, et al. (2019) and Gordon et al. (2018), ori-
ginally modified from Reed et al. (2005)). The latter is critic-
Based on BEME Guidance (Hammick et al. 2010), we
ally important, because only when the development is
devised and piloted a data extraction form to be com-
robustly described, can educators or researchers hope to
pleted online within Google Sheets to allow synchronous replicate the results in other contexts.
review and sharing of extracted data. For the first element, if sufficient data on study design
Data extracted included: and outcomes were provided, we used the risk of bias tool
(i.e. Higgins criteria) for randomized-control trials (Sterne
! Paper identifiers (author(s), date)
et al. 2019) and the ROBINS-I tool (Risk Of Bias In Non-
! Context (geographic location, local COVID-19 specific
randomized Studies of Interventions) for non-randomised
details, education level, institutional setting, number trials (Sterne et al. 2016) in line with current Cochrane
of learners) handbook advice. If no such details were given, the quality
! Description of intervention (focus of development, pur- of the study design and outcomes were not assessed. For
pose of deployment, brief summary of development, the second element, we considered whether the authors
further description of development) explicitly reported on five key areas related to the educa-
! Intervention outcome (Kirkpatrick outcome, summary of tional development. A visual ranking system (Gordon and
results, plans for future study) Gibbs 2014) was used to report risk of bias for these five
! Risk of bias (underpinning bias, resource bias, setting areas (e.g. underpinning bias, resource bias, setting bias,
bias, content bias, development limitations) educational bias, and content bias). Items were judged to
! Other details (key points for discussion, lessons learnt, be of high quality and low risk of bias (green), unclear
summary of conclusions, appropriateness of conclusion, quality and risk (yellow) or high risk and low quality related
any other comments by extractor) to lack of reporting (red). This ranking system is shown in
Table 1.
Two studies were extracted by all authors independently Thresholds for judgements were discussed during pilot-
and a meeting was held to ensure appropriateness of the ing of the data extraction form. All judgements were made
extraction forms and shared understanding of terms to independently by two authors and disagreements were
enhance inter-rater reliability. Extraction was then com- resolved through discussion or involvement of a single
pleted by two authors independently and disputes were third author (MG). No weighting or overall rank is given, as
resolved by involving a third author (MG) and discussing no item is more important than another. Rather the judge-
until a full consensus was reached at regular research ment in each area is presented so readers can assess areas
team meetings. of stronger and weaker reporting.
Of note, for both elements, poor reporting does not
necessarily mean the educational development is of poor
Quality assessment
quality, but it increases the risk that such poor quality may
While many methods have been utilised to assess quality exist, hence the use of the terminology ‘risk of bias’ in
and judge risk of bias in medical education reviews, no reporting. Importantly, poor reporting limits utility for read-
consensus method exists (Buckley et al. 2009; CASP 2014; ers, as they will struggle to determine if the educational
Gordon et al. 2018). The review team postulated, in line development is transferable to their context.
with previous BEME reviews, that this is partly related to
the complexity of educational developments and therefore
Synthesis of evidence
requires an approach that can address and account for this
complexity. Thus, we considered two distinct quality ele- A descriptive synthesis of included studies was completed
ments: 1) the risk of bias or quality of the study design utilizing data from the extraction form to summarize ‘what
when outcomes were reported (similar to the Cochrane was done.’ This summary described the timing of publica-
tradition) and 2) the risk of bias or quality of reporting for tion, the setting (undergraduate, postgraduate, mixed), the
the educational development itself (as previously used by geographical location and COVID-19 specific contextual
 MEDICAL TEACHER 1205

factors, the type and number of participants, the focus of recommended implementation of physical distancing and
the educational developments and the purpose of the cancellation of all gatherings of more than 10 people
deployments. Outcomes (when available) were classified in (Almarzooq et al. 2020; Murdock et al. 2020). Face-to-face
accordance with Kirkpatrick’s model of evaluation to deter- didactic education was suspended first (Calhoun et al.
mine ‘did it work’ (Kirkpatrick and Kirkpatrick 2016). Quality 2020; Hannon et al. 2020) and then, the Association of
assessment for the five areas were reported. We planned American Medical Colleges recommended suspension of all
for meta-analysis; however, suitably homogenous outcome direct patient contact responsibilities for medical students
data was not found. We close with lessons learned (i.e. (Soled et al. 2020). Some hospitals were at capacity, requir-
‘what’s next’) as stated in the primary papers by ing redeployments of the workforce (Balanchivadze and
the authors. Donthireddy 2020) and cancellation of elective surgical pro-
cedures (Chick et al. 2020; Roy and Cecchini 2020). In Asia,
studies reported government enforced lockdowns and
Results restrictive measures, including the closure of medical cam-
The search was performed on 24th May 2020. A total of puses (Singh et al. 2020; Srinivasan 2020; Veasuvalingam
7448 titles were found, with a further 28 identified through and Goodson 2020). Studies in Singapore reported the
hand searching MedEdPublish. After deduplication, 6215 escalation of the national pandemic alert to Disease
remained. Through title and abstract screening, 6004 stud- Outbreak Response System Condition (DORSCON)-Orange
ies were excluded. A total of 213 studies were considered resulting in quarantining, temperature screenings and vis-
for full text screening and 164 were excluded. Inter-rater itor restrictions at hospitals (Boursicot et al. 2020;
reliability at the screening phase was j ¼ 0.933 (95% CI Kanneganti et al. 2020; Samarasekera et al. 2020). In
0.927–0.94), representing almost perfect alignment. The pri- Europe, countries implemented national restrictions on
mary reasons for exclusion were as follows: the article rep- non-essential activities, invoked lockdowns and moved all
resented an editorial or opinion piece without deployment educational activities online (Finn et al. 2020; Moszkowicz
of a change (90), the article described a theoretical devel- et al. 2020; Torres et al. 2020). Some governments (e.g.
opment or idea with no actual intervention (71), and the Italy and Denmark) responded to the pandemic by boost-
ing the workforce through expedited graduation or tem-
article was restricted to other health care professionals and
porary voluntary employment of medical students (Lapolla
did not include medics (3). Forty-nine studies were
and Mingoli 2020; Rasmussen et al. 2020). In Central
included in the final analysis. The flow diagram for included
America, South America and Africa, studies described the
studies is shown in Figure 1 (PRISMA. 2015).
suspension of face-to-face education and the move to
online teaching (Fernandez-Altuna et al. 2020; Gaber et al.
Publications 2020; Parisi et al. 2020).

The four earliest studies that were included in the review

were published in March, 22 studies were published in Level of medical education, institutional setting and
April, and 23 studies were published in May. Of the 49 number of participants involved
publications, 15 were published in a new virtual issue of Twenty-four studies (49%) described developments and
Medical Education entitled ‘Adaptations,’ designed to changes in undergraduate medical education (UME) pro-
rapidly share insights and innovations from health grams in response to the COVID-19 pandemic. Eighteen
professions educators in response to COVID-19 (Eva and studies (37%) reported developments in postgraduate edu-
Anderson 2020). cation programs, including within graduate medical educa-
tion (GME) or continuing medical education (CME). Six
studies (12%) described mixed learners. One study (2%) did
Classification of studies not describe the learner level. Twenty-five studies (51%)
Table 2 presents the number of studies in terms of geo- were conducted in universities, twenty-one studies (43%) in
graphical location, the level of medical education and the hospital settings, and three (6%) were unspecified or
institutional setting. multi-site.
Seven studies (Boodman et al. 2020; Fernandez-Altuna
et al. 2020; Keegan et al. 2020; Lapolla and Mingoli 2020;
Geographical location and local COVID-19 specific Murdock et al. 2020; Rose et al. 2020; Samarasekera et al.
details 2020) describe educational interventions with large partici-
Twenty-three studies (47%) were conducted in North pant groups of over 1000. Thirteen studies (Blake et al.
America, including fourteen studies in the United States, 2020; Boursicot et al. 2020; Calhoun et al. 2020; Choi
eight studies in Canada and one study in Mexico. A further et al. 2020; Cleland et al. 2020; Gaber et al. 2020; Haines
twelve studies (25%) were conducted in Asia, ten studies et al. 2020; Hannon et al. 2020; Kang et al. 2020;
(20%) in Europe, two studies (4%) in Africa, and one study Kanneganti et al. 2020; Rasmussen et al. 2020; Singh et al.
(2%) in South America. Only one study (2%) was 2020; Soled et al. 2020) describe interventions with partici-
international. pant groups between 26 and 1000 (min ¼ 32, max ¼ 906,
In Canada, local COVID-19 restrictions limited group median ¼ 108). Nine studies (Almarzooq et al. 2020; Burns
gatherings (Keegan et al. 2020) and learners were with- and Wenger 2020; Buonsenso et al. 2020; Balanchivadze
drawn from clinical placements (Boodman et al. 2020; and Donthireddy 2020; Christensen et al. 2020; Roy and
Haines et al. 2020; Johnston et al. 2020). In the United Cecchini 2020; Hofmann et al. 2020; Johnston et al. 2020;
States, the Centre for Disease Control and Prevention Srinivasan 2020) describe interventions with participant
1206 M. GORDON ET AL.

Records identified through database Additional records identified by hand

Identification
searching searching
(n = 7,448) (n = 28)

Records after duplicates removed

(n = 6,215)
Screening

Records screened Records excluded

(n = 6,215) (n = 6,002)
Eligibility

Full-text articles assessed Full-text articles

for eligibility excluded, with
(n = 213) reasons
(n = 164)
Included

Studies included in
synthesis
(n = 49)

Figure 1. PRISMA flow diagram for included studies.

Table 2. Origin characteristics of included studies.

Number of Studies
Level of medical education# Institutional Setting
Location U P M ND Hospital University Unspecified
United States 5 8 1 10 3 1
Canada 4 2 1 1 3 5 –
Central America – – 1 – – 1 –
South America 1 – – 1 –
Europe 5 3 2 5 4 1
Africa 1 1 – – 2 –
Asia 8 4 3 9 –
International – – 1 – – 1
Totals: 49 24 (49%) 18 (37%) 6 (12%) 1 (2%) 21 (43%) 25 (51%) 3 (6%)
#U: undergraduate; P: postgraduate (GME/CME); M: mixed; ND: not described

groups between 1 and 25 (min ¼ 1, max ¼ 25, median ¼ on 55 participants in the initial study, but also reported on
14). The remaining twenty studies did not specify the num- early dissemination metrics, noting that the intervention
ber of participants involved in the educational intervention was accessed by 17,633 users globally within one week of
described. The Blake et al. (2020) study not only reported the digital launch.
 MEDICAL TEACHER 1207

Educational outcomes its design, as well as outcomes at several levels of

Kirkpatrick’s hierarchy (Blake et al. 2020). Conversely, for
While not all studies used the term ‘Kirkpatrick’s outcomes,’
the six studies judged at high risk of bias, the quality was
sixteen studies reported them and one study described
pervasively poor, with missing information in all extracted
multiple levels. Thirteen studies described level 1 (i.e. reac-
areas and no details of any educational outcomes. What is
tion), four studies (8%) described level 2 (i.e. learning) and
not clear, is if these studies simply represent poor reporting
one study (2%) described level 3 (i.e. behavioural change).
or poor-quality educational research. Fortunately, these
Thirty-three studies (67%) did not report any Kirkpatrick’s
papers make up a minority (12%) of the studies included.
outcomes, however, seven of these articles explicitly stated
Full details of the quality assessments and other character-
a plan for future evaluation of educational effectiveness.
istics of included studies are described in Supplementary
One study described three levels of Kirkpatrick’s outcomes
Table and a visual representation of key results is pre-
(Blake et al. 2020). They utilized ‘agile methodology,’ a
sented in an infographic (Figure 2).
robust three-step process for rapid program development
and were able to conduct a comprehensive program evalu-
ation within three weeks of the outbreak in the United Summary of educational developments
Kingdom. Impressively, eighty-two percent of users (n ¼ 55)
in the pilot study reported applying the information (i.e. Forty-nine educational developments were described. Of
changing behaviour) in their work or home lives after these, 40 utilised online learning approaches in whole or in
engaging with the digital resource. part. Thirty-three papers described adaptations to existing
educational programmes, and sixteen described new edu-
cational offerings. The focus of the developments were
Quality assessment/risk of bias broadly categorised as follows:
Quality of the study design when outcomes were
reported ! Pivoting to online education delivery (53%).
There was one Randomised controlled trial (Christensen ! Training for treating patients with COVID-19 (16%).
et al. 2020). This was judged for methodological quality ! Clinical service reconfigurations to support response to
using the Cochrane Higgins criteria (Sterne et al. 2019). COVID-19 (12%).
Randomisation, allocation concealment, incomplete out- ! Assessment (12%).
come reporting, selective reporting and other sources were ! Faculty development (6%).
all of low risk of bias with high quality reporting. As this ! Learner support, mental health and wellbeing (4%).
was an open label trial, detection bias was high risk. There ! Selection and admissions (4%).
were no other trials reported. The majority of papers (67%)
did not offer interventional outcome data and those that A small proportion of papers (6%) addressed multiple of
did can best be described as ‘educational case studies’ these categories.
rather than other study designs. As such, ROBINS-I evalu-
ation was not undertaken to assess quality of these studies. Pivoting education delivery
Twenty-six papers described delivering existing educational
Quality of reporting for the educational development programmes through online platforms in response to local
The risk of bias framework for the reporting quality of the restrictions imposed including limitations on gatherings
developments was applied. There was only one study con- and physical distancing. Of these, thirteen were targeted at
sidered at low risk of bias and high quality in all five undergraduate medical students, ten at postgraduate med-
domains (Blake et al. 2020). There were a further six studies ical trainees, two at both undergraduate and postgraduate,
that whilst not having full reporting of detail, did report in and one did not describe their population.
all areas with varying amounts of detail (Brown et al. 2020; These papers have been sub-categorised into three
Buonsenso 2020; Choi et al. 2020; Johnston et al. 2020; groups based on their context and focus:
Murdock et al. 2020; Samarasekera et al. 2020). There were
six studies that did not report on any of the five domains A. Using video conferencing to deliver the same teaching
that were judged at high risk of bias and low quality for all approaches for non-clinical learning (e.g. seminars,
(Ahmed et al. 2020; Boodman et al. 2020; Burns and simulated sessions, team-based learning).
Wenger 2020; Haines et al. 2020; Keegan et al. 2020; B. Replacing clinical placement based learning with other
Lubarsky 2020). Within each study and within each domain, teaching methods online.
the distribution of reporting was capricious with no par- C. Supporting continued experiential learning/clinical
ticular area systematically reported in a different manner to contact without physical presence in clinical workpla-
others. Supplementary Table illustrates the individual rat- ces (e.g. supervised phone or video consultations).
ings for each area for all the studies and will support the
reader in considering which primary studies may offer A. Same teaching approaches, online. Fifteen papers
reporting for future replication of developments. described replacing face-to-face teaching in the classroom
Given the heterogeneity of reporting within the majority with online learning using similar educational approaches
of studies, it is hard to comment on any patterns or correl- (Agarwal et al. 2020; Almarzooq et al. 2020; Balanchivadze
ation. However, it is worth noting that the one paper and Donthireddy 2020; Durrani 2020; Fernandez-Altuna
judged at low risk of bias was well-reported and well- et al. 2020; Gaber et al. 2020; Khan 2020; Parisi et al. 2020;
designed in all areas, at all stages, with details provided on Rose et al. 2020; Singh et al. 2020; Srinivasan 2020; Sudhir
1208 M. GORDON ET AL.

Figure 2. Infographic summarizing key findings.

et al. 2020; Taylor et al. 2020; Torres et al. 2020; standardized patient or peer feedback. Three papers used a
Veasuvalingam and Goodson 2020). Twelve of these combination of synchronous and asynchronous teaching
employed synchronous learning on video conferencing approaches, although details of the balance were not
platforms. These included delivering seminars (Agarwal reported. The synchronous components of these were simi-
et al. 2020; Almarzooq et al. 2020; Balanchivadze and lar to those described above. The asynchronous compo-
Donthireddy 2020; Rose et al. 2020; Singh et al. 2020; nents involved making recordings of previous lectures
Srinivasan 2020), debates (Durrani 2020), team-based learn- available and making additional learning resources avail-
ing (Gaber et al. 2020), simulation sessions (Torres et al. able through curation or de novo creation (Taylor et al.
2020), and clinical skills sessions (Khan 2020; Parisi et al. 2020; Veasuvalingam and Goodson 2020). In the papers
2020; Sudhir et al. 2020). The authors of studies that uti- that used asynchronous approaches, emphasis was placed
lized synchronous learning formats often talked about the on the need for organization and structure to support
importance of learner engagement. In the studies that uti- learning in the virtual environment. No developments
lized a seminar or debate format, learner engagement was reported exclusively asynchronous learning, and the over-
promoted using online chat features, electronic ‘hand-rais- whelming emphasis was on synchronous remote learning.
ing’ for questions, and online polling. In the one paper that One paper described moving a whole curriculum online for
discussed team-based learning, engagement was promoted a university in Mexico of 8,000 students, 18,000 residents,
using breakout rooms to host groups of 25 students com- and 5,000 faculty (Fernandez-Altuna et al. 2020). They
pleting the team readiness assessment test. In the simula- adopted a new digital distance learning platform for online
tion and clinical skills sessions, engagement was facilitated delivery of virtual classrooms and academic consultancies
through skill building interactions, and instructor, and supporting work from home.
 MEDICAL TEACHER 1209

B. Replacing clinical placement based learning. Seven the logistics in order to persevere with face-to-face OSCEs
papers described replacing or supplementing clinical place- (Boursicot et al. 2020; Samarasekera et al. 2020). By using
ment based learning with other teaching approaches PPE, expanding the number of sites for testing, cohorting
(Burns and Wenger 2020; Calhoun et al. 2020; Chick et al. learners, and removing real patients from the assessments,
2020; Kanneganti et al. 2020; Lubarsky 2020; Moszkowicz the authors were able to successfully implement the
et al. 2020; Roy and Cecchini 2020). Authors noted that exams. The third delivered an online OSCE using Zoom,
while these interventions were important for continued replacing physical examination with a narration of what
learning they could not replace certain face-to-face activ- they would do (Hannon et al. 2020). The authors concluded
ities (e.g. time in the operating room (Chick et al. 2020). that remote OSCEs were not as effective as in-person for
Two papers described replacing clinical placements in sur- assessing clinical skills. Three papers described written
gery with a mix of online synchronous and asynchronous assessments (Samarasekera et al. 2020; Lapolla and Mingoli
teaching using a combination of videoconferencing, flipped 2020; Veasuvalingam and Goodson 2020). The first split the
classrooms with question and answer time, video review of candidates from one site to six smaller sites in order to
surgical procedures, and surgical simulators (Chick et al. enable in-person examinations with physical distancing.
2020; Kanneganti et al. 2020). One paper outlined video-
The second cancelled their national licensing exam in order
conferencing of anatomy content for surgery students,
to support early graduation of final year medical students
though the exact nature of the intervention (i.e. if there
(Lapolla and Mingoli 2020). The third transitioned to forma-
were dissections) was unclear (Moszkowicz et al. 2020).
tive on-line quizzes and short tests with feedback to
One paper described an entirely virtual clinical elective
enhance and promote remote learning (Veasuvalingam and
using a combination of synchronous seminars, small group
Goodson 2020). The other two papers described assess-
discussions, and role-plays (Burns and Wenger 2020).
ment item writing workshops that were both delivered
Four papers described replacing or supplementing clinical
online using Zoom instead of face-to-face (Ahmed et al.
placements with asynchronous learning opportunities.
These included practice questions (Chick et al. 2020), inde- 2020; Eltayar et al. 2020).
pendent projects (Lubarsky 2020), interpretation of
example slides for postgraduate pathology trainees (Roy
and Cecchini 2020), procedural videos (Kanneganti et al. Training for treating patients with COVID-19
2020), and videoconferencing and e-learning modules Eight papers described new educational interventions
(Kanneganti et al. 2020). One paper described a redesigned designed for doctors (including postgraduate trainees) that
undergraduate curriculum to accommodate for a shortened were treating patients with confirmed or suspected COVID-
academic year (assuming learners will be able to return to 19 (Boodman et al. 2020; Buonsenso et al. 2020; Choi et al.
clinical placements). They reduced the duration of all place- 2020; Christensen et al. 2020; Gardiner et al. 2020; Hanel
ments by a third and supplemented selected placements et al. 2020; Kang et al. 2020; Merali et al. 2020). These
with online virtual placements (Calhoun et al. 2020). papers varied in their focus: either on particular groups of
providers or on particular procedures. Four papers
C. Supporting continued clinical contact. Four papers described training in safe endotracheal intubation for
described supporting some form of continued clinical con- COVID-19 positive patients or persons under investigation
tact using approaches to mitigate risk for learners missing (Choi et al. 2020; Gardiner et al. 2020; Hanel et al. 2020;
out on in-person patient care opportunities (Chick et al. Kang et al. 2020). One paper described the use of ultravio-
2020; Hofmann et al. 2020; Johnston et al. 2020; Oldenburg let fluorescent powder during simulated intubation in order
and Marsch 2020). Activities included supervised telephone to demonstrate aerosol generation during this procedure
or video consultations for undergraduate medical students (Gardiner et al. 2020). One paper described a 10-week
(Johnston et al. 2020) or postgraduate trainees (Chick et al. online course in internal medicine for doctors redeployed
2020; Oldenburg and Marsch 2020), with feedback from from sub-speciality services (Merali et al. 2020). Another
the supervisor either offline or with the patient present, paper described training in lung ultrasound for obstetrics
and virtual ward rounds for undergraduate medical stu- and gynaecology consultants with existing ultrasound
dents using an iPad on wheels (Hofmann et al. 2020) to expertise to facilitate the care of pregnant patients with
see, hear and interact with COVID-19 patients and their COVID-19 (Buonsenso et al. 2020). Three papers described
physicians. Clearly, these studies are not workplace-based in situ simulation programmes to train doctors in new pro-
in the traditional sense, but they do use authentic patient tocols for intubation in the emergency department (Hanel
interactions separate from other forms of learning. et al. 2020), in obstetric emergencies (Kang et al. 2020),
and in the intensive care unit (Choi et al. 2020). One paper
Assessment described an approach to training medical students and
Seven papers described adaptations to assessment proc- junior doctors in donning and doffing personal protective
esses (Ahmed et al. 2020; Boursicot et al. 2020; Eltayar equipment (PPE) (Christensen et al. 2020). These authors
et al. 2020; Hannon et al. 2020; Lapolla and Mingoli 2020; conducted a randomised control trial comparing in-person
Samarasekera et al. 2020; Veasuvalingam and Goodson instructor led training with remote video-based instruction.
2020). Three of these described adaptations to assessing Finally, one paper described the development of a newslet-
clinical skills through objective structured clinical examina- ter to disseminate evidence-based responses to clinical
tions (OSCEs) in the context of physical distancing. All three questions raised by doctors treating COVID-19 patients
were for undergraduate medical students. Two redesigned (Boodman et al. 2020).
1210 M. GORDON ET AL.

Clinical service reconfiguration Finn et al. 2020; Khan 2020; Rose et al. 2020), ‘positive’ or
Six papers described retraining or redeploying learners to ‘valuable’ or ‘useful’ in 4 studies (Choi et al. 2020; Gaber
support the response to increased clinical service pressures. et al. 2020; Lubarsky 2020; Taylor et al. 2020), ‘successful’
These included the accelerated graduation of medical stu- or ‘sufficient’ or ‘equivalent’ in 7 studies (Buonsenso et al.
dents (Lapolla and Mingoli 2020), redeployment of post- 2020; Burns and Wenger 2020; Christensen et al. 2020;
graduate clinical trainees (from haematology and oncology Hanel et al. 2020; Rasmussen et al. 2020; Torres et al. 2020;
to general medicine) to support care of COVID-19 patients Ungtrakul et al. 2020). No study was reported by the
(Balanchivadze and Donthireddy 2020), and reconfiguration authors as wholly unsuccessful or unfeasible, however,
of routine speciality care in order to avoid trainee viral some developments were noted to be less desirable than
exposure (Agarwal et al. 2020). Three papers described in-person activities, most notably among activities replac-
using medical students to support clinical care, including ing clinical placements (Chick et al. 2020). In two studies
launching medical student response teams to support the authors reported that students preferred the teaching
physicians and public health agencies (Haines et al. 2020; and assessment method pre-COVID, namely in an online
Soled et al. 2020) and training medical students to work as instruction using Google Classroom with a mix of lectures,
ventilator or nursing assistants (Rasmussen et al. 2020). practical demonstrations and case discussions (Singh et al.
2020) and an online OSCE (Hannon et al. 2020).
Faculty development Positive aspects of remote learning highlighted by
Three papers described faculty development programmes authors included enhanced effectiveness, flexibility, effi-
(Cleland et al. 2020; Finn et al. 2020; Keegan et al. 2020). ciency, engagement, communication and community
Two focused on supporting medical educators involved in (Almarzooq et al. 2020; Blake et al. 2020; Durrani 2020;
adapting programmes in response to COVID-19. These Keegan et al. 2020; Rose et al. 2020). Videoconferencing
included the curation of a set of resources (Keegan et al. tools were generally noted to be easy for facilitators and
2020) and the delivery of an online webinar aimed at shar- students to use in a personalized and intuitive manner due
ing best practice (Cleland et al. 2020). One paper described to their user-friendly interfaces (Sudhir et al. 2020), how-
the development of a twitter community of practice for ever, some encountered challenges with novel technolo-
medical education researchers (Finn et al. 2020). gies and struggled with issues related to WiFi access and
bandwidth (Chick et al. 2020). A few papers did discuss
Learner support, mental health and wellbeing problems and challenges that could prove helpful to
Two papers described interventions targeted at supporting groups attempting to build on these experiences: faculty
learners’ wellbeing (Blake et al. 2020; Brown et al. 2020). and learners need to be oriented to video-conferencing
The first used Barnet et al. (2014) seven-step framework to platforms (e.g. mute microphones in large group but not
implement an online community for doctoral students in small group meetings, utilize the chat or hand raising func-
medical education in order to mitigate against social isola- tion to speak or participate); restructuring is time intensive
tion (Brown et al. 2020). The second described the develop- and requires communication, teamwork and the collective
ment of a digital package to support health professions support of all members of the staff (Veasuvalingam and
workers’ and students’ mental health and wellbeing (Blake Goodson 2020); not all simulations can be replaced virtually
et al. 2020). or online, so pre-briefing and preparation are critical to
success (Sudhir et al. 2020); remote platforms may support
Selection and admissions technical skill development, but they may not support non-
verbal communication or physical exam skill development
Two papers described revised admissions procedures for (Eltayar et al. 2020; Hannon et al. 2020).
medical school (Ungtrakul et al. 2020; Samarasekera et al. Many study authors noted that these activities were
2020). The first describes replacing face-to-face multiple developed, analysed, and published within a very short
mini interview (MMI) with an online version using a video period and emphasized the potential of setting the stage
conferencing platform that required omission of their team- for subsequent investigation and studies as time allowed.
work scenario (Ungtrakul et al. 2020). The second changed They noted that many of these developments (e.g.
the content of their admissions interviews and held them increased online learning, precepting clinical care via tele-
via Zoom instead (Samarasekera et al. 2020). They also health) were likely here to stay. Seven studies highlighted
adjusted their Focused Skills Assessment (which assesses the sustainability of interventions beyond the pandemic
non-cognitive skills) from 5 stations to 2, eliminating the (Boodman et al. 2020; Kanneganti et al. 2020; Keegan et al.
teamwork scenario and focusing instead on a portfolio sta-
2020; Oldenburg and Marsch 2020; Srinivasan 2020;
tion and a new scenario-based station similar to a
Ungtrakul et al. 2020; Veasuvalingam and Goodson 2020),
Situational Judgement Test.
with the last study stating that ‘the shift online is trans-
formational’ and ‘though not all will be different, this turn-
Conclusions of study authors ing point has increased faith in technology sparking a
This section is a summary of the lessons learned and con- change in behaviour away from traditional approaches.’
clusions by the primary study authors, rather than the
review authors views. Most authors described the intro- Discussion
duced changes in positive terms, using statements such as
Summary of results
‘overwhelmingly positive,’ ‘very positive,’ ‘high quality,’
‘highly satisfied’ in 7 studies (Ahmed et al. 2020; The forty-nine included papers describe a variety of ways
Almarzooq et al. 2020; Blake et al. 2020; Eltayar et al. 2020; to pivot education to virtual spaces which was previously
 MEDICAL TEACHER 1211

classroom or patient-based. Whilst these developments (low-stakes) and summative (high stakes) assessment in un-
were forced into fruition by the COVID-19 pandemic, the proctored or remote proctored contexts are urgently
likelihood is that many will persist for the foreseeable needed. This is particularly critical in the United States and
future. In this first wave of papers, several developments other places where national bodies (e.g. the National Board
were described that support online learning across the of Medical Examiners) have implemented significant assess-
continuum with important implications for practice: ment changes (e.g. suspension of the United States
Educators using video conferencing to deliver instruction Medical Licensing Examination Step 2 Clinical Skills exam;
synchronously should attend to learner engagement (akin move to remote proctored, summative clinical subject
to active learning strategies in the classroom). As noted by exams at the end of clerkships.)
Ahmed et al. (2020), promoting engagement requires both
raising awareness of the importance of engagement and
Quality and completeness of the evidence base
filling educator’s toolboxes with adaptations to existing
teaching strategies ‘rephrased in light of the virtual plat- Despite the hurdles that included the very short time since
form.’ Educators using remote platforms for asynchronous the advent of COVID-19, a few papers were very well done
instruction need to create organization and structure to and represented excellent scholarship, with high quality
support learning. Short-term supplementation of clinical reporting of developments, impressive evaluation of impact
placement-based learning is clearly feasible, as is continued or in one case, both (Blake et al. 2020). Blake et al. (2020)
experiential learning without physical presence, such as developed a digital learning package with the purpose of
engagement of learners in telehealth. Means of maintaining mitigating the impacts of COVID-19 on mental health by
meaningful clinical contact are to date underexplored, par- protecting and promoting the psychological wellbeing of
ticularly amongst undergraduates. healthcare workers during and after the outbreak. The
This review revealed a fundamental paradox. Whereas digital package was notable for its usability, practicality,
service and workplace-based learning have previously been and effectiveness at meeting providers well-being needs,
closely integrated, these have now become more discreet, while being delivered at an acceptable cost. The authors
and the purpose and associated risks more explicit for followed a rigorous three-step iterative design process in
each. Service delivery itself has been transformed by the developing the package that can serve as a model for rapid
COVID-19 pandemic. While much of patient care remains in development and deployment of an educational interven-
person, a significant portion has shifting to a virtual envir- tion. Another paper (Christensen et al. 2020) conducted a
onment. In order to enable future sustainability of service, randomized control trial of PPE donning and doffing com-
we need to enable on-going patient-based training for paring live instructor-led training with video-based instruc-
learners with an appropriate balance of telehealth and in- tion. The results led to the conclusion of equivocal
person activities. A few studies in this review focused on educational effectiveness, with the implication that PPE
the incorporation of trainees into telehealth appointments training can be safely conducted virtually, a critically
(Chick et al. 2020; Johnston et al. 2020; Oldenburg and important finding for training and safety of the healthcare
Marsch 2020), yet more studies of this type are urgently workforce. When evaluated using Kirkpatrick’s outcomes
needed given the rather seismic shift in clinical care. Most scale, these two studies reached Levels 1–3 (Blake et al.
undergraduate papers focused on removing medical stu- 2020) and Level 2 (Christensen et al. 2020) and were con-
dents from the clinical context to minimise risk. This cannot sidered to have no or relatively low risk of bias.
be a long-term strategy. Three papers described medical The majority of papers, however, focussed on sharing
student contributions to service delivery (Haines et al. experiences, rather than robust evaluation or research
2020; Rasmussen et al. 2020; Soled et al. 2020). A few post- enquiry. As with all educational research, it is hard to
graduate papers highlighted ways in which physical (face- decide whether this reflects primary educational and
to-face) patient contact could be maintained while mitigat- research weaknesses or reporting issues. Such research
ing risk using PPE and physical distancing (Choi et al. 2020; weaknesses could be understandable given the rapid
Hanel et al. 2020; Kang et al. 2020). Future undergraduate developments when it comes to outcome evaluation but
developments might draw on lessons learned from these are harder to justify when considering the reporting of
studies to ensure that medical students can continue to developments. Any high-quality development should
engage in safe, in-person clinical learning. clearly define the underpinning theoretical frameworks,
Based on this review, it appears that assessment devel- articulate the resources needed for the development,
opments and adjustments were quite different across define the setting, describe the educational methods, and
undergraduate and postgraduate sectors, likely reflecting the content of the development to promote replicability
the discreet progression of undergraduates prior to licens- across different contexts. It is therefore disappointing and
ing and independent clinical practice. Undergraduate pro- highlights a clear gap in the evidence base, that many did
grammes have had to rapidly adapt their assessment not present this.
processes, or progress students without summative assess- This observed educational quality has implications for
ment (Lapolla and Mingoli 2020) in order to license new the continuation and extension of these developments,
graduates. Postgraduate assessment has tended to be post- which may well persist beyond the end of the pandemic as
poned and/or regulations adjusted to reflect COVID-related independent or as hybrid innovations (i.e. integrated with
delays. Since in-person (e.g. physically present) assessments traditional educational experiences). The rapid nature of
may not be able to resume soon, further studies that the developments likely contributed to the relative absence
address assessment, particularly those further exploring of significant conclusions/discussion about long-term
remote OSCE examinations would and formative effects and again represents a current gap in the evidence
1212 M. GORDON ET AL.

base for educators and other stakeholders. Clearly, as evi- This likely reflects the increased willingness of groups to
denced by Blake et al. (2020) and Christensen et al. (2020), report and editors to publish successful (vs. unsuccessful)
both quality scholarship and reporting thereof is possible, developments. We strongly recommend more balanced
and authors should look to their work as models for reporting and publication, as there is much to learn
future work. from failures.
The risk of bias related reporting the development
details is very telling within this review. This does not in
Comparison with existing literature any way disadvantage papers for not presenting outcomes,
This is a new and rapidly evolving situation that has but rather is guided by the principle that when reporting a
resulted in very rapid deployment of educational develop- development in education, sufficient detail must be given
ments. Much of the literature (per our criteria) is reflected to allow readers to judge the quality of an intervention
in this review. One previous systematic review has been themselves, compare with other developments and pos-
published on medical education developments during sibly replicate. Reporting was lacking in all key areas, with
COVID-19 (Dedeilia et al. 2020). That review was performed the majority of studies in all categories rated as high risk,
on articles published before 18th April 2020. Due to differ- meaning no material of any form was given to judge these
ing methodologies and the rapid expansion of the evi- key areas. Whilst some studies were capricious, providing
dence base, only three of the included articles in our details in some key areas that can still offer value to read-
review were included in their review (i.e. Chick et al. 2020; ers, it is limiting to this rapidly evolving field to not have
Moszkowicz et al. 2020; Soled et al. 2020). Of note, we spe- details of underpinning theory, resources needed, content
cifically excluded letters to the editor, commentaries, edito- used, the settings for deployment or teaching methods
rials, and perspectives, which comprised the bulk of their employed. Robust reporting does not confer any added
review. They concluded that their review ‘summarized the cost to the authors or ethical considerations and can add
available literature on the issue, which mostly consist(ed) much for educators and researchers trying to advance the
of anecdotal communications without empirical evidence, field. The barriers to including such content are not clear,
due to the short time window and unexpectedness of the and this limits the strength of the evidence overall.
COVID-19 pandemic.’ Clearly the evidence base has some-
what improved since their review, and there are examples
Recommendations for future research and practice
of quality scholarship (e.g. Blake et al. 2020).
This review provides some helpful direction for future pub-
lications. Based on this review, we have identified ample
Strengths and limitations
description of shifts to on-line platforms to deliver existing
The strengths of this rapid review include an ‘a priori content (e.g. using on-line seminar instead of classroom
protocol,’ reporting using a STORIES approach (Gordon and delivery). There is, however, less detailed literature around
Gibbs 2014), a comprehensive search strategy developed supporting traditional and new clinical workspace-based
through piloting, risk of bias assessment including an easy learning, particular for undergraduate learners. We argue
visual tool for representation, and timeliness of the review that this is where a focus for future research should lie.
to inform other educators in the pandemic. We aimed to This review has synthesised postgraduate and undergradu-
ensure rigor was not sacrificed by the rapidness of the ate literature and there may be some helpful insights to
review, yet there were limitations. Our selection of 4 elec- inform undergraduate patient-based learning in the future.
tronic databases was less than other reviews may select, There are some obvious gaps identified in this review.
but in line with other reviews within BEME. Future reviews Gaps in assessment were noted above. Admission and
may include a wider selection. Whilst we hand searched selection to medical school are not yet well explored, and
MedEdPublish, we did not hand search all non-indexed studies on selection into postgraduate training are entirely
medical education journals. Our study selection and extrac- lacking. Further research is urgently needed to examine
tion was all done in duplicate but by multiple author pairs these important fields, particularly in relation to retaining
to allow a rapid turnaround. This reduced the scope for equity and diversity principles in a virtual environment.
measures of inter-rater reliability and potentially increased Similarly, despite literature describing a range of innovative
the risk of inconsistent judgements during data extraction. ways to deliver teaching, there is relatively little existing lit-
Future reviews must consider this issue. Finally, we refined erature focusing on faculty development or support. The
our inclusion and exclusion criteria to ensure the practical- identified literature did not make visible any fundamental
ity and feasibility of a rapid review, focusing on studies opportunities or theories for change within medical educa-
describing developments that had already been deployed, tion. This review focuses on a relatively short time frame of
as well as on studies involving medics (i.e. physicians or publication and future publications may explore in more
physicians in training). Important innovations may have detail potential opportunities for change and innovation
been missed in opinion pieces or editorials. Literature produced by this global crisis.
focused on other health professions certainly warrants its There are also some more generic and methodological
own review in the future. As we are still early in the pan- points to be made regarding the evidence base within this
demic, the literature base is rapidly evolving. By the time review. Our review has sought to gather useful data on
this article is published, several additional reviews will likely developments that could guide future educators, yet in this
already be warranted. area, many papers were lacking. We would invite authors,
Concerning the literature base, we noted a tendency of peer reviewers and editors to consider the importance of
groups to largely report successful developments. such reporting in future studies to answer vital and simple
 MEDICAL TEACHER 1213

questions–‘what?,’ ‘so what?,’ ‘now what?’ This can support

Mohan Pammi, MD, PhD, Co-Director of a BEME International
dissemination and replication, and further research, build- Collaborating Centre and Associate Professor of Neonatology at Texas
ing on methods and ensuring iterative evolution within Children’s Hospital, Houston, Texas, USA.
the field.
Satid Thammasitboon, MD, MHPE is Director of the Center for
Research, Innovation and Scholarship in Medical Education (CRIS), Co-
Director of a BEME International Collaborating Centre, and Associate
Conclusions Professor of Pediatric Critical Care Medicine at Texas Children’s
Hospital, Houston, Texas, USA.
This review highlights a number of areas of change in the
Sophie Park is a General Practitioner, an NIHR School of Primary Care
immediate aftermath of the educational response to the
(SPCR) Evidence Synthesis Working Group (ESWG) Lead, Director of a
COVID-19 pandemic. A rapid shift to synchronous and BEME International Collaborating Centre, and Director of Medical
asynchronous remote learning occurred that will likely per- Education (Primary Care and Community) at UCL Medical School,
sist beyond the pandemic, and attention must be paid to University College London, UK.
learner engagement, structure and organization in the Teresa Pawlikowska is a General Practitioner, Director of a BEME
future. Early developments supported alternatives to clin- International Collaborating Centre, the Inaugural Director of the Health
ical placements or continued clinical exposure using tele- Professions Education Centre (HPEC), RCSI University of Medicine and
health, PPE, and physical distancing. A few articles of Health Sciences, Dublin, Ireland.
exceptional quality, most notably a digital learning package Eliot L Rees is a Lecturer in Medical Education at Keele University, a
to support well-bring (Blake et al. 2020) were identified PhD candidate at University College London, and Chair of Trainees in
that can serve as models to guide future educational devel- the Association for the Study of Medical Education (TASME),
Newcastle, UK.
opments and reporting. Gaps in the literature were identi-
fied with additional studies needed in the areas of Andrea Jane Doyle is a medical physicist and Research Officer for the
assessment, admissions and selection to post-graduate Health Professions Education Centre (HPEC), RCSI University of
Medicine and Health Sciences, Dublin, Ireland.
training, and faculty development. While there was often a
lack of practical detail to support the educational commu- Michelle Daniel, MD, MHPE is Chair of the BEME Review Committee,
nity in enactment of novel interventions and limited evalu- Associate Editor for Systematic Reviews for Medical Teacher, Assistant
Dean for Curriculum and Associate Professor of Emergency Medicine
ation data, the range of options deployed offers much
and Learning Health Sciences at the University of Michigan Medical
guidance for the medical education community. There School in Ann Arbor, Michigan, USA.
were indications that outcome data and additional details
will be reported and therefore an update review may be
warranted in the near future.
ORCID
Eliot L. Rees http://orcid.org/0000-0002-6458-5808

Acknowledgements
References
The authors would like to acknowledge the patients who have suf-
Agarwal S, Sabadia S, Abou-Fayssal N, Kurzweil A, Balcer LJ, Galetta SL.
fered from or lost their lives to COVID-19, our colleagues who have
2020. Training in neurology: flexibility and adaptability of a neur-
bravely battled this pandemic at great personal cost, and our learners
ology training program at the epicenter of COVID-19. Neurology.
for their resilience and adaptability in response to enormous changes
94(24):e2608–e2614.
in medical education.
Ahmed S, Shehata M, Hassanien M. 2020. Emerging faculty needs
for enhancing student engagement on a virtual platform.
MedEdPublish. 9(1):49. https://www.mededpublish.org/manuscripts/
Disclosure statement 2965.
Almarzooq Z, Lopes M, Kochar A. 2020. Virtual learning during the
No potential conflict of interest was reported by the author(s). COVID-19 pandemic: a disruptive technology in graduate medical
education. J Am Coll Cardiol. 75(20):2635–2638.
Balanchivadze N, Donthireddy V. 2020. Hematology/oncology fellowship
Notes on contributors emergency restructuring in response to the COVID-19 pandemic—
Morris Gordon, MBChB, PHD, MMed is Cochrane Coordinating Editor, Henry Ford Hospital, Michigan. JCO Oncol Pract. DOI:10.1200/OP.20.
Chair of the BEME Executive Committee, and a Professor of Evidence 00261. https://ascopubs.org/doi/full/10.1200/OP.20.00261.
Synthesis and Systematic Review, University of Central Lancashire, Barnett S, Jones SC, Caton T, Iverson D, Bennett S, Robinson L. 2014.
Preston, UK. Implementing a virtual community of practice for family physician
training: a mixed-methods case study. J Med Internet Res. 16(3):e83.
Madalena Patricio, PhD is Past-President of the Association for Bedford J, Enria D, Giesecke J, Heymann DL, Ihekweazu C, Kobinger G,
Medical Education in Europe (AMEE), Chair of BEME, and Director of Lane HC, Memish Z, Oh MD, Sall AA, et al. 2020. WHO strategic and
the Department of Medical Education at the Faculty of Medicine, technical advisory group for infectious hazards. COVID-19: towards
University of Lisbon, Portugal. controlling of a pandemic. Lancet. 395(10229):1015–1018.
Blake H, Bermingham F, Johnson G, Tabner A. 2020. Mitigating the
Laura Horne is a Clinical Fellow in Medical Education, Blackpool psychological impact of COVID-19 on healthcare workers: a digital
Victoria Hospital, Blackpool, UK. learning package. Int J Environ Res Public Health. 17(9):2997.
Boodman C, Lee S, Bullard J. 2020. Idle medical students review
Alexandra Muston is a Clinical Fellow in Medical Education, Blackpool
Victoria Hospital, Blackpool, UK. emerging COVID-19 research. Med Educ Online. 25(1):1770562.
Boursicot K, Kemp S, Ong TH, Wijaya L, Goh SH, Freeman K, Curran I.
Sebastian R Alston is Director of a BEME International Collaborating 2020. Conducting a high-stakes OSCE in a COVID-19 environment.
Centre and a Professor of Pathology at the Alabama College of MedEdPublish. 9(1):2939. https://www.mededpublish.org/manuscripts/
Osteopathic Medicine, Dothan, Alabama, USA. 2939.
1214 M. GORDON ET AL.

Brown ME, Archer RL, Finn GM. 2020. A virtual postgraduate commu- Educ. 54(7):666–667. https://onlinelibrary.wiley.com/doi/full/10.1111/
nity of practice. Med Educ. DOI:10.1111/medu.14214. https://online- medu.14198.
library.wiley.com/doi/full/10.1111/medu.14214. Gardiner C, Veall J, Lockhart S. 2020. The use of UV fluorescent powder
Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, Morley for COVID-19 airway management simulation training. Anaesthesia.
D, Pollard D, Ashcroft T, Popovic C, et al. 2009. The educational 75(7):964–965. [accessed 2020 June 25]. https://onlinelibrary.wiley.
effects of portfolios on undergraduate student learning: a Best com/doi/full/10.1111/anae.15089.
Evidence Medical Education (BEME) systematic review. BEME guide Gordon M, Daniel M, Patricio M. 2019. What do we mean by
no. 11. Med Teach. 31(4):282–298. ‘systematic’ in health education systematic reviews and why it mat-
Buonsenso D, Moro F, Inchingolo R, Smargiassi A, Demi L, Soldati G, ters!!. Med Teach. 41(8):956–957.
Moroni R, Lanzone A, Scambia G, Testa AC. 2020. Effectiveness of a Gordon M, Farnan J, Grafton-Clarke C, Ahmed R, Gurbutt D, McLachlan
‘fast lung ultrasound teaching program’ for gynecologists/obstetri- J, Daniel M. 2019. Non-technical skills assessments in undergraduate
cians dealing with pregnant women with suspicion of COVID-19 medical education: a focused BEME systematic review: BEME Guide
infection. Ultrasound Obst Gyn. 56(1):110–111. https://obgyn.online- No. 54. Med Teach. 41(7):732–745.
library.wiley.com/doi/abs/10.1002/uog.22066. Gordon M, Gibbs T. 2014. STORIES statement: publication standards
Burns R, Wenger J. 2020. A remotely conducted paediatric bootcamp for healthcare education evidence synthesis. BMC Med. 12(1):143.
for fourth-year medical students. Med Educ. 54(7):668–669. Gordon M, Hill E, Stojan J, Daniel M. 2018. Educational interventions to
Calhoun KE, Yale LA, Whipple ME, Allen SM, Wood DE, Tatum RP. Improve Handover in Health Care: an updated systematic review.
2020. The impact of COVID-19 on medical student surgical educa- Acad Med. 93(8):1234–1244.
tion: implementing extreme pandemic response measures in a Gordon M, Patricio M, Horne L, Muston A, Alston S, Pammi M,
widely distributed surgical clerkship experience. Am J Surg. 220(1): Thammasitboon S, Park S, Pawlikowska T, Rees E, et al. 2020.
44–47. https://www.americanjournalofsurgery.com/article/S0002- Protocol: rapid review of developments in medical education in
9610(20).30229-4/pdf. response to the COVID-19 pandemic: a BEME systematic review.
Chick RC, Clifton GT, Peace KM, Propper BW, Hale DF, Alseidi AA, Dundee, UK; [accessed 2020 June 25]. https://bemecollaboration.
Vreeland TJ. 2020. Using technology to maintain the education of org/Reviews+In+Progress/
residents during the COVID-19 pandemic. J Surg Educ. 77(4): Rapid+review+of+developments+in+medical+education+in+respon-
729–732. https://www.sciencedirect.com/science/article/pii/S193172 se+to+the+COVID+19+pandemic/.
0420300842. Haines MJ, Yu A CM, Ching G, Kestler M. 2020. Integrating COVID-19
Choi GYS, Wan WTP, Chan AKM, Tong SK, Poon ST, Joynt GM. 2020. volunteer response into the year 3 MD curriculum. Med Educ. DOI:
Preparedness for COVID-19: in situ simulation to enhance infection 10.1111/medu.14254. https://onlinelibrary.wiley.com/doi/abs/10.
control systems in the intensive care unit. Br J Anaesth. 125(2):
1111/medu.14254.
e236–e239. https://bjanaesthesia.org/article/S0007-0912(20).30202-6/
Hammick M, Dornan T, Steinert Y. 2010. Conducting a best evidence
pdf.
systematic review. Part 1: from idea to data coding. BEME Guide
Christensen L, Rasmussen CS, Benfield T, Franc JM. 2020. A randomized
No. 13. Med Teach. 32(1):3–15.
trial of instructor-led training versus video lesson in training health
Hanel E, Bilic M, Hassall K, Hastings M, Jazuli F, Ha M, Trotter B, Fraser
care providers in proper donning and doffing of personal protective
C, Rutledge G. 2020. Virtual application of in situ simulation during
equipment. Disaster Med Public. DOI:10.1017/dmp.2020.56. https://
a pandemic. CJEM. 1–4. [accessed 2020 June 25]. https://www.cam-
www.cambridge.org/core/journals/disaster-medicine-and-public-health-
bridge.org/core/journals/canadian-journal-of-emergency-medicine/
preparedness/article/randomized-trial-of-instructorled-training-versus-
article/virtual-application-of-in-situ-simulation-during-a-pandemic/
video-lesson-in-training-health-care-providers-in-proper-donning-and-
668BA5EEAAF128CC100BDD5A88DD77D8.
doffing-of-personal-protective-equipment/CF08F4727DA9D536883EC
Hannon P, Lappe K, Griffin C, Roussel D, Colbert-Getz J. 2020. An
BFD04BC2570.
objective structured clinical examination: from examination room to
Cleland J, McKimm J, Fuller R, Taylor D, Janczukowicz J, Gibbs T. 2020.
Zoom breakout room. Med Educ. DOI:10.1111/medu.14241. https://
Adapting to the impact of COVID-19: sharing stories, sharing prac-
onlinelibrary.wiley.com/doi/full/10.1111/medu.14241.
tice. Med Teach. 42(7):1–4. https://www.tandfonline.com/doi/full/
Hofmann H, Harding C, Youm J, Wiechmann W. 2020. Virtual bedside
10.1080/0142159X.2020.1757635.
Cook DA, Bordage G, Schmidt HG. 2008. Description, justification and teaching rounds with patients with COVID-19. Med Educ. https://
clarification: a framework for classifying the purposes of research in onlinelibrary.wiley.com/doi/abs/10.1111/medu.14223.
medical education. Med Educ. 42(2):128–133. Johns Hopkins University (JHU). 2020. COVID-19 dashboard by the
Critical Appraisal Skills Programme (CASP). 2014. CASP Qualitative Center for Systems Science and Engineering (CSSE) at Johns
Checklist; [accessed 2020 July 17]. https://casp-uk.net/wp-content/ Hopkins University. Baltimore, Maryland; [accessed 2020 June 25].
uploads/2018/01/CASP-Systematic-Review-Checklist_2018.pdf. https://coronavirus.jhu.edu/map.html.
Dedeilia A, Sotiropoulos MG, Hanrahan JG, Janga D, Dedeilia P, Sideris Johnston A, Barrick K, Jivraj F, Ram R. 2020. The Virtual Check-In’: a
M. 2020. Medical and surgical education challenges and innovations tool to facilitate virtual patient interaction for early clinical learners
in the COVID-19 era: a systematic review. In vivo. 34(3 Suppl): in a longitudinal integrated clerkship. MedEdPublish. 9(1):3125.
1603–1611. https://www.mededpublish.org/manuscripts/3125.
Durrani M. 2020. Debate style lecturing to engage and enrich resident Kang Y, Deng L, Zhang D, Wang Y, Wang G, Mei L, Zhou G, Shu H.
education virtually. Med Educ. DOI:10.1111/medu.14217. https:// 2020. A practice of anesthesia scenario design for emergency cesar-
onlinelibrary.wiley.com/doi/full/10.1111/medu.14217. ean section in patients with COVID-19 infection based on the role
Eltayar AN, Eldesoky NI, Khalifa H, Rashed S. 2020. Online faculty of standard patient. BST. 14(3):222–226. https://www.jstage.jst.go.
development using cognitive apprenticeship in response to COVID jp/article/bst/advpub/0/advpub_2020.03066/_article/-char/ja/.
19. Med Educ. 54(7):665–666. https://onlinelibrary.wiley.com/doi/ Kanneganti A, Sia CH, Ashokka B, Ooi SBS. 2020. Continuing medical
full/10.1111/medu.14190. education during a pandemic: an academic institution’s experience.
Eva KW, Anderson MB. 2020. Medical education adaptations: really Postgrad Med J. 96(1137):384–386. https://pmj.bmj.com/content/
good stuff for educational transition during a pandemic. Med Educ. early/2020/05/28/postgradmedj-2020-137840.
54(6):494. Keegan DA, Chan M, Chan T. 2020. Helping medical educators world-
Fernandez-Altuna M, Gutierrez Rayon D, Ramirez Resendiz M, Cruz wide pivot their curricula online: PivotMedEd.com. Med Educ. 54(8):
Mendez P, Tovar Lopez KA. 2020. Experience of the biggest med 766–767. https://onlinelibrary.wiley.com/doi/abs/10.1111/medu.
school in Mexico during the COVID-19 pandemic. MedEdPublish. 14220.
9(1):3088. https://www.mededpublish.org/manuscripts/3088. Khan H. 2020. An adaptation of Peyton’s 4-stage approach to deliver
Finn GM, Brown MEL, Laughey W, Due~ nas A. 2020. #pandemicpeda- clinical skills teaching remotely. MedEdPublish. 9(1):3031. https://
gogy: using twitter for knowledge exchange. Med Educ. DOI:10. www.mededpublish.org/manuscripts/3031.
1111/medu.14242. https://onlinelibrary.wiley.com/doi/full/10.1111/ Kirkpatrick JD, Kirkpatrick WL. 2016. Four levels of training evaluation.
medu.14242. East Peoria (IL). Versa Press, Inc.
Gaber DA, Shehata MH, Amin HA. 2020. Online team-based learning Lapolla P, Mingoli A. 2020. COVID-19 changes medical education in
sessions as interactive methodologies during the pandemic. Med Italy: will other countries follow? Postgrad Med J. 96(1137):375–376.
 MEDICAL TEACHER 1215

https://pmj.bmj.com/content/postgradmedj/early/2020/05/13/post- lessons learnt managing the COVID-19 crisis by School of Medicine.

gradmedj-2020-137876.full.pdf. National University of Singapore. MedEdPublish. 9(1):3034. https://
Lubarsky S. 2020. Movie night! An entertaining online educational www.mededpublish.org/manuscripts/3034.
method for introducing students to common presentations in neur- Singh K, Srivastav S, BhardwaJ A, Dixit A, Misra S. 2020. Medical educa-
ology. Med Educ. DOI:10.1111/medu.14218. https://onlinelibrary. tion during the COVID-19 pandemic: a single institution experience.
wiley.com/doi/full/10.1111/medu.14218. Indian Pediatr. 57(7):678–679. https://pubmed.ncbi.nlm.nih.gov/
Merali Z, Carayannopoulos KL, Lai A. 2020. All hands on deck: creation 32366728/.
of an online internal medicine redeployment curriculum. Med Educ. Soled D, Goel S, Barry D, Erfani P, Joseph N, Kochis M, Uppal N,
DOI:10.1111/medu.14213. https://onlinelibrary.wiley.com/doi/full/10. Velasquez D, Vora K, Scott KW. 2020. Medical student mobilization
1111/medu.14213.
during a crisis. Acad Med. DOI:10.1097/ACM.0000000000003401.
Moszkowicz D, Duboc H, Dubertret C, Roux D, Bretagnol F. 2020. Daily
https://journals.lww.com/academicmedicine/Abstract/9000/Medical_
medical education for confined students during COVID-19 pan-
Student_Mobilization_During_A_Crisis_.97220.aspx.
demic: a simple videoconference solution. Clin Anat. DOI:10.1002/
Srinivasan DK. 2020. Medical students’ perceptions and an anatomy
ca.23601. https://onlinelibrary.wiley.com/doi/full/10.1002/ca.23601.
Murdock HM, Penner JC, Le S, Nematollahi S. 2020. Virtual morning teacher;s personal experience using an e-learning platform for tutori-
report during COVID-19: a novel model for case-based teaching als during the Covid-19 Crisis. Anat Sci Educ. 13(3):318–319. https://
conferences. Med Educ. DOI:10.1111/medu.14226. https://onlineli- anatomypubs.onlinelibrary.wiley.com/doi/abs/10.1002/ase.1970.
brary.wiley.com/doi/abs/10.1111/medu.14226. Sterne JAC, Hern!an MA, McAleenan A, Reeves BC, Higgins JPT. 2019.
Oldenburg R, Marsch A. 2020. Optimizing teledermatology visits for Chapter 25: assessing risk of bias in a non-randomized study.
dermatology resident education during the COVID-19 pandemic. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ,
J Am Acad Dermatol. 82(6):e229. https://www.jaad.org/article/ Welch VA, editors. Cochrane handbook for systematic reviews of
S0190-9622(20).30520-X/pdf interventions version 6.0. Hoboken: John Wiley & Sons, Inc. www.
Parisi MCR, Frutuoso L, Benevides SSN, Barreira NHM, Silva JLG, Pereir training.cochrane.org/handbook.
MC, Cecilio-Fernandes D. 2020. The challenges and benefits of Sterne JAC, Hern!an MA, Reeves BC, Savovi!c J, Berkman ND,
online teaching about diabetes during the COVID-19 pandemic. Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, et al.
Diabetes Metab Syndr. 14(4):575–576. https://www.sciencedirect. 2016. ROBINS-I: a tool for assessing risk of bias in non-randomized
com/science/article/pii/S1871402120301089?via%3Dihub. studies of interventions. BMJ. 355:i4919.
PRISMA. 2015. Preferred Reporting Items for Systematic Reviews and Sudhir M, Mascarenhas S, Isaac J, Alfroukh J, Abdul Rahuman S. 2020.
Meta-Analyses (PRISMA); [accessed 2020 June 25]. http://prisma- Adapting to the need of the hour: communication skills simulation
statement.org/. session using an online platform during COVID-19. MedEdPublish.
Rasmussen S, Sperling P, Poulsen MS, Emmersen J, Andersen S. 2020. 9(1):3048. https://www.mededpublish.org/manuscripts/3048.
Medical students for health-care staff shortages during the COVID- Taylor D, Grant J, Hamdy H, Grant L, Marei H, Venkatramana M. 2020.
19 pandemic. The Lancet. 395(10234):e79–80–e80.
Transformation to learning from a distance. MedEdPublish. 9(1):
Reed D, Price E, Windish D, Wright S, Gozu A, Hsu E, et al. 2005.
2999. https://www.mededpublish.org/manuscripts/2999.
Challenges in systematic reviews of educational intervention stud-
Torres A, Doman !ska-Glonek E, Dzikowski W, Korulczyk J, Torres K.
ies. Ann Intern Med. 142(12_Part_2):1080–1099.
2020. Transition to on-line is possible: solution for simulation-based
Rose C, Mott S, Alvarez A, Lin M. 2020. Physically distant, educationally
connected: interactive conferencing in the era of COVID-19. Med teaching during pandemic. Med Educ. DOI:10.1111/medu.14245.
Educ. 54(8):758–759. https://onlinelibrary.wiley.com/doi/pdf/10. https://onlinelibrary.wiley.com/doi/abs/10.1111/medu.14245.
1111/medu.14192. Ungtrakul T, Lamlertthon W, Boonchoo B, Auewarakul C. 2020. Virtual
Roy SF, Cecchini MJ. 2020. Implementing a structured digital-based multiple mini-interview during a COVID-19 pandemic. Med Educ. 54(8):
online pathology curriculum for trainees at the time of COVID-19. J 764–765. https://onlinelibrary.wiley.com/doi/abs/10.1111/medu.14207.
Clin Path. DOI:10.1136/jclinpath-2020-206682. https://jcp.bmj.com/ Veasuvalingam B, Goodson ML. 2020. Falling back on technology mind-
content/early/2020/05/04/jclinpath-2020-206682. fully during COVID-19 pandemic: NUMed campus experience.
Samarasekera DD, Goh DLM, Yeo SP, Ngiam NSP, Aw MM, Lim MM, MedEdPublish. 9(1):2998. https://www.mededpublish.org/manuscripts/
Pillai S, Lee SS, Mahadevan M, Kow A, et al. 2020. Response and 2998.