Received: 15 October 2020 | Revised: 15 March 2021 | Accepted: 7 April 2021

DOI: 10.1002/aet2.10604

ORIGINAL CONTRIBUTION

Website usability analysis of United States emergency

medicine residencies

Edwin Fundingsland1 | Joseph Fike1 | Joshua Calvano1 | Ali Raja MD2 |

Deborah Lai3 | Sara Silacci4 | Mary Haas MD5 | Teresa Chan MD6 |
Shuhan He MD7

1
Rocky Vista University College of
Osteopathic Medicine, Parker, Colorado, Abstract
USA
Objectives: The Council of Residency Directors (CORD) in Emergency Medicine (EM)
2
Department of Emergency Medicine,
Massachusetts General Hospital, Boston,
has recommended that all residency programs should conduct virtual interviews for
Massachusetts, USA the 2020 to 2021 application cycle due to the COVID-­19 pandemic. While factors
3
Division of Psychology and Language such as geographical region, city, program size, or hospital affiliation are not modifi-
Sciences, University College London,
London, UK able, EM residencies can bridge the information gap created by a lack of face-­to-­face
4
Center for Innovation in Digital interaction by representing themselves digitally. Measuring usability provides an ob-
HealthCare, Massachusetts General
jective method for EM residencies to improve their Web presence and effectively
Hospital, Boston, Massachusetts, USA
5
Department of Emergency Medicine, represent themselves to applicants.
University of Michigan Medical School, Methods: Our sample set included 55 U.S. EM residency program websites. Using
Ann Arbor, Michigan, USA
6 methodology replicated from previous literature on health care website usability, we
Division of Emergency Medicine,
Department of Medicine, Program for divided usability into four categories for quantifiable analysis: accessibility, marketing,
Faculty Development, Faculty of Health
content quality, and technology. Analysis was performed on each website and scored in
Sciences, McMaster University, Hamilton,
Ontario, Canada all four categories. A “general usability” score was calculated for each website using a
7
Center for Innovation in Digital composite of the key factors within the four categories. Using a weighted percentage
HealthCare, Massachusetts General
Hospital, Boston, Massachusetts, USA
across all of the factors, an overall score was calculated.
Results: Content quality was the overall highest scoring category (mean ± SD = 5.4, SE =
Correspondence
Edwin Fundingsland, Rocky Vista
0.33). The overall lowest performing category was technology (mean ± SD = 0.8 ± 0.09,
University College of Osteopathic SE = 0.01).
Medicine, 8401 S. Chambers Road, Parker,
CO 80134, USA.
Conclusions: Measuring usability can help EM residency programs identify ways to
Email: [email protected] improve their Web presence. To effectively promote their programs, residencies need

Supervising Editor: Jason Wagner, MD.

quality content that communicates their key features. Our recommendation is for all
residency programs to periodically perform website audits and apply the usability
measures outlined to improve their digital presence, especially during times when
face-­to-­face interactions will be limited.

KEYWORDS
emergency medicine residencies, digital health, healthcare website, usability testing, website
usability, Web crawler

Presented at the American College of Emergency Physicians Research Forum (Virtual Conference), October 2020.

© 2021 by the Society for Academic Emergency Medicine

AEM Educ Train. 2021;5:e10604.  wileyonlinelibrary.com/journal/aet2 | 1 of 8

https://doi.org/10.1002/aet2.10604
2 of 8 | FUNDINGSLAND et al.

I NTRO D U C TI O N and overall ease of navigation.6 Previous research has analyzed us-
ability in e-­commerce, e-­government, mobile news apps, and library
Background websites.7-­10 With regard to health care–­related websites, usability
has been analyzed in hospital, children's hospital, cancer center, and
With COVID-­19 upending the traditional residency application pro- digital health care center websites.11-­14 A website with enhanced us-
cess, residency programs and medical students are finding themselves ability tends to garner higher levels of engagement and therefore
needing to adapt to changes in the 2020 to 2021 cycle. This is primarily better accomplishes its goals.15-­17 Accordingly, industries outside of
due to fewer opportunities for in-­person contact and communication. health care have established standardized guidelines for measuring
The Coalition for Physician Accountability has made several recom- usability in the areas of accessibility, content quality, marketing, and
mendations in an effort to minimize in-­person interaction with resi- technology.15-­17 Expectations of user experience have been estab-
dency programs including the limitation of away rotations, conduction lished, and health care websites are facing pressures to conform to
of virtual interviews, and virtual orientation and tours.1 The Council these requirements.18,19
of Residency Directors (CORD) in Emergency Medicine (EM) has rec- Because EM residency program websites will now play a larger
ommended that all EM residency programs conduct their interviews role than ever in recruiting potential applicants, the importance of
virtually.2 Similarly, the Association of Faculties of Medicine of Canada evaluating and optimizing usability has increased. Residency pro-
Resident Matching Committee (ARMC) decided that all interviews for gram websites have previously been analyzed for content quality
the 2021 Medicine Subspecialty Match, Pediatric Subspecialty Match, within the specialties of general surgery, neurosurgery, diagnostic
and Family Medicine/Enhanced Skills Match will be in a virtual format and interventional radiology, dermatology, physical medicine and
including those for local candidates.3 Traditionally, EM residency in- rehabilitation, cardiothoracic surgery, urology, orthopedic surgery,
terviews have allowed programs to showcase their cities and facilities. otolaryngology, plastic surgery, vascular surgery, and radiation on-
Interviews have also conventionally offered students and programs cology.19-­30 To the best of our knowledge, no prior analysis of EM
the opportunity to interact socially through events such as mixers residency website usability has been performed. This analysis may
and/or dinners. These factors remain difficult to overcome virtually. inform EM residencies how to improve their online presence and
Additionally, CORD has recommended that students and EM resi- more effectively represent themselves to potential applicants.
dency programs work to minimize the amount of visiting student rota-
tions, to reduce the amount of travel during this pandemic.2 Although
two EM rotations including one home and one “away” or “visiting” Objectives
clerkship was the pre–­COVID-­19 standard recommendation for medi-
cal students, students may now only be able to rotate at their home Aim 1 was to categorize EM residency programs and their websites;
4
program. Visiting student rotations have traditionally afforded more aim 2 was to utilize a previously published usability scoring system
contact between applicants, personnel, and programs, allowing for to objectively and quantitatively analyze their websites;11 and aim 3
improved assessment of compatibility. These rotations give students was to identify themes to suggest areas of improvement among EM
the chance to learn the setting (i.e., county, community, academic) residency websites.
5
they prefer and augment their perception of overall “fit.” Due to these
changes, both students and EM residency programs will make high-­
stakes decisions with less information than in previous years. Although M E TH O D S
programs cannot change the circumstances surrounding COVID-­19 or
their own nonmodifiable factors (i.e., geographical region, city, pro- We conducted a cross-­sectional usability audit of U.S. EM residency
gram size, hospital affiliation), they can better represent themselves websites.
digitally via their websites and social media. A residency program's
website is often their first impression on an applicant. Although the
content contained on the website is crucial, the manner in which this Sample selection
content is presented is also important. Understanding the technical as-
pects of website usability can help one website stand out from another Our target website population was U.S. EM residency programs.
and should not be overlooked when building a website. With a lack of We started with 251 programs that were listed as Accreditation
face-­to-­face interaction, programs can minimize the information gap Council for Graduate Medical Education (ACGME) accredited on
by enhancing their website usability. the Electronic Residency Application Service (ERAS) website. We
refined our sample set by only including programs that use their
own primary domain or subdomain. Programs using a subpage of a
Website usability for EM residency programs larger domain (i.e., hospital or university) were excluded as the anal-
ysis would include non–­residency-­related content (i.e., patient care,
Usability extends beyond a website's external appearance; it also patient portals, residency programs other than EM). Without using
encompasses variables of “user experience” including website errors the exclusion criteria there would likely be artificially high scores
WEBSITE USABILITY ANALYSIS OF U.S. EM RESIDENCIES | 3 of 8

for pages with a large primary domain that added a small residency of the tool to analyze a given target factor (i.e., website speed) in
subpage. Websites that were inconclusive or showed errors upon a user-­friendly way. The primary tool used for website evaluation
analysis were also excluded. Our final sample set included 55 U.S. was a “Web crawler.” A Web crawler uses the website’s URL to cre-
EM residency programs. This process is represented in Figure 1. ate a topographical map of a website and its subpages to analyze
it for errors, content, and metadata including titles, keywords, and
descriptions.31 There were two authors involved in data gathering
Overview and rating who were thoroughly trained by an expert. Both raters fa-
miliarized themselves with the instruction manuals associated with
All data were collected between May 27, 2020, and June 7, 2020, each tool to ensure an accurate and reproducible analysis. The same
using tools that assessed the website usability of each program. We two authors then divided the tools among themselves with only one
replicated a methodology that was previously outlined by Calvano author assigned to each tool and collected data for their given set
et al.11 who ranked the usability of digital health care center web- of tools. This ensured that the data gathered using a certain tool
sites. We maintained the definitions and scoring system described was not influenced by different computer capabilities or Internet
by Calvano et al. and applied the same formulas to determine the connections, thereby minimizing potential discrepancies. Factors
usability of the websites assessed in our study. The four categories that may rely on the users’ Internet connection (i.e., speed) were run
that were identified and defined in previous literature as aspects of using two different tools and averaged to provide fair and accurate
usability are: values.
The resulting scores given by each tool were then assigned to
1. Accessibility—­ability of users with lower levels of computer one of four categories: 1) accessibility, 2) content, 3) marketing, or
literacy to access and navigate the website. 4) technology. A “general usability” score was calculated using a
2. Marketing—­ability of the website to be found through search combination of key factors relating to each of the four categories.
engines. Finally, an “overall usability” score, which looked at variables across
3. Content quality—­lack of grammatical errors, frequency of content the previous categories, was used to provide a ranking system. A
updates, content relevancy, and readability. description, information about the rating scale, and the significance
4. Technology—­website download speed, quality of the program- of each category are described in Table S1, which Calvano et al. pro-
ming code, and website infrastructure.11–­13 vided for our explicit use.

Data analysis Accessibility

We built a database of ACGME-­accredited EM residency program Accessibility is a category intended to represent how well a web-
websites that used their own primary domain or subdomain. We then site caters to a diverse population, regardless of the level of literacy,
scored each website according to a variety of usability tools outlined technical skills, or presence of disabilities. Accessibility includes the
thoroughly by Calvano et al. in Data Supplement S1, Table S1 (avail- following variables: meta description, functionality, readability, and
able as supporting information in the online version of this paper, overall layout. Meta description refers to the “snippet” page sum-
which is available at http://onlin​
elibr​
ary.wiley.com/doi/10.1002/ mary that appears when a site is the result of a search engine in-
aet2.10604/​full). These tools were selected based on the ability quiry. Functionality looks at features allowing users to view aspects

F I G U R E 1 Sample selection criteria for

EM residency websites
4 of 8 | FUNDINGSLAND et al.

of a website with content levels appropriate for user understanding, computers, tablets, and mobile phones). Back-­end design includes
regardless of their literacy levels. An estimated 43% of American the programming code that runs the website. Programming code and
adults have been observed to have basic or below basic literacy other Web components (i.e., databases) are stored on servers that
rates.32 The use of assistive technologies also falls under accessibil- allow user access on any suitable device. Additionally, the speed of a
ity. This refers to features such as screen readers/magnifiers of a site (amount of time it takes to load) is server-­dependent. This plays
33
website. With tools to apply algorithmic scales, the websites were an important part in adding and maintaining users. According to a
ranked on their level of reading difficulty as well as approximat- recent Google study, a website that takes longer than 3 seconds to
ing the grade level required for content comprehension in order to load on a mobile device loses approximately 53% of its users and the
grade accessibility. average mobile website speed is around 18 seconds.34

Content quality General usability

Content quality assesses both the positive and the negative ele- General usability encompasses the metrics from the previous four
ments of a website's published content. Content quality includes the categories. This score assesses the overall quality of a website and
following variables: information relevance, generated metadata, use provides a point of reference for EM residencies to begin an audit of
of multimedia for imagery, and relevancy of written text. Relevancy their websites. The more specific categories allow them to look for
refers to the content's pertinence and accuracy to a particular topic areas of improvement.
at a specific point in time. In our context, websites dedicated to pre-
senting information about an EM residency program were evaluated
on the ability to provide both relevant and accurate information to Overall usability
applicants (i.e., application requirements, curriculum details, ben-
efits). Evaluation of multimedia addresses both quantity and quality Overall usability is the rank order calculation we used for complete
(i.e., resolution) of a website's multimedia. Metadata function adds assessment of all major and minor variables across the five previ-
support to the composed content. Analysis of written text looks at ous categories. Percentages were assigned accordingly to create a
both grammar and spelling. weighted, comprehensive usability ranking system.

Marketing R E S U LT S

Marketing addresses the ease of discovery of a particular website. Due to individual technical issues on specific websites, six websites
There is specific emphasis on its search engine results pages (SERP). were eliminated from our original set of 61. Most of these issues
When Internet users place an online search via a search engine (i.e., were related to errors with the Web crawler, potentially due to the
Google), SERP refers to the order in which websites are presented lack of index restrictions put in place by the website administra-
to the user. Websites that present higher on the list have increased tors. Scores were assigned to the remaining (N = 55) EM residency
visibility making SERPs a crucial factor in digital marketing. Search websites.
engine optimization (SEO) is an entire field dedicated to the optimi- Accessibility had a mean (±SD) score of 1.9 (±0.62; standard
zation of this practice. SEO is an effective way health care websites, error [SE] = 0.08). Content quality was the highest mean (±SD)
including EM residency programs, can create a similar corporate scoring category by a significant margin with a score of 5.4 (±2.48;
presence to other industries. The specifics of SEO auditing, however, SE = 0.33). Marketing had a mean (±SD) score of 1.3 (±0.48; SE =
are outside of the scope of this study. 0.06). The overall lowest performing category was technology, with
a mean (±SD) score of 0.7 (±0.09; SE = 0.01). General usability had
a mean (±SD) score of 1.3 (±0.39; SE = 0.05). Summary statistics of
Technology all categories are shown in Table 1. The overall rankings for the 55
assessed websites are presented in Table S2.
Technology evaluates a website's technical functionality. Rather Leaders among ranking categories include the following:
than content, it looks at the quality of technological design and per- accessibility—­Christiana Care Health Services (3.5); content
formance. This includes front-­end design, user experience, back-­end quality—­Christiana Care Health Services (11.7); marketing—­seven-­
coding infrastructure, and server management. Front-­end design way tie between Stanford University, UCLA David Geffen School of
refers to what is visible to users browsing a website. Analysis of Medicine, University of Arkansas for Medical Sciences, University
front-­end design includes looking at aspects of hypertext markup of California (Irvine), University of Florida College of Medicine
language (HTML), which assesses the ease of navigation based on Jacksonville, Virginia Commonwealth University Health Systems,
layout and how well a website is scalable across different devices (i.e., and Washington University (2.1); technology—­five-­way tie between
WEBSITE USABILITY ANALYSIS OF U.S. EM RESIDENCIES | 5 of 8

TA B L E 1 EM residency websites:
Standard
summary statistics from usability analysis
Category Mean (SE) Deviation Minimum Maximum

Accessibility 1.9 (0.08) 0.62 0.6 3.5

Content quality 5.4 (0.33) 2.48 0.2 11.7
Marketing 1.3 (0.06) 0.48 0.5 2.1
Technology 0.7 (0.01) 0.09 0.6 0.9
General usability 1.3 (0.05) 0.39 0.5 2.4

Abbreviation: SE, standard error.

Brookdale University Hospital and Medical Center, Christiana be resembled by a lack of server capacity or infrequent website/
Care Health Services, Tower Health, University of Arkansas for social media audits. Increasing website speed is a method in which
Medical Sciences, and the University of Florida College of Medicine websites can instantly improve their technology score. Increasing
Jacksonville (0.9); and general usability—­Christiana Care Health speed can largely be achieved by minimizing the amount of conflict-
Services Program (2.4). The top ranked website for overall usability ing technology on the back-­end server. Additionally, efforts can be
was also Christiana Care Health Services Program (3.2). For the cat- made to improve front-­end design by enhancing ease of navigation
egories that ended in a tie, the websites had scores within a 100th of and making websites scalable across different devices. By working
a decimal point. Table S2 does not reflect these as a tie. with experts in user experience and user interface design, websites
can address these areas.
Our results show that marketing also scored low among EM
DISCUSSION residency program websites. If looking for improvement, programs
can work with experts in SEO to optimize their websites to become
After thorough investigation of usability, the general usability cat- more discoverable via search engines. They can also increase ef-
egory was found to be low performing, indicating overall room for forts in promoting their websites via social media or other affiliated
improvement amongst EM residency websites. Content quality was websites to increase their referral traffic. Promoting overall brand
the overall highest scoring category by a significant margin and tech- strength would also increase overall website traffic.
nology was the lowest performing category. When comparing our research to similar studies performed on
A low mean general usability score indicates that these EM res- different residency programs, the results are fairly consistent re-
idency programs do not have a complete understanding of usabil- gardless of specialty. Although studies evaluated websites based on
ity metrics and what is necessary for overall website quality. This tools and/or metrics that differed from our methodology, all similar
is resembled by the disproportionate scoring between the analyzed studies across residencies indicated that there needs to be improve-
categories. On the surface, it is easy for programs to view the quality ment to website presence from most institutions.19-­30 Our methods
of their website simply based on their content. However, by utilizing were a direct continuation of a study by Calvano et al.11 that ranked
usability analysis these websites can examine beyond what is on the the usability of digital health care center websites. Previous research
surface and discover other aspects in which they can improve. allowed the authors to compare usability trends in health care, in-
Regarding content quality, our data shows that EM residency cluding digital health care centers, hospitals, and children's hospi-
websites have placed heavy emphasis on providing accurate and rel- tals.11–­13 In previously published usability studies, content quality
evant information about their residency programs. This represents was also found to be the highest ranking category.11–­13 Health care
the notation that, from a logistical standpoint, EM residency web- organizations as a whole have placed emphasis on providing factual
sites are primarily concerned with their content. It is important to consumer scientific information. Overall, health care websites are
inform these programs, however, that content quality is only a par- primarily concerned with their content and, therefore, neglect other
tial contributor and other factors need to be addressed to improve aspects of usability. Technology being our lowest mean ranked cat-
general usability. In the event that a program was looking to improve egory was also consistent with previous research.11–­13 This suggests
their content, they could start by performing frequent audits, mak- a lack of importance placed on digital and information technology in
ing sure their content is up to date. They could also address whether the field of health care.
the information they are presenting is relevant to what applicants The biggest difference between our findings and previous find-
are looking for. Additionally, evaluating both the quality and the ings was that an evaluation of children's hospital websites found
quantity of their multimedia and confirming that their websites do accessibility to be the lowest score category as opposed to technol-
not contain spelling or grammatical errors will improve their overall ogy.13 Accessibility also ranked low in other previous studies.11,12 In
content quality. our study, accessibility was found to be the second highest scoring
Technology being the lowest mean ranked category suggests category. It appears that EM residency programs have an increased
a lack of importance placed on digital and information technol- understanding of the importance of accessibility compared to other
ogy among these residency programs. A lack of investment may health care organizations, indicating that they may be focused on
6 of 8 | FUNDINGSLAND et al.

creating content that can be easily accessed and comprehended by websites that they only make available to applicants they plan to
a diverse population. However, if EM residency websites are look- interview. Because there is not a standardized list containing these
ing to improve their accessibility, they could focus on making their secondary websites, they would be tenuous to find and therefore
platforms easy to navigate, publish content that is easy to read for we did not include them in our analysis. Due to this limitation, we
those with low literacy levels, and confirm compatibility with screen believe that it would be beneficial to include these websites into fu-
readers and magnifiers. An assumption could be made that individ- ture studies.
uals accessing residency websites likely have sufficient levels of A minor limitation was in the assessment of a website's social
education/literacy necessary for appropriate comprehension, and media presence. Some websites did not have direct links to their
therefore this measure is unnecessary. However, one goal of this social media profiles. In these cases, Facebook and Twitter's self-­
research was to promote a consistent standard of evaluation across hosted search engines were used. Oftentimes the desired page was
websites, regardless of the website type. Standardization of website distant from the top results. This created uncertainty as to whether
analysis in the health care sector has been neglected in many areas all of the official social media pages were discovered and empha-
and this is an important practice in other industries.15-­17 Additionally, sizes the necessity of embedding social media links to improve user
it is key to have a standardized framework for understanding equity experience.
for users such as applicants who speak English as a second language An additional limitation included the measurement of website
(i.e., international applicants) and the family members of applicants. speed. This measurement can be variable with dependence on the
These data would benefit from future analysis comparing web- time of data collection. Depending on the time, there may be differ-
site usability between EM residency programs that were established ences in the Internet connectivity or changes to the website servers
pre-­Web (pre-­1990) and post-­Web. This would address if pre-­Web or computer hardware. This bias was minimized using the same com-
programs are underperforming in usability metrics compared to later puter and network to run all of the tools. Finally, data were collected
established programs and provide additional insight on the groups over a span of 12 days meaning that some of the information may
of residency programs that would benefit the most from improving have changed since the initial evaluation.
their website usability.
Health care is evolving through technology to improve quality of
care while decreasing costs.35 For these reasons, usability has be- CO N C LU S I O N
come an important method for analyzing website presence through-
out the health care sector, including education and medical training. In the 2020 to 2021 residency application cycle, the majority of
With additional pressures that COVID-­19 is placing on this year's medical students pursuing emergency medicine will make residency
application cycle and interview season, EM residency Web presence selection decisions without physically meeting program members or
has become more important than ever. seeing facilities in person. Many potential residents will be looking
to a program's website and social media presence to gain a better
understanding of their compatibility with the program. Our results
LI M ITATI O N S provide EM residencies with areas upon which to focus improve-
ment in website usability efforts. The need for overall refinement is
The authors recognize that this study includes limitations. Perhaps highlighted by the mean general usability score of 1.3. Our data have
the largest being the amount of EM residency websites that the identified that content quality is the highest rated usability category
authors were able to accurately analyze. Out of the 251 ACGME-­ and technology is the lowest. We recommend that EM residency
accredited EM residencies, only 55 met the inclusion criteria. This programs include periodic usability audits of their websites to make
was primarily due to a limitation in the methods. The website analy- sure they are adequately performing in all categories.
sis tools evaluate a website's entire primary domain or subdomain
while many EM residencies use a subpage of a larger domain (i.e., C O N FL I C T O F I N T E R E S T
hospital or university). If these websites were included, data would The authors have no potential conflicts to disclose.
have been generated based on the entire hospital/university's do-
main, most of which is not residency related. For example, a primary AU T H O R C O N T R I B U T I O N S
domain may have an estimated domain age of 30+ years, but the Study concept and design: Joshua Calvano, Shuhan He, Deborah
EM residency subpage may have only been added 10 years ago. This Lai. Acquisition of data: Edwin Fundingsland, Joseph Fike. Analysis
would have misrepresented the data and given these subpages rank- and interpretation of data: Edwin Fundingsland, Joseph Fike, Joshua
ing advantages and thus were excluded. Calvano, Ali Raja, Deborah Lai, Sara Silacci, Mary Haas, Teresa Chan,
Additionally, our sample of EM residency program websites only Shuhan He. Drafting of the manuscript: Edwin Fundingsland, Joseph
included each program's official, public website. Certain programs Fike, Joshua Calvano, Ali Raja, Deborah Lai, Sara Silacci, Mary Haas,
use separate websites/blogs that allow them to circumvent institu- Teresa Chan, Shuhan He. Critical revision of the manuscript for
tional IT blocks. Other programs have private/password-­protected important intellectual content: Edwin Fundingsland, Joseph Fike,
WEBSITE USABILITY ANALYSIS OF U.S. EM RESIDENCIES | 7 of 8

Joshua Calvano, Ali Raja, Deborah Lai, Sara Silacci, Mary Haas, 13. Huerta TR, Walker DM, Ford EW. An evaluation and ranking of chil-
Teresa Chan, Shuhan He. Statistical expertise: Ali Raja, Deborah dren’s hospital websites in the United States. J Med Internet Res.
2016;18(8):e228.
Lai, Sara Silacci, Mary Haas, Teresa Chan, Shuhan He. Acquisition
14. Huerta TR, Walker DM, Ford EW. Cancer center website rankings
of funding: N/A. in the USA: expanding benchmarks and standards for effective
public outreach and education. J Cancer Educ. 2017;32(2):364-­373.
ORCID 15. Oermann MH, Lowery NF, Thornley J. Evaluation of web sites on
management of pain in children. Pain Manag Nurs. 2003;4(3):99-­105.
Edwin Fundingsland https://orcid.org/0000-0002-7100-7555
16. Oermann MH, Lesley ML, VanderWal JS. Using web sites on quality
Joseph Fike https://orcid.org/0000-0003-4043-2185 health care for teaching consumers in public libraries. Qual Manag
Joshua Calvano https://orcid.org/0000-0003-3938-2336 Health Care. 2005;14(3):188-­195.
Ali Raja https://orcid.org/0000-0002-8909-3586 17. Oermann MH, McInerney SM. An evaluation of sepsis web sites for
patient and family education. Plast Surg Nurs. 2007;27(4):192-­196.
Deborah Lai https://orcid.org/0000-0003-2542-4793
18. Mittler JN, Volmar KM, Shaw BW, Christianson JB, Scanlon DP.
Sara Silacci https://orcid.org/0000-0002-7271-7145
Using websites to engage consumers in managing their health and
Mary Haas https://orcid.org/0000-0002-9506-5928 healthcare. Am J Manag Care. 2012;18(6 Suppl):s177-­s184.
Teresa Chan https://orcid.org/0000-0001-6104-462X 19. Liang CJ, Chen HJ. A study of the impacts of website quality on
Shuhan He https://orcid.org/0000-0002-9902-5077 customer relationship performance. Total Qual Manag Bus Excell.
2009;20(9):971-­988.
20. Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB. General
REFERENCES surgery residency program websites: usefulness and usability for
1. The Coalition for Physician Accountability’s Work Group on resident applicants. Curr Surg. 2004;61(2):236-­240.
Medical Students Moving Across Institutions for Post Graduate 21. Skovrlj B, Silvestre J, Ibeh C, Abbatematteo JM, Mocco J.
Training. Updated recommendations on away rotations for medi- Neurosurgery residency websites: a critical evaluation. World
cal education institutions of LCME®-­ accredited, U.S. Osteopathic, Neurosurg. 2015;84(3):727-­733.
and Non-­U.S. Medical School Applicants. Coalition for Physician 22. Ahmed SA, Hyman C, Eltorai AE, Ahn SH. Evaluation of integrated
Accountability; 2021. interventional radiology residency websites. R I Med J (2013).
2. Emergency Medicine Residents’ Association; American Academy of 2019;102(6):19-­23.
Emergency Medicine; Council of Residency Directors in Emergency 23. Ashack KA, Burton KA, Soh JM, et al. Evaluating dermatology res-
Medicine, et al. Consensus Statement on the 2020-­2021 Residency idency program websites. Dermatol Online J. 2016;22(3):13030/
Application Process for US Medical Students Planning Careers in qt7rx8j2dn.
Emergency Medicine in the Main Residency Match. Irving, TX: 24. Patel SJ, Abdullah MS, Yeh PC, Abdullah Z, Jayaram P. Content eval-
Emergency Medicine Residents’ Association; 2020. uation of physical medicine and rehabilitation residency websites.
3. Virtual Interviews for the 2021 Medicine Subspecialty Match, PM&R. 2020;12(10):1003-­1008.
Pediatric Subspecialty Match and Family Medicine, Enhanced Skills 25. Novin SA, Yi PH, Vanderplas T, Magid D. How well do we represent
Match. The Association of Faculties of Medicine of Canada web- ourselves? A Student-­centric analysis of radiology residency web-
site. 2020. Accessed September 3, 2020. https://afmc.ca/en/media​ site content. Curr Probl Diagn Radiol. 2019;48(5):427-­432.
-­relea​ses/may-­26-­2020 26. Miller VM, Padilla LA, Schuh A, et al. Evaluation of cardiothoracic
4. Shandro J, Kessler R, Schrepel C, Jauregui J. Advising medical stu- surgery residency and fellowship program websites. J Surg Res.
dents during COVID-­19: the case for a single emergency medicine 2019;246:200-­206.
rotation for all. AEM Educ Train. 2020;4(3):318-­320. 27. Patel BG, Gallo K, Cherullo EE, Chow AK. Content analysis of
5. Garmel GM, Pettis HM, Lane DR, et al. Clerkships in emergency ACGME accredited urology residency program webpages. Urology.
medicine. J Emerg Med. 2020;58(4):e215-­e222. 2020;138:11-­15
6. Griffiths KM, Christensen H. Website quality indicators for con- 28. Oladeji LO, Yu JC, Oladeji AK, Ponce BA. How useful are orthopedic
sumers. J Med Internet Res. 2005;7(5):e55. surgery residency web pages? J Surg Educ. 2015;72(6):1185-­1189.
7. Kuan HH, Bock GW, Vathanophas V. Comparing the Effects of 29. Svider PF, Gupta A, Johnson AP, et al. Evaluation of otolaryngology
Usability on Customer Conversion and Retention at E-­Commerce residency program websites. JAMA. 2014;140(10):956.
websites. In: Proceedings of the 38th Annual Hawaii International 30. Silvestre J, Tomlinson-­Hansen S, Fosnot J, Taylor JA. Plastic surgery
Conference on System Sciences. IEEE; p. 174a-­174a. Accessed May residency websites: a critical analysis of accessibility and content.
2, 2020. http://ieeex​plore.ieee.org/docum​ent/13855​8 4/ Ann Plast Surg. 2014;72(3):265-­269.
8. Huang Z, Benyoucef M. Usability and credibility of e-­government 31. Devi RS, Manjula D, Siddharth RK. An efficient approach for web
websites. Gov Inf Q. 2014;31(4):584-­595. indexing of big data through hyperlinks in web crawling. Sci World J.
9. Jeong W, Jung HH. Usability study on newspaper mobile websites. 2015;2015:739286.
OCLC Syst Serv Int Digit Libr Perspect. 2012;28(4):180-­198. 32. Kutner M, Greenberg E, Jin Y, Boyle B, de Hsu YC. Literacy in
10. Na VR. An analysis of usability features of library web sites. Ann Libr Everyday Life: Results from the 2003 National Assessment of Adult
Inf Stud. 2008;55(2). Literacy. Washington, DC: National Center for Education Statistics;
11. Calvano J, Fundingsland E Jr, Lai D, Silacci S, Raja A, He S. Website 2007.
rankings for digital health centers in the USA: applying usabil- 33. Ismail A, Kuppusamy KS, Nengroo AS. Multi-­tool accessibility as-
ity testing for public engagement (preprint). JMIR Human Factor. sessment of government department websites:a case-­study with
https://doi.org/10.2916/20721 JKGAD. Disabil Rehabil Assist Technol. 2018;13(6):504-­516.
12. Huerta TR, Hefner JL, Ford EW, McAlearney AS, Menachemi N. 34. New Industry Benchmarks for Mobile Page Speed -­ Think With
Hospital website rankings in the United States: expanding bench- Google. Google website. 2020. Accessed May 2, 2020. https://
marks and standards for effective consumer engagement. J Med www.think​w ithg​o ogle.com/marke​t ing-­r esou​r ces/data-­m easu​
Internet Res. 2014;16(2):e64. remen​t/mobil​e-­page-­speed​-­new-­indus​try-­bench​marks/.
8 of 8 | FUNDINGSLAND et al.

35. DePasse JW, Chen CE, Sawyer A, Jethwani K, Sim I. Academic medi-
cal centers as digital health catalysts. Healthcare. 2014;2(3):​173-­176. How to cite this article: Fundingsland E, Fike J, Calvano J,
et al. Website usability analysis of United States emergency
medicine residencies. AEM Educ Train. 2021;5:e10604.
S U P P O R T I N G I N FO R M AT I O N
https://doi.org/10.1002/aet2.10604
Additional supporting information may be found online in the
Supporting Information section.