van Lent et al.

BMC Health Services Research 2010, 10:253

http://www.biomedcentral.com/1472-6963/10/253

RESEARCH ARTICLE Open Access

International benchmarking of specialty

hospitals. A series of case studies on
comprehensive cancer centres
Wineke AM van Lent1*, Relinde D de Beer2, Wim H van Harten1,3

Abstract
Background: Benchmarking is one of the methods used in business that is applied to hospitals to improve the
management of their operations. International comparison between hospitals can explain performance differences.
As there is a trend towards specialization of hospitals, this study examines the benchmarking process and the
success factors of benchmarking in international specialized cancer centres.
Methods: Three independent international benchmarking studies on operations management in cancer centres
were conducted. The first study included three comprehensive cancer centres (CCC), three chemotherapy day units
(CDU) were involved in the second study and four radiotherapy departments were included in the final study. Per
multiple case study a research protocol was used to structure the benchmarking process. After reviewing the
multiple case studies, the resulting description was used to study the research objectives.
Results: We adapted and evaluated existing benchmarking processes through formalizing stakeholder involvement
and verifying the comparability of the partners. We also devised a framework to structure the indicators to produce
a coherent indicator set and better improvement suggestions. Evaluating the feasibility of benchmarking as a tool
to improve hospital processes led to mixed results. Case study 1 resulted in general recommendations for the
organizations involved. In case study 2, the combination of benchmarking and lean management led in one CDU
to a 24% increase in bed utilization and a 12% increase in productivity. Three radiotherapy departments of case
study 3, were considering implementing the recommendations.
Additionally, success factors, such as a well-defined and small project scope, partner selection based on clear
criteria, stakeholder involvement, simple and well-structured indicators, analysis of both the process and its results
and, adapt the identified better working methods to the own setting, were found.
Conclusions: The improved benchmarking process and the success factors can produce relevant input to improve
the operations management of specialty hospitals.

Background interest in the performance of health services and the

Society is struggling with the challenge of cost contain- practices leading to excellent performance.
ment in health care; costs are expected to grow consid- Research on operations management (OM) studies the
erably, mainly due to population ageing and the production and delivery of products and services [1]. In
introduction of new technologies. Additionally, the order to improve their efficiency, hospitals are introdu-
workforce required to deliver the health care services is cing OM practices, like benchmarking.
showing a relative decline. This has created growing Benchmarking is defined as “the search for- and imple-
mentation of best practices” [2], it originated in the man-
ufacturing industry and is now widely applied in
* Correspondence: [email protected] healthcare. Gift and Mosel provide the following defini-
1
Division of Psychosocial Research and Epidemiology, Netherlands Cancer tion for healthcare: “... benchmarking is the continual
Institute - Antoni van Leeuwenhoek Hospital, PO Box 902031006, BE
Amsterdam, The Netherlands
and collaborative discipline of measuring and comparing
Full list of author information is available at the end of the article the results of key work processes with those of the best
© 2010 van Lent et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative
Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work is properly cited.
van Lent et al. BMC Health Services Research 2010, 10:253 Page 2 of 11
http://www.biomedcentral.com/1472-6963/10/253

performers. It is learning how to adapt these best prac- uncertain whether they are suitable for application to
tices to achieve breakthrough process improvements and hospitals. Hospital services may be described as profes-
build healthier communities” [3]. sional bureaucracies with characteristics like multiple
The literature presents numerous benchmarking pro- stakeholders and possibly conflicting professional and
cesses [4,5]. Spendolini [5] compared 24 benchmarking business objectives. Van Hoorn et al [6] described a
processes and found four common characteristics in all benchmarking process for healthcare, which is illu-
of them, see Figure 1. Most benchmarking processes ori- strated in Figure 1. This process [6] stressed the impor-
ginated in manufacturing industries; therefore it is tance of creating project support and emphasized the

Figure 1 Benchmarking processes compared.

van Lent et al. BMC Health Services Research 2010, 10:253 Page 3 of 11
http://www.biomedcentral.com/1472-6963/10/253

need to assess the comparability of the organizations involving specialty hospitals concentrated on the differ-
and the involvement of stakeholders in the development ences with general hospitals [10] whereas identifying
of indicators. optimal practices, especially regarding operations man-
Health services research (HSR) applied benchmarking agement, was seldom the topic of research.
mainly to identify best practices for national health sys- Because specialty hospitals represent a trend and the
tems and treatments. The WHO World Health Report opinions about the added value are divided, more
[7] concluded that although health status between coun- insight into the benchmarking process in specialty hos-
tries was comparable, healthcare costs differed consider- pitals could be useful to study differences in organiza-
ably. Nevertheless, the “knowledge on the determinants tion and performance and the identification of optimal
of the health system performance, as distinct from under- work procedures.
standing health status, remains very limited.” This con- Benchmarking of operations management in specialty
clusion underlines the possibility in understanding hospitals has not been frequently examined. By the end
international practices as an instrument to improve of 2009, we could find only 23 papers in PubMed about
healthcare performance. International benchmarking operations management in specialty hospitals, 6 of them
helps to explain for instance efficiency differences in concerning cancer centres. About half of the 23 papers
hospitals and it supports hospitals to improve their turned out to be a mismatch with the research topic.
processes. Most of the relevant papers appeared to be non-scientific,
Although international benchmarking on operations mentioned just a few outcomes, and emphasized the
management may improve hospital processes, research experiences of the project members. Only four publica-
on this subject is limited. It seems that so far most tions reported on a competitive benchmark for specialty
attention is given to the comparison of healthcare sys- hospitals, but none described benchmarking in an inter-
tems on a national level and to the development of indi- national setting, nor did they focus on the benchmarking
cators. The importance of indicator development is process or the success factors.
highlighted by Groene et al [8] who found 11 national
indicator development projects in a systematic review. Research questions
This focus on indicators has also been adopted by We conclude that international benchmarking as part of
healthcare agencies, like the National Health Service an approach to improve performance in specialty hospi-
(NHS) in the UK, the Joint Commission on Accreditation tals, has not been the subject of thorough research.
of Healthcare Organizations (JCAHO) in the USA, and Therefore, we address the following research questions:
for-profit service providers. Under the term benchmark-
ing, these organizations use indicators to publish hospital 1. What is the most suitable process for benchmarking
performance rankings, assuming that they foster compe- operations management in international comprehen-
tition and lead to the dissemination of best practices [9]. sive cancer centres or departments (benchmarking
However, most rankings do not provide thorough insight process) to improve hospitals?
into the organizational practices that led to the measured 2. What are the success factors for international
performance although this insight is required to improve benchmarking in comprehensive cancer centres (suc-
healthcare processes, as they are often based on readily cess factors)?
available administrative data sets [9].
We conclude that benchmarking as a tool to improve
operations management in hospitals is not well Methods
described and possibly not well developed. Study design
International benchmarking with the objective to identify
Specialty hospitals and benchmarking OM improvements in specialty hospitals is examined on
In order to become more efficient, healthcare is also the basis of three independent multiple case studies in
showing a trend towards specialization of hospitals (or comprehensive cancer centres. We used multiple case
their units). Schneider et al [10] described specialty hos- studies, because they are suitable for exploratory investi-
pitals as hospitals “that treat patients with specific medi- gations and allow in-depth research. Each multiple case
cal conditions or those in need of specific medical or study consisted of international comprehensive cancer
surgical procedures.” The number of specialty hospitals centres (CCC) or departments within a CCC, as these
is increasing [10-12]. Porter, Herzlinger and Christensen may be representative for specialty hospitals operating in
[13-15] suggested that specialization improves the per- an internationally competitive environment. A compre-
formance, because it results in a better organization of hensive cancer centre means a (partly) tertiary hospital
processes, improved patient satisfaction, more cost- specializing in the treatment of oncology patients, which
effective treatments and better outcomes. Most research is also involved in education and translational research.
van Lent et al. BMC Health Services Research 2010, 10:253 Page 4 of 11
http://www.biomedcentral.com/1472-6963/10/253

Each multiple case study concerned a different hospi- Case study research protocol
tal level: total hospital level, unit level and department To increase the reliability and validity of the case studies,
level. Multiple case study 1 was limited to the compari- the researchers developed a separate research protocol
son of operations management within CCCs. Three for each case study [17]. The protocols described the
CCC’s were included. In study 2 a small project scope selection criteria for the hospitals involved, the bench-
was defined to enable to go through the complete marking process, and the indicators. As case research
benchmarking process, including the translation of more protocols need to be tested [16], we piloted the research
optimal working procedures and the evaluation of the protocol in the initiating hospital. A distinction between
implemented changes. Three chemotherapy day units HSR benchmarking and the approach taken in this paper
(CDUs) were the cases for this study. In study 3 the was that our process focused on gaining insight into the
scope was widened to a department, but the study was organizational aspects, thus creating learning opportu-
limited to the delivery of recommendations to the nities to improve performance. This research did not
involved organizations. This study especially evaluated emphasize the development of extensively validated indi-
the involvement of internal stakeholders and the indica- cators or procedures to validate the comparability of
tor development process. Radiotherapy departments organizations (for example on case mix).
were the cases of this study. In multiple case study 1 the benchmarking process
was based on Spendolini’s benchmarking process [5]
Case selection (see Figure 1 and Table 2). Since this is a general model
The purpose of the case studies was an international that has been based on benchmarking experiences in
comparison with well-known, similar organizations to manufacturing industries, we scrutinized it, and when
identify better working methods in operations manage- necessary adapted it to ensure a comprehensive and
ment in specialty hospitals. The selected cases had to appropriate benchmarking approach. Table 2 describes
match the research objectives [16]. Since scarcely any the benchmarking process used in each case. The
objective data on best practices for OM in (specialty) benchmarking process used in each multiple case study
hospitals were available it was impossible to select differs on details as the lessons learned were integrated
cases based on performance. Therefore convenient in the next multiple case study.
sampling was the most obvious way to obtain mean- In multiple case study 1, we expanded Spendolini’s
ingful results. benchmarking process [5] to include a framework that
Together with the stakeholders of the initiating centre structured the indicators (Table 2, step 6), ensured com-
the researchers developed inclusion criteria to verify to parability and covered all relevant aspects. We selected the
organizational comparability. Table 1 summarizes the EFQM (European Foundation for Quality Management)
three multiple case studies and their inclusion criteria. model because it considered strategic aspects, the pro-
Patient characteristics were not verified in advance, cesses and the outcomes. Another reason is that the
since the mission and strategy of the comprehensive EFQM model and its USA variant, the Malcolm Baldridge
cancer centres suggested a similar case mix. Besides, Quality Award (MBQA), are used in many hospitals [18].
better working methods could also be identified when Additionally, the step involving ‘Collect and Analyse
patient characteristics differ. benchmarking information’ was broken up into four
Management approached potential participants, when- phases: i) develop relevant and comparable indicators, step
ever participants fulfilled the criteria and agreed to par- 7; ii) stakeholders select indicators, step 8; iii) measure the
ticipate, they were included. The organizations involved indicators, step 9; and iv) analyse performance differences,
are presented anonymously in the text. step 10. Finally, we separated the ‘take action’ into two

Table 1 Overview of the case studies

Multiple case study 1 Multiple case study 2 Multiple case study 3
Operations management in Comprehensive Cancer Operations management in Chemotherapy Operations management in Radiotherapy
Centres (CCC) day units (CDU) departments (RT)
Total organization Unit level Department level
Comprehensive cancer centres Part of comprehensive cancer centres Part of comprehensive cancer centres
Only medication related treatments Size: minimum of three linear accelerators
1 with perceived high efficiency, 1 with >50 Strategy
beds
Patient case mix
3 European partners 3; 2 from Europe, 1 USA 4 European partners
van Lent et al. BMC Health Services Research 2010, 10:253 Page 5 of 11
http://www.biomedcentral.com/1472-6963/10/253

Table 2 Benchmarking process employed per multiple case study

Benchmark activity Spendolini Case 1: Case 2: Case 3:
CCC CDU Radiotherapy
1. Determine what to benchmark + + + +
2. Form a benchmarking team + + + +
3. Choose benchmarking partners + + + +
4. Define and verify the main characteristics of the partners - - + +
5. Identify stakeholders - - - +
6. Construct a framework to structure the indicators - + + +
7. Develop relevant and comparable indicators - + + +
8. Stakeholders select indicators - + + +
9. Measure the set of performance indicators - + + +
10. Analyse performance differences + + + +
11. Take action: results were presented in a report and recommendations + + - +
were given
12. Develop improvement plans - - + -
13. Implement the improvement plans - - + -
14. Evaluation of the implementation - - + -
+ = used in benchmarking case study.
- = not used in benchmarking case study.

phases: develop improvement plans (step 12) and imple- collect data for each case. Quantitative data were
ment improvement plans (step 13). retrieved from annual reports and requested from the
Table 2 shows the benchmarking process used in mul- administrative departments, whereas qualitative data
tiple case study 2. Compared to study 1, we added step 4. were mainly collected by conducting semi-structured
In this step we verified the comparability of the partners interviews during the site visits. In the CDU case, we
using the patient case mix (based on the ICD-9 coding also used direct observations to gain a better under-
system and treatment urgency) and the services delivered standing of the processes that led to the results.
by the CDU. To increase the validity of the data, the outcomes of
In multiple case study 3 we additionally used input the indicators were presented to the contact persons of
from a benchmarking process for healthcare developed the relevant comprehensive cancer centres. Most quanti-
by Van Hoorn et al (20), since this became available after tative indicators were collected from databases and were
study 2. This study emphasized the indicator develop- verified with the stakeholders; this process of triangula-
ment process, the involvement of internal stakeholders tion increased the validity of the data [22].
and the comparability of the results. Compared to case
study 2, step 5 - identification of stakeholders - was Data analysis
added. In a stakeholder analysis [19-21] we identified Per multiple case study the data for each indicator were
cancer centre management, radiotherapy department compared. In cases of exceptional outcomes the persons
management, radiation oncologists and clinical physicists who delivered the data were asked to comment on the
as stakeholders. In collaboration with the stakeholders, differences. These explanations helped us to understand
the benchmarking team earmarked ‘commitment’ and differences between the organizations. Besides compar-
‘shared ownership’ for improvement suggestions. ing individual indicators, we took the total indicator set
At the start of each benchmark literature was searched into consideration, because a good score on one indica-
for relevant indicators. Stakeholders of the initiating tor seemed to affect the performance on another indica-
organization provided feedback, resulting in a reduced tor. For example a high utilization rate is related to
list of indicators. Although some only described a situa- longer access delays.
tion or condition, most indicators consisted of a numera- After reviewing each multiple case study the research
tor and a denominator. For example, the number of team examined the feasibility, actual process and success
patients treated per linear accelerator per opening hour. factors of international benchmarking in comprehensive
cancer centres.
Data collection methods
Industrial engineering and management students col- Results
lected the data according to the research protocols. We Below we describe the findings to the research ques-
used both qualitative and quantitative methods to tions. Per question the results of the multiple case
van Lent et al. BMC Health Services Research 2010, 10:253 Page 6 of 11
http://www.biomedcentral.com/1472-6963/10/253

studies are presented. The indicators presented are the ICD-9 coding system, the percentage of urgent
examples of the indicators used to analyse the organiza- patients and the duration of the treatments. The deliv-
tions involved. ered services were examined based on the main techni-
ques used for treatments. The patient case mix and
Question 1: benchmarking process services offered were similar, see Table 4 [23].
Multiple case study 1: comprehensive cancer centres (CCC) In Table 4 the estimated number of patient visits for
The methods section already described the benchmark- 2005 shows that all CDUs were growing rapidly. CDU 3
ing process which is summarized in Table 2. The clearly outperformed the others on the number of
adapted benchmarking process of Spendolini [5] was patients treated per bed and the number of patients
workable, but adjustments might increase the generation treated per nurse or staff member, and provided possibly
of improvement suggestions regarding operations man- more optimal working methods for the planning proce-
agement. Although the CCCs were satisfied with the dure, reduction of non-value adding activities and nur-
results, they commented that the results would not sing staff utilization.
always be applied in change processes because they The benchmark resulted in recommendations for
were uncertain that the same performance could be improving patient planning and work procedures con-
achieved in their setting because the processes might cerning resources (bed, nurses and medication) needed
still not be sufficiently comparable. for a medical procedure. A multidisciplinary team
The selected indicators distinguished between the implemented the recommendations by translating the
total organization level, diagnostics, surgery, medica- lessons learned from CDU3 to CDU1. During this trans-
tion related treatments, radiotherapy and research. The lation process, CDU1 also used lean management prin-
results showed possibilities for improvements. For ciples to obtain even better results. For more details
example, Table 3 shows that the percentages of staffing about this improvement project, see Van Lent et al [23].
costs were comparable, but the percentage of non- This resulted in a 24% growth in the number of patient
medical support staff ranged between 24% (CCC2) and visits, a 12% to 14% increase in staff productivity and an
15% (CCC1). CCC2 could thus learn from CCC1 to 80% reduction of overtime while the average expected
reduce the percentage of support staff. Regarding treatment duration remained stable.
radiotherapy, CCC2 treated 53% more patients per lin- Multiple case study 3: radiotherapy departments (RT)
ear accelerator, CCC3 could learn from CCC2 to We further adapted the benchmarking process based on
improve their performance. Actually embarking on the work of Van Hoorn et al (20), on verifying the com-
related improvement activities would however require parability of hospitals and developing indicators that
further research. achieve consensus among stakeholders. This suggested
Multiple case study 2: chemotherapy day units (CDU) the researchers to examine the role of the stakeholders
Since the CDUs found verification of their comparability and the development of indicators more thoroughly.
useful, e.g. in respect of patient case mix treated and the Just as in study 2, the tumour types of the patients
services delivered, we included this as a new step in the (ICD-codes) were used to verify the comparability of the
benchmarking process (see Table 2, step 4). A self-made involved radiotherapy departments (Table 2, step 4).
instrument was developed to test the comparability Since these data were not available for the departments
involved organizations. The case mix was examined with we checked the comparability of the ICD-codes on a

Table 3 Examples of indicators for the benchmarking of CCC

Indicator Indicator type CCC1 CCC2 CCC3
Percentage staff costs on total costs (in %) Organizational 61 64 58
level
Percentage of supportive staff on total staff in full-time Organizational 15% 24% Not available
equivalents level
Number of hospital beds (admission longer than 24 hours) Department level 174 327 56, only beds for radiotherapy and internal
medicine
Number of surgeries performed per OR Department level 558 741 Not identifiable
Number of patients treated per linear accelerator Department level 490* 480 313
Number of patient visits per CT scanner Department level 7648 9047 Only for a specific treatment for one location
available
Percentage of staff costs on total costs of radiology Department level 44 65 Not available
department
* = number of treatment series, not patients. 1 patient can receive multiple series.
van Lent et al. BMC Health Services Research 2010, 10:253 Page 7 of 11
http://www.biomedcentral.com/1472-6963/10/253

Table 4 Examples of indicators for the benchmarking of CDU, see also [23]
Items compared CDU 1 CDU 2 CDU 3
Patient case mix 23 out of 30 21-27 out of 30 23 out of 30
Services offered 28 out of 36 30 out of 36 28 out of 36
Total patient visits 2004 11.152 80.000 12.371
Estimated total patient visits 2005 in November 12.000 107.000 12.500
Indexed average number of patients treated per bed per month (not corrected for differences in 44 77 100
opening hours)
Indexed average number of patient visits per month per total CDU staff 58 44 100
Indexed average number of patient visits per nurse per month 62 53 100

national level, assuming that the patients of radiotherapy Our analysis revealed that radiotherapy centre 1 (RT1)
centres reflected the national data on the use of interna- seemed to have the most optimal working method for
tional treatment protocols. The comparability was risk analysis, waiting times, patient satisfaction and
acceptable. scientific publications. RT3 and RT4 achieved better
Compared to study 2, step 5 - identification of stake- results regarding the Linear Accelerator utilization.
holders with a stakeholder analysis [19-21] - was added. Although the organizations involved accepted the results
The improved benchmarking process (see Table 2) and recognized the improvement opportunities, they
resulted in better acceptance of the indicators, although wanted more details before implementation because
it proved difficult to obtain all the requested data. they did not have sufficient insight into the underlying
The stakeholder analysis supported was also useful for organizational processes and the coherence between the
the development of the indicators. Just as in the other indicators. A change in respect of one indicator (like a
studies an initial list of indicators was based on relevant reduction of Linear Accelerator downtime) might affect
literature. The stakeholders identified in step 5 provided the performance regarding another one (Linear Accel-
feedback on the relevancy, measurability and compar- erator utilization).
ability of the selected indicators. As a result indicators RT1 started to work on their patient satisfaction and
were removed, adapted and added. After the benchmark risk analysis score and a switch to measuring waiting
the indicator set was evaluated. time per tumour type instead of general waiting times is
Table 5 presents examples of the benchmarked out- being considered. Furthermore, RT1 studied opportu-
comes. For patient satisfaction and risk analysis, we nities to reduce planned downtime during regular work-
measured whether the departments systematically ing hours. RT2 examined its inclusion rate for clinical
applied the plan-do-check-act cycle to achieve improve- trials and the information included in management
ments. None of the organizations performed all phases reports. This should support them in making their sta-
of the cycle; even the most optimal procedure did not tus as a high-quality radiotherapy centre transparent.
keep track of the changes. RT4 has been working on a system to register misses

Table 5 Examples of indicators for the benchmarking of radiotherapy departments

Examples of indicators RT1 RT2 RT3 RT4
Patient satisfaction, stage in PDCA cycle Check-Act Plan-Do Do-Check Plan- Do
Risk analysis, stage in PDCA cycle Do- Check Do-Check Plan-Do Plan-Do
Average impact points per publication and total publications 5.6 out of 297 2.3 out of 55 2.8 out of 68 Not available
Percentage of patients in clinical trials 4.4% 0.7% 10.7% 3.45%
Percentage of treatment planning with: 0 0 0 40
Simulator 91 98 77 56
CT 8 2 7 0
MRI 1 0 17 3
PET
Patients treated per Linear Accelerator per standard opening hour 4.5 4.6 5.4 5.6
Number of hours of downtime for planned maintenance per Linear Accelerator 156 173 47,5 84
van Lent et al. BMC Health Services Research 2010, 10:253 Page 8 of 11
http://www.biomedcentral.com/1472-6963/10/253

(part of the risk analysis) and it used the indicator out- beam). Thus, indicators measuring the percentage of
comes to measure how many investments in staff and patients included in clinical trials did not only reflect
equipment are needed to remain a high-quality radio- the organizational quality of the process, but rather the
therapy centre. availability to scarce resources. This highlighted the
importance of careful partner selection.
Question 2: success factors for international Comparable to multiple case study two, it is impossi-
benchmarking ble to define a single most optimal working method for
Multiple case study 1: CCC a department without considering its operational priori-
International benchmarking of a CCC on operations ties. This should be taken into account whenever the
management is complex. Due to different reimburse- team identifies a learning opportunity.
ment and accounting systems, the use of financial indi- All indicators were measured over a one-year period.
cators was especially complex. Moreover, differences A discussion with the benchmarking partners revealed
between external environments (mainly caused by gov- that some indicators were subject to large year-to-year
ernment regulations) and the organizational choices variations. Examples are the average impact points per
resulted in difficulties with data availability. publication and the number of patients included in a
Furthermore, policy affected the data directly and the clinical trial. Thus, measuring indicators over a one year
organizational structure often determined the proce- period as done in this case study, does not always give a
dures for data collection and aggregation. The adminis- good impression of the performance.
trative organization of CCC3 was not yet capable of
providing data for all activities as an identifiable unit on Discussion
that level of organization because it shared resources Based on our results we present the following answers
with a general hospital. This problem was exacerbated to the research questions.
because the CCC was in the middle of a merger and the
data registration systems were not yet completely inte- 1. What is the most suitable process for benchmarking
grated. As the oncology surgeries could not be identified operations management in international comprehen-
separately, it was impossible to verify the exact numbers. sive cancer centres or departments (benchmarking
This case study used simple indicators, like patient-staff process) to improve hospitals?
ratios or patient-resource ratios that could easily be Figure 1 shows the recommended benchmarking
collected. process based on this study and compares it with
An identifiable unit or department such as radiother- the benchmarking processes of Spendolini [5] and
apy, radiology or a chemotherapy day unit seemed more Van Hoorn et al [6].
suitable for benchmarking as this simplifies data collec- Compared to case study 3 (see Table 2) we have
tion. Radiology departments could be compared if refer- added translation of the improvement opportunities
ring policies are comparable. Specialized surgical to the individual situation as a specification of step
departments seemed difficult to benchmark, due to pro- 12 (develop improvement plans) and the evaluation
blems with data availability, indicator definitions and the of the results and the benchmarking project (step
organizational embedding of the operating theatre. 13). The project team has to establish consensus on
Multiple case study 2: CDU the content of each step.
The small project scope together with the use of inter- The results on the feasibility of benchmarking as a
views and observations resulted in improved insight into tool to improved hospital processes are mixed. Mul-
the organizational principles that delivered the results. tiple case study 1 provided insight into the bench-
The benchmark made the partners aware that other marking process and gave indications for
organizations with similar problems were able to achieve improvement opportunities. For study 2 we pre-
better outcomes. This resulted in useful recommenda- sented evidence of improvements. Although imple-
tions that have been implemented in CDU1. The man- mentation was conducted together with lean
agement of CDU1 reported that the verification of the management (see [23]), the benchmark enabled dis-
comparability had resulted in increased confidence in cussion about the working procedures and pre-
the identified improvement opportunities. vented a reinventing of the wheel because it gave
Multiple case study 3: RT direction to the improvements. Study 3 resulted in
The results revealed that organization-specific character- recommendations that are being considered for
istics influenced the outcomes because some depart- implementation. Altogether our conclusion confirms
ments, like radiotherapy, are quite dependent on the work of De Korne et al [24] who concluded
technology (for example, most clinical trials require Lin- after an international benchmarking initiative of eye
ear Accelerators with special functions, like a cone hospitals that it is possible but “not so easy to
van Lent et al. BMC Health Services Research 2010, 10:253 Page 9 of 11
http://www.biomedcentral.com/1472-6963/10/253

compare performance in an international setting, Often information on hospitals is collected on a

especially if the goal is to quantify performance gaps national level. Thus, specialty hospitals with few com-
or to identify best practices.” petitors in their own country will more frequently
2. What are critical success factors for international encounter a lack of data on comparable organiza-
benchmarking in comprehensive cancer centres? tions. In these cases, possible partners should be con-
Table 6 summarizes the success factors and relates tacted and screened on the inclusion criteria.
them to the steps described in the proposed bench- c. Stakeholder involvement is crucial for obtaining
marking process in Figure 1. consensus about the indicators, providing informa-
a. Before embarking on benchmarking to improve tion on data availability and reliability, and assisting
hospital (unit) performance, three additional condi- in data collection.
tions should be met. First, the internal stakeholders d. Both quantitative and qualitative data should be
need to be convinced that others have developed used to determine the performance and construct a
solutions for problems that can be translated to their format that enables a structured comparison, such as
own setting (as in the CDU case), otherwise they the EFQM model.
might frustrate the project, by either questioning the e. Do not spend too much time on the reliability of
validity of the outcomes or withholding implementa- the indicators, keep them simple so that enough
tion. Second, management must reserve sufficient time can be spent on the analysis of the underlying
resources (time and money) to execute the total pro- processes because this leads to improvement sugges-
cess, including the development of improvement tions [4,24]. Qualitative indicators, observations and
plans. Third, a manageable-sized project scope is interviews should be used for this purpose. This is
required for a thorough analysis of the selected pro- especially important in an international benchmark
cess and its results. The scope should be limited to a with a score for differences in and between organiza-
well-defined problem, for example capacity utiliza- tions. A comparison of financial data has to be
tion. This can be found in an identifiable department undertaken with care, especially in an international
or unit, but it can also be a small process that benchmark with different price levels and national
involves several departments. reimbursement systems per country. Comparative
b. The initiating organization should define criteria purchasing power enables cost comparison [25];
to verify the comparability of the benchmarking however this is time consuming, whereas non-finan-
partners based on the subject and the process. Cases cial indicators sometimes achieve a good comparison
two and three showed the usefulness of the criteria within a shorter time frame.
for creating support for the benchmark and prevent- f. For those indicators showing a large annual varia-
ing comparability difficulties that were related to tion in outcomes, measurement over a number of
organization-specific characteristics. De Korne et al years should be considered.
[24] concluded that a comparison with peers pro- g. Adapt the identified better working methods, so
vided an incentive to professionals to change current that they comply with other practices in the
practices. organization.

Table 6 Success factors for the proposed benchmarking process

Success factors for international benchmarking on operations management Step in Figure 1
Internal stakeholders should be convinced that others might have developed solutions for problems that can be translated to Step 1
their own setting.
Management must reserve sufficient resources for the total benchmark. Step 1
Limit the scope to a well-defined problem. Step 1
Define criteria to verify the comparability of the benchmarking partners based on the subject and the process. Step 4
Construct a format that enables a structured comparison. Step 6
Use both quantitative and qualitative data for measurement. Step 7
Involve stakeholders to obtain consensus about the indicators, to provide information on data availability and reliability, and to Step 5,6,7,8,9
assist in data collection.
Keep indicators simple so that enough time can be spent on the analysis of the underlying processes. Step 7,8,9,10
For those indicators showing a large annual variation in outcomes, measurement over a number of years should be considered. Step 7,9,10
Adapt the identified better working methods, so that they comply with other practices in the organization. Step 11
van Lent et al. BMC Health Services Research 2010, 10:253 Page 10 of 11
http://www.biomedcentral.com/1472-6963/10/253

To our knowledge, this is the first attempt that exam- methods. As data availability and comparability seems
ined international benchmarking on operations manage- more frequently a problem in an international context,
ment in (speciality) hospitals. The approach we followed we recommend the use of international benchmarking
made it possible to improve the structure of interna- only if comparable organizations are not available within
tional benchmarking processes. This process in combi- the same country.
nation with the provided success factors may increase Although Gift and Mosel [3] stated that benchmarking
the chance that benchmarking results in improved is a continuous process, the cases were only bench-
operations management performance in specialty hospi- marked once. Recurrent measuring seems only useful if
tals like comprehensive cancer centres. different outcomes can be expected within short time
A limitation is that our benchmarking process was frames, and the partners are ready for a long-term
only tested in three multiple case studies involving three commitment.
to four cases. Involving larger series could be useful to
further improve the validity of the benchmarking pro- Conclusions
cess. Furthermore, our multiple case studies were lim- This study generated more insight into the process of
ited to cancer centres, but we presume that the international benchmarking as a tool to improve opera-
benchmarking process is valid for other multidisciplin- tions management in specialty hospitals. All multiple
ary specialty hospitals. Single specialty hospitals might case studies provided areas for improvement and multi-
be easier to compare. Further research is required to ple case study 2 presented the results of a successful
confirm this. As the benchmarking process seems more improvement project based on international benchmark-
time consuming in an international setting as system ing. The provided method and the success factors can
differences add to the complexity, we suggest that the be used in international benchmarking projects on
described process is useful for benchmarking in a operations management in speciality hospitals.
national or regional setting provided the objective is to
identify relevant operations aspects into sufficient depth.
Acknowledgements
To our knowledge there is no accepted guideline or We acknowledge all participating staff and the management of the
norm describing a complete indicator set for comparing participating cancer centres. We would like to acknowledge M. Verheij for
the operations management performance in hospitals or his support on the radiotherapy benchmark. The authors acknowledge the
helpful suggestions from the reviewers.
hospital departments. Per multiple case study we
defined an initial list of indicators, based on relevant lit- Author details
1
erature and stakeholder feedback. Stakeholders provided Division of Psychosocial Research and Epidemiology, Netherlands Cancer
Institute - Antoni van Leeuwenhoek Hospital, PO Box 902031006, BE
feedback on the relevancy, measurability and compar- Amsterdam, The Netherlands. 2Ministry of Health, Welfare and Sport, The
ability. As a result indicators were removed, adapted Hague, The Netherlands. 3Department of Health Technology Services,
and sometimes added. A limitation of this approach is Research School of Management and Governance, University of Twente,
Enschede, The Netherlands.
that more emphasis could be laid on the methodological
quality of the indicators. However, combining the Authors’ contributions
benchmarking process with a thorough and detailed WvL developed the research design and carried out case studies 1 and 2.
She also drafted the manuscript. RdB improved the research design for case
process of indicator development could further improve study 3 and conducted this case study. Furthermore, she contributed to
the benchmarking, but will prove to be complex and revision of the manuscript. WvH was involved in the research design of all
demanding. In this way generic indicator sets on opera- case studies, supervised them and contributed to the intellectual content of
the manuscript. All authors read and approved the final manuscript.
tions management could become available.
The developed indicator sets enabled the assessment Competing interests
of the operations management of specialty hospitals and Wim H van Harten is a member of the executive board of the Netherlands
Cancer Institute - Antoni van Leeuwenhoek Hospital. Wineke AM van Lent is
generated suggestions for improvement. Collecting also employed by this organization as a PhD-student and advisor on patient
and interpreting data, however, has to be done carefully logistics.
and must be based on the total indicator set as there is
Received: 2 November 2009 Accepted: 31 August 2010
not (yet) one single best method to organize processes. Published: 31 August 2010
For example a good performance by one indicator (utili-
zation rate) is often associated with a negative effect on References
another indicator (long waiting times). 1. Slack N, Chambers S, Johnston R: Operations Management Harlow: Prentice
Hall 2001.
A limitation of the sampling method is that it remains 2. Camp RC: Business Process Benchmarking: Finding and Implementing Best
uncertain whether the best practices within the sector Practices Wisconsin: Milwaukee 1995.
have actually been included. Because information on 3. Mosel D, Gift B: Collaborative benchmarking in health care. Jt Comm J
Qual Improv 1994, 20:239.
best practices was not available, we used personal man- 4. Camp RC: A Bible for Benchmarking, by Xerox. Fin Exc 1993, 9:23.
agement contacts to select presumed good working 5. Spendolini Michael J: The Benchmarking Book New York: Amacom 1992.
van Lent et al. BMC Health Services Research 2010, 10:253 Page 11 of 11
http://www.biomedcentral.com/1472-6963/10/253

6. Van Hoorn A, Van Houdenhoven M, Wullink G: Een nieuw stappenplan

voor benchmarking. Management Executive 2006, 1-15.
7. World Health Organization: The World Health Report 2000. Health Systems:
Improving Performance WHO, Geneva 2000.
8. Groene O, Skau JKH, Frolich A: An international review of projects on
hospital performance assessment. International Journal for Quality in
Health Care 2008, 20:162.
9. Powell AE, Davies HTO, Thomson RG: Using routine comparative data to
assess the quality of health care: understanding and avoiding common
pitfalls. Qual Saf Health Care 2003, 12:122.
10. Schneider JE, Miller TR, Ohsfeldt RL, Morrisey MA, Zelner BA, Li P: The
Economics of Specialty Hospitals. Med Care Res Rev 2008, 65(5):531.
11. United States Government Accountability Office [GAO]: Specialty Hospitals:
Geographic Location, Services Provided, and Financial Performance., GAO-04-
167 Washington, DC: GAO 2003 [http://www.gao.gov/new.items/d04167.
pdf], last accessed: 16/Sept./2009.
12. United States Government Accountability Office [GAO]: Specialty Hospitals:
Information on Potential Facilities, GAO-05-647R Washington, DC: GAO 2005
[http://www.gao.gov/new.items/d05647r.pdf], last accessed: 16/Sept./2009.
13. Porter ME, Teisberg EO: Redefining health care: creating value-based
competition on results Boston, Massachusetts: Harvard Business School Press
2006.
14. Christensen CM, Grossman JH, Hwang J: The Innovator’s Prescription. A
Disruptive Solution for Health Care United States of America: McGraw-Hill
2009.
15. Herzlinger RE: Market-Driven Health Care: Who Wins, Who Loses in the
Transformation of America’s Largest Service Industry Perseus Books Group
1999.
16. Karlsson C: Researching Operations Management Routledge 2009.
17. Yin R: Case study research: Design and methods Sage Pubns 2008.
18. Minkman M, Ahaus K, Huijsman R: Performance improvement based on
integrated quality management models: what evidence do we have? A
systematic literature review. Int J Qual Health Care 2007, 19:90.
19. Mitchell RK, Agle BR, Wood DJ: Toward a Theory of Stakeholder
Identification and Salience: Defining the Principle of Who and What
Really Counts. Acad Manage Rev 1997, 22:853-886.
20. Brugha R, Varvasovszky Z: Stakeholder analysis: a review. Health Policy Plan
2000, 15:239-246.
21. Carignani V: Management of change in health care organisations and
human resource role: Socio-economics in radiology. Eur J Radiol 2000,
33:8-13.
22. McCutcheon DM, Meredith JR: Conducting case study research in
operations management. Journal of Operations Management 1993,
11:239-256.
23. Van Lent W, Goedbloed N, Van Harten W: Improving the efficiency of a
chemotherapy day unit: Applying a business approach to oncology. Eur
J Cancer 2009, 45(5):800-806.
24. de Korne DF, Sol KJCA, van Wijngaarden JDH, van Vliet EJ, Custers T,
Cubbon M, et al: Evaluation of an international benchmarking initiative
in nine eye hospitals. Health Care Management Review 2010, 35(1):23-35.
25. Van Harten WH, Van Luenen HG, Van Bokhorst L: Benchmarking Biology
Research Organisations Using a New, Dedicated Tool. Molecular Oncology
2010, 4(1):12-18.

Pre-publication history
The pre-publication history for this paper can be accessed here:
http://www.biomedcentral.com/1472-6963/10/253/prepub

doi:10.1186/1472-6963-10-253 Submit your next manuscript to BioMed Central

Cite this article as: van Lent et al.: International benchmarking of
specialty hospitals. A series of case studies on comprehensive cancer
and take full advantage of:
centres. BMC Health Services Research 2010 10:253.
• Convenient online submission
• Thorough peer review
• No space constraints or color figure charges
• Immediate publication on acceptance
• Inclusion in PubMed, CAS, Scopus and Google Scholar
• Research which is freely available for redistribution

Submit your manuscript at

www.biomedcentral.com/submit