Ep09a3 Sample
Ep09a3 Sample
EP09-A3
Measurement Procedure Comparison and
E
Bias Estimation Using Patient Samples;
Approved Guideline—Third Edition
PL
M
SA
A guideline for global application developed through the Clinical and Laboratory Standards Institute consensus process.
Clinical and Laboratory Standards Institute
Setting the standard for quality in clinical laboratory testing around the world.
The Clinical and Laboratory Standards Institute (CLSI) is a not-for-profit membership organization that brings
together the varied perspectives and expertise of the worldwide laboratory community for the advancement of
a common cause: to foster excellence in laboratory medicine by developing and implementing clinical laboratory
standards and guidelines that help laboratories fulfill their responsibilities with efficiency, effectiveness, and
global applicability.
Consensus Process
Consensus—the substantial agreement by materially affected, competent, and interested parties—is core to the
development of all CLSI documents. It does not always connote unanimous agreement, but does mean that the
participants in the development of a consensus document have considered and resolved all relevant objections
E
and accept the resulting agreement.
Commenting on Documents
CLSI documents undergo periodic evaluation and modification to keep pace with advancements in technologies,
PL
procedures, methods, and protocols affecting the laboratory or health care.
CLSI’s consensus process depends on experts who volunteer to serve as contributing authors and/or as
participants in the reviewing and commenting process. At the end of each comment period, the committee that
developed the document is obligated to review all comments, respond in writing to all substantive comments,
and revise the draft document as appropriate.
Comments on published CLSI documents are equally essential, and may be submitted by anyone, at any time, on
M
any document. All comments are addressed according to the consensus process by a committee of experts.
Appeals Process
If it is believed that an objection has not been adequately addressed, the process for appeals is documented in
the CLSI Standards Development Policies and Process document.
SA
All comments and responses submitted on draft and published documents are retained on file at CLSI and are
available upon request.
Get Involved—Volunteer!
Do you use CLSI documents in your workplace? Do you see room for improvement? Would you like to get
involved in the revision process? Or maybe you see a need to develop a new document for an emerging
technology? CLSI wants to hear from you. We are always looking for volunteers. By donating your time and
talents to improve the standards that affect your own work, you will play an active role in improving public
health across the globe.
Volume 33 Number 11
Jeffrey R. Budd, PhD
A. Paul Durham, MA
E
Thomas E. Gwise, PhD
Beimar Iriarte, MS
Anders Kallner, MD, PhD
Kristian Linnet, MD, PhD
Robert Magari, PhD
Jeffrey E. Vaks, PhD
Abstract
PL
Clinical and Laboratory Standards Institute document EP09-A3—Measurement Procedure Comparison and Bias Estimation
Using Patient Samples; Approved Guideline—Third Edition is written for laboratorians and manufacturers. It describes
procedures for determining the bias between two measurement procedures, and it identifies factors for consideration when
M
designing and analyzing a measurement procedure comparison experiment using split patient samples. An overview of the
measurement procedure comparison experiment includes considerations for both manufacturers and laboratorians. Details on how
to create difference and scatter plots for visual inspection of the data are provided. Once the data are characterized, various
methods are introduced for quantifying the relationship between two measurement procedures, including bias estimates and
regression techniques. The final section contains recommendations for manufacturers’ evaluation of bias and statement format for
bias claims.
SA
Clinical and Laboratory Standards Institute (CLSI). Measurement Procedure Comparison and Bias Estimation Using Patient
Samples; Approved Guideline—Third Edition. CLSI document EP09-A3 (ISBN 1-56238-887-8 [Print]; ISBN 1-56238-888-6
[Electronic]). Clinical and Laboratory Standards Institute, 950 West Valley Road, Suite 2500, Wayne, Pennsylvania 19087 USA,
2013.
The Clinical and Laboratory Standards Institute consensus process, which is the mechanism for moving a document through
two or more levels of review by the health care community, is an ongoing process. Users should expect revised editions of any
given document. Because rapid changes in technology may affect the procedures, methods, and protocols in a standard or
guideline, users should replace outdated editions with the current editions of CLSI documents. Current editions are listed in
the CLSI catalog and posted on our website at www.clsi.org. If your organization is not a member and would like to become
one, and to request a copy of the catalog, contact us at: Telephone: 610.688.0100; Fax: 610.688.0700; E-Mail:
[email protected]; Website: www.clsi.org.
Number 11 EP09-A3
Copyright ©2013 Clinical and Laboratory Standards Institute. Except as stated below, any reproduction of
content from a CLSI copyrighted standard, guideline, companion product, or other material requires
express written consent from CLSI. All rights reserved. Interested parties may send permission requests to
[email protected].
CLSI hereby grants permission to each individual member or purchaser to make a single reproduction of
this publication for use in its laboratory procedure manual at a single site. To request permission to use
this publication in any other manner, e-mail [email protected].
E
Suggested Citation
CLSI. Measurement Procedure Comparison and Bias Estimation Using Patient Samples; Approved
Proposed Guideline
January 1986
Tentative Guideline
April 1993
PL
Guideline—Third Edition. CLSI document EP09-A3. Wayne, PA: Clinical and Laboratory Standards
Institute; 2013.
M
Approved Guideline
December 1995
ii
Volume 33 EP09-A3
Contents
Abstract ....................................................................................................................................................i
Committee Membership........................................................................................................................ iii
Foreword .............................................................................................................................................. vii
1 Scope .......................................................................................................................................... 1
2 Introduction ................................................................................................................................ 1
2.1 Overview of the Measurement Procedure Comparison Study ...................................... 2
2.2 Primary Purposes for Measurement Procedure Comparisons ....................................... 2
3 Standard Precautions .................................................................................................................. 4
4 Terminology............................................................................................................................... 4
E
4.1 A Note on Terminology ................................................................................................ 4
4.2 Definitions .................................................................................................................... 5
4.3 Symbols Used in the Text ............................................................................................. 8
4.4 Abbreviations and Acronyms ....................................................................................... 8
5
6
6.2
6.3
6.4
6.5
PL
Measurement Procedure–Familiarization Period ....................................................................... 9
Measurement Procedure Comparison Studies ........................................................................... 9
6.1 Study Samples............................................................................................................. 10
Comparative Measurement Procedure ........................................................................ 11
Number of Samples .................................................................................................... 12
Factors Affecting the Measurement Procedure Comparison ...................................... 12
Sample Sequence ........................................................................................................ 14
M
6.6 Time and Duration ...................................................................................................... 14
6.7 Inspection of Data During Collection ......................................................................... 14
6.8 Quality Control ........................................................................................................... 15
6.9 Documentation of Rejected Data ................................................................................ 15
7 Considerations for Clinical Laboratories ................................................................................. 15
SA
v
Number 11 EP09-A3
Contents (Continued)
11.1 Manufacturer’s Statement of Bias Performance Claims ............................................. 37
11.2 Laboratory’s Statement of Bias Performance ............................................................. 38
References ............................................................................................................................................. 39
Appendix A. Confidence Interval of a Median Estimate of Bias Between Measurement
Procedures ............................................................................................................................................. 41
Appendix B. Detecting Aberrant Results (Outliers) ............................................................................. 45
Appendix C. Ordinary Linear Regression............................................................................................. 48
Appendix D. Weighted Least Squares Regression (Weighted Ordinary Linear Regression) ............... 50
Appendix E. Deming Regression .......................................................................................................... 56
E
Appendix F. Constant CV (Weighted) Deming Regression ................................................................. 60
Appendix G. Passing-Bablok Regression ............................................................................................. 62
Appendix H. Jackknife Approach for Estimating Standard Errors for Bias and Regression
Parameters ............................................................................................................................................. 65
PL
Appendix I. A Practical Example Illustrating Bias Estimation and Measurement Procedure
Comparison Techniques........................................................................................................................ 67
Appendix J. Example Datasets.............................................................................................................. 74
The Quality Management System Approach ........................................................................................ 78
Related CLSI Reference Materials ....................................................................................................... 79
M
SA
vi
Volume 33 EP09-A3
Foreword
Measurement procedure comparison is one of the most common techniques used by both manufacturers
and clinical laboratorians to estimate the bias of an in vitro diagnostic (IVD) measurement procedure
relative to a comparator. It involves the comparison of results from patient samples from two
measurement procedures intended to measure the same component (eg, concentration of a measurand)
with the key determination being the estimate of bias between them.
A number of different scenarios exist in which measurement procedure comparison studies are indicated.
For both the manufacturer and the clinical laboratorian, the ideal scenario is the comparison of a
candidate measurement procedure to a generally accepted standard or reference measurement procedure.
In the case of a manufacturer, this involves the establishment and perhaps verification of performance
claims for bias, while in the case of a laboratorian, it involves introducing a measurement procedure into
the laboratory, including verification of such manufacturer claims (specifications). The scope of the
experimental and data-handling procedures for these two purposes will differ. In either case the
E
assumption that the reference measurement procedure provides “true” values means that bias (systematic
measurement error) is estimated.
Quite commonly, however, there is no standard or reference measurement procedure. The manufacturer
PL
instead compares a candidate measurement procedure to the best measurement procedure currently
available. The laboratorian usually compares the candidate and an available procedure. Then, there may
not be a “true” value and the “difference,” rather than the “bias,” is estimated.
Given the variety of performance characteristics of IVD measurement procedures, a single experimental
design is not appropriate for all types of laboratorian and manufacturer measurement procedure
comparisons. Therefore, performance characteristics such as measuring interval and precision profile are
taken into account in structuring an experiment for comparing two measurement procedures. Multiple
worked examples are presented.
M
This document is intended to promote effective and correct data analysis and reporting using standard
experimental and statistical methods.
It is recommended that manufacturers of clinical laboratory measurement procedures and/or devices use
this document to establish and standardize their bias performance claims. Many different forms have been
SA
used for such claims, and they have not always been sufficiently specific to allow user verification.
A number of changes and additions are included in this revision of the document, including:
More reasons for comparisons based on patient samples (factor comparisons [eg, sample tube
types])
vii
Number 11 EP09-A3
Key Words
Alternative regression methods, bias, evaluation protocol, experimental design, linear regression,
measurement procedure comparison, outliers, quality control, residuals
E
PL
M
SA
viii
Volume 33 EP09-A3
1 Scope
This document provides guidance for designing an experiment and selecting methods to quantify
systematic measurement error (bias or difference) between measurement procedures based on comparing
patient samples. It provides procedures to determine the average bias between two measurement
procedures either across their measuring intervals or at selected concentrations. Intended users of this
guideline are manufacturers of in vitro diagnostic (IVD) reagents—which includes those who create
laboratory-developed tests—as well as regulatory bodies and clinical laboratory personnel.
This document is for use with measurement procedures that provide quantitative numerical results. This
E
document is not intended for use with ordinal IVD measurement procedures, commonly referred to as
qualitative procedures (see CLSI document EP121). This document is not intended to address evaluation
of random error (see CLSI documents EP052 and EP153) or to determine the total error inherent in a
comparison of measurement procedures (see CLSI document EP214). It is not intended to measure the
variability of multiple replicates collected during the measurement of a sample, nor is it intended to
2 Introduction
PL
measure the bias of individual measurements such as those resulting from sample interference (as covered
The purpose of this document is to establish good practices at measuring average bias over the measuring
interval in a population of patient samples, relative to a comparative or reference method. Difference plots
are used to visually portray the relationship between measurement procedures to evaluate if the
M
relationship is consistent with a constant difference or as a constant percentage difference (constant CV)
over the measuring interval. The plots are also used to determine the bias estimate from such plots
through either an average or a median. Given the knowledge gained from the difference plots, users are
provided with regression fit options to characterize bias in terms of slope and intercept and bias estimates
at selected concentrations.
SA
This document describes multiple situations in which measurement procedures are compared, each of
which has its own experimental requirements. These requirements dictate differences in the number of
factors to incorporate into the experimental design, the number of samples, and the number of replicates
for each sample. The situations covered in this document assume a study is comparing two procedures
that measure the same quantity by using measurement procedure results from study samples.
In selecting an analysis technique for a set of data, a stepwise process is described that starts with visual
data inspection using difference and scatter plots. The data from difference plots can then be used to
estimate the bias (or percent bias) between measurement procedures. Clinical laboratorians may require
no further analysis. The document continues, however, by describing various regression techniques and
their underlying assumptions that help determine which one should be used in each situation. Such
techniques can, in many cases, provide more robust estimates of bias, so clinical laboratories may wish to
use them. Manufacturers will use them in almost all cases. The goal throughout the document is to
propose a set of techniques for determining bias between measurement procedures and to detail the
strengths and weaknesses of these techniques for given situations.
A brief description of measurement procedure comparison scenarios is provided in the following sections.
Section 2.1 is a general overview common to all scenarios. Sections 2.2.1 and 2.2.2 are intended for
©
Clinical and Laboratory Standards Institute. All rights reserved. 1
Number 11 EP09-A3
manufacturers or research laboratories that have created a candidate measurement procedure. Section
2.2.3 is intended for the typical clinical laboratorian.
The purpose of a measurement procedure comparison study must first be determined, along with the two
measurement procedures to be used in the study. The three primary purposes for such a study are
introduced in Section 2.2. Other purposes for such studies are described in Section 10.
For any well-conducted study, the personnel performing the measurements must be familiar with the
instrument systems used in the study. The familiarization period is described in Section 5.
There are numerous considerations for conducting any measurement procedure comparison study.
General considerations, including sample selection and handling, are covered in Section 6, while those
specific only for studies by clinical laboratories are covered in Section 7.
E
Once the data are collected for the study, they must be reviewed to determine if the goals of data
collection have been met and to characterize the data for interval and distribution of measurements, and
other factors that can dictate what techniques are used for data analysis. Visual data inspection techniques
PL
are suggested in Section 8 for this purpose.
The outcome of a measurement procedure comparison study is a quantification of the bias between two
such procedures. This bias can be expressed as an average bias over the measured interval or a bias at a
selected concentration. The techniques that can be used to supply such bias estimates are described in
Section 9.
Finally, Section 11 discusses the steps for comparing estimated bias to acceptance criteria and for stating
performance claims.
M
Throughout this document, the terms in Table 1 are used to describe the measurement procedures to be
compared.
Manufacturers must establish the relationship of any candidate measurement procedure of measurand
quantification with a comparative measurement procedure, ideally a reference measurement procedure.
Typically, when such a reference measurement procedure is available, the desired result of the
comparison is no significant bias between them. However, often a new (candidate) measurement
procedure is developed as an improvement over a comparative measurement procedure (eg, an automated
in vitro diagnostic (IVD) procedure to replace a microplate procedure). In such situations the primary goal
is to establish the bias between them. It is recommended that at least 100 patient samples with measurand
values spanning the common measuring interval of the two measurement procedures be used for
establishment of bias claims. Influential factors can be included in the experimental design such as
©
2 Clinical and Laboratory Standards Institute. All rights reserved.
Number 11 EP09-A3
EP09-A3 addresses the QSE indicated by an “X.” For a description of the other documents listed in the grid, please
refer to the Related CLSI Reference Materials section, beginning on the following page.
E Event Management
Customer Focus
Nonconforming
Documents and
Purchasing and
Improvement
Facilities and
Organization
Management
Management
Assessments
Information
Equipment
Personnel
Continual
Inventory
Records
Process
Safety
PL X
EP28
EP31
EP05
EP06
EP07
EP12
EP14
EP15
M
EP17
EP21
I/LA21 I/LA21 I/LA21 I/LA21 I/LA21 I/LA21 I/LA21 I/LA21
I/LA28
M29
Path of Workflow
SA
A path of workflow is the description of the necessary processes to deliver the particular product or service that the
organization or entity provides. A laboratory path of workflow consists of the sequential processes: preexamination,
examination, and postexamination and their respective sequential subprocesses. All laboratories follow these
processes to deliver the laboratory’s services, namely quality laboratory information.
EP09-A3 does not address any of the clinical laboratory path of workflow steps. For a description of the documents
listed in the grid, please refer to the Related CLSI Reference Materials section, beginning on the following page.
Results reporting
Sample transport
and archiving
Interpretation
Examination
Examination
follow-up
ordering
Sample
©
78 Clinical and Laboratory Standards Institute. All rights reserved.
Volume 33 EP09-A3
EP07-A2 Interference Testing in Clinical Chemistry; Approved Guideline—Second Edition (2005). This document
provides background information, guidance, and experimental procedures for investigating, identifying, and
characterizing the effects of interfering substances on clinical chemistry test results.
E
EP12-A2 User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline—Second Edition
(2008). This document provides a consistent approach for protocol design and data analysis when evaluating
qualitative diagnostic tests. Guidance is provided for both precision and method-comparison studies.
EP14-A2 Evaluation of Matrix Effects; Approved Guideline—Second Edition (2005). This document provides
guidance for evaluating the bias in analyte measurements that is due to the sample matrix (physiological or
EP15-A2
EP17-A2
PL
artificial) when two measurement procedures are compared.
User Verification of Performance for Precision and Trueness; Approved Guideline—Second Edition
(2006). This document describes the demonstration of method precision and trueness for clinical laboratory
quantitative methods utilizing a protocol designed to be completed within five working days or less.
EP28-A3c Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory; Approved
Guideline—Third Edition (2010). This document contains guidelines for determining reference values and
reference intervals for quantitative clinical laboratory tests. A CLSI-IFCC joint project.
EP31-A-IR Verification of Comparability of Patient Results Within One Health Care System; Approved Guideline
(Interim Revision) (2012). This document provides guidance on how to verify comparability of quantitative
laboratory results for individual patients within a health care system. A CLSI-IFCC joint project.
I/LA21-A2 Clinical Evaluation of Immunoassays; Approved Guideline—Second Edition (2008). This document
addresses the need for clinical evaluation of new immunoassays and new applications of existing assays, as
well as multiple assay formats and their uses. As a guide to designing and executing a clinical evaluation, this
document will aid developers of “in-house” assays for institutional use, developers of assays used for
monitoring pharmacologic effects of new drugs or biologics, and clinical and regulatory personnel responsible
for commercializing products.
CLSI documents are continually reviewed and revised through the CLSI consensus process; therefore, readers should refer to
the most current editions.
©
Clinical and Laboratory Standards Institute. All rights reserved. 79
Number 11 EP09-A3
M29-A3 Protection of Laboratory Workers From Occupationally Acquired Infections; Approved Guideline—
Third Edition (2005). Based on US regulations, this document provides guidance on the risk of transmission
of infectious agents by aerosols, droplets, blood, and body substances in a laboratory setting; specific
precautions for preventing the laboratory transmission of microbial infection from laboratory instruments and
materials; and recommendations for the management of exposure to infectious agents.
E
PL
M
SA
©
80 Clinical and Laboratory Standards Institute. All rights reserved.
Explore the Latest
Offerings from CLSI!
As we continue to set the global standard
for quality in laboratory testing, we’re
adding initiatives to bring even more
value to our members and customers.
E
PL Power Forward
with this Official
Interactive Guide
Fundamentals for implementing
a quality management system
in the clinical laboratory.
Individuals
Introducing CLSI’s New Individual Membership! CLSI is offering a new membership opportunity for individuals
who support or volunteer for CLSI but whose organizations are not currently members.
Full Member ($250)—Professionals associated with the health care profession and/or clinical and laboratory services
Benefits include:
Participation on document development committees
Voting on all documents (concurrent with delegate voting)
Participation in governance activities (vote for the Board of Directors, be nominated for the Board, and
be eligible to be selected for Board committee service)
Discount on educational products
A 25% discount on products and services
Effective January 1, 2013, all CLSI volunteers are required to be members at one of the above levels if their organization is not a
CLSI member. For current volunteers (those who are still actively on committees as of January 1), we have waived the requirement
of membership until the end of their current volunteer assignment, and they may continue participating without incurring any
membership fees. Please feel free to contact CLSI’s Membership department with any questions at [email protected].
Introducing CLSI’s
About CLSI New Membership
Opportunities
M
The Clinical and Laboratory Standards Institute (CLSI) is a not-for-profit
membership organization that brings together the varied perspectives and
expertise of the worldwide laboratory community for the advancement
of a common cause: to foster excellence in laboratory medicine by
950 West Valley Road, Suite 2500, Wayne, PA 19087 USA
More Options. More
developing and implementing clinical standards and guidelines that help Benefits. More Value.
P: 610.688.0100 Toll Free (US): 877.447.1888
laboratories fulfill their responsibilities with efficiency, effectiveness, and
F: 610.688.0700 E: [email protected]
global applicability.
We’ve made it even easier for your organization to take
full advantage of the standards resources and networking
opportunities available through membership with CLSI.
e CLIPSE
TM
950 West Valley Road, Suite 2500, Wayne, PA 19087 USA PRINT ISBN 1-56238-887-8
P: 610.688.0100 Toll Free (US): 877.447.1888 F: 610.688.0700 ELECTRONIC ISBN 1-56238-888-6
E: [email protected] www.clsi.org