Chapter 3:

Method Evaluation and Quality Control

By
By Michael
Michael W.
W. Rogers,
Rogers, Cindi
Cindi Bullock
Bullock Letsos,
Letsos, Matthew
Matthew
P.
P. A.
A. Henderson,
Henderson, Monte
Monte S.
S. Willis,
Willis, and
and Christoper
Christoper R.
R.
McCudden
McCudden
Quality Control

• Now that we’ve talked about method validation…

– how does instrument give a numerical
value/concentration?
– so far, we have taken for granted… calibration

• How does calibration work?

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Calibration
– a set of known concentrations
(calibrators = assayed)
– value set by the manufacturer through a definitive or
reference method
– some instruments/tests are calibrated frequently
while others don’t need to be

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• Calibration
– calibration done with:
• new lots of reagents (chemistry)
• troubleshooting tests that fail QC
• on a set schedule (ex: 14/28 days)
• Calibrators
– prepared differently than controls
– controls must therefore not be used as calibrators
– calibrators may be used as controls in some
situations

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• Calibrators
– value assigned by manufacturer is programmed into
analyzer
– used for comparison when measuring unknown
samples (patients)
– standard curve is created from the calibrators tested
– the number of points needed (ie. # of calibrators) for
each analyte, the concentrations of each calibrator
and the number of times the calibrator is run, set by
the manufacturer
– Passes or Fails

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
Calibrator 1

Calibrator 2

Calibrator 3

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• After instrument calibration and method validation

which…
Ensures tests meets or exceeds acceptable criteria and
allowable error

• Must continually ensure valid results

• Process known as Quality Control (QC)

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Quality control acts to reduce error

• Verifies the stability and accuracy of testing system

• Adhere to regulatory requirements

• Benchmark for possible improvements to analytical

methods and instruments

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Basic Concepts

• Descriptive Statistics: Measures of Center, Spread, and

Shape
– Measures of Center
• Mean: average
• Median: middle point
• Mode: most frequently occurring value
– Spread: how data are distributed
– Shape: distribution
• Gaussian: mean, median, and mode are identical;
distribution is symmetrical (bell curve)

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Performance monitoring

• Accomplished by running stable control materials

• Comparing to expected values

• Upper and lower limits known as control limits

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Chemistry analyzers can run dozens of different tests

• If control are acceptable then we can be confident the

system is operating properly

• Unknown samples we run are true results

• Until when?

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• If result fails/is not the same as known value

– violation occurs
– investigate
– link to potential cause
– troubleshoot/fix problem

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Specimens called QC material

– stable material that can be tested repeatedly
– sufficient volume to last many months
– similar matrix to the unknown samples being run
– variation between vials should be minimal
– values should span clinical range

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• most chemistry tests run 2 levels (every

shift/day/rack/batch)
• exception is immunoassays - run 3 levels
• most often lyophilized
– more stable
– must be reconstituted to manufacturers specifications
(proper diluent)
• can be assayed or unassayed
• some sites will prepare their own QC material

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Liquid vs Lyophilized
 
Liquid Lyophilized
 Easy  cheaper
Pro  No reconstitution
required (errors
associated)
 May behave Reconstitution required
Con
differently than (errors associated)
patient samples
 More expensive

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Lyophilized QC
– Follow manufacturer’s instructions
– Use volumetric pipette
– Mix properly
– *Label with date, time and initials
– Place in refrigerator ASAP
– Let warm to room temp before analyzing

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Chemistry QC run once every 24 hours*

• Hematology slightly different
• Common to have 2 levels (Lev I and Lev II or
Normal/Abnormal or Pos/Neg)
• Multi-analyte controls = ran many tests from same
aliquot of sample
– ex: electrolytes, glucose, urea, creatinine, calcium,
magnesium, AST, ALT, LD, total protein, albumin
– same 500uL sample

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Qualitative
– agglutination vs. no agglutination
– stains that give proper outcomes (pink is pink, blue is
blue)
– growth vs. no growth

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Quantitative
– numerical result matches expected value
– automation
– chemistry and hematology
– accuracy and precision

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• remember…

• if we repeatedly analyze a sample we obtain a frequency

distribution histogram
• if operating properly, values will be closest to the center
as they occur more frequently

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Gaussian Distribution
– Normal distribution
– Describes many continuous laboratory variables
– Bell Curve
– Distribution follows:

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Once we have our Gaussian Distribution we

calculate out SD

  𝟐

𝑺𝑫=
• describes spread around mean
√ ∑ ( 𝑿 − 𝑿´ )
𝒏 −𝟏

• indicator of precision
• lower the value = more precise the method
• reporting SD in same units as data

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• A 68% of the time the

data will fall + 1SD,

• B 95.5% of time it will fall

+ 2SD and

• C 99.7% of time it will fall

+ 3SD.

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• SD as a percentage of the mean

• unitless or percentage
• measure of dispersion
• lower the value = more precise the method
• Can be used to compare the precision of different
methods

𝑺𝑫
  𝑺𝑫
 
𝑪𝑽 = or % 𝑪𝑽 = × 𝟏𝟎𝟎 %
𝑿
´ 𝑿
´

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• Quality Control (QC) Control Charts
– Graphically represent observed values of a control material
over time in context of upper and lower limit controls
(SDs).
– When values fall within control limits, method has
performed adequately.
– When values fall outside control limits, problems could be
developing.
– Control limits expressed as mean ± standard deviation.
– Graphical representation by Levey and Jennings

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Operation of a Quality Control System

– Three-stage process:
1. Establishing allowable statistical limits of
variation
2. Using limits as criteria for evaluating quality
control data
3. Taking action to remedy errors when indicated
• Multirule procedure developed to further judge
whether control results indicate out-of-control
situations

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• 1. Establishing/verifying limits
– QC is run 20x
– 20 days (no less than 10 days + monitoring)
– the more data, the better curve
– calculate Mean ± SD
– use new mean and old SD
– once more data is obtained, new means and SDs can be
adjusted

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• Results outside control limits = possible analytical change
– 1/20 = 2 SD
– 3/1000 = 3 SD
• Some laboratories use 2SD as a control limit, while some use
3SD as control limit and 2SD as a warning

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• When values are out of control it means they have

exceeded the control limits which is statistically unlikely
so patient testing should be halted and investigation into
problem initiated and corrected before patient testing is
resumed.

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Proficiency Testing
– A method used to validate a particular measurement
process.
– Required by CAP, CLIA, IQMH to maintain lab accreditation.
– Most clinical labs use proficiency program provided by CAP.
– Process
• A series of unknown samples is sent to lab from program.
• Samples are analyzed in same manner as patient
specimens.
• Results are reported to program and compiled with
results from all other labs participating in survey.
• Performance report is sent back to each participating lab.

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Proficiency Testing (cont.)

– Strict requirements
1. Lab must incorporate proficiency testing into its
routine workflow as much as possible.
2. Test values/samples must not be shared with
other labs at any time during testing cycle.
3. Proficiency samples are tested by bench
technical staff who normally do patient testing.
4. Testing should be completed within usual time.

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Calibrators and Controls
Box of Calibrators

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Inside the box of Calibrators

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Protein Calibrators

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control-Multiqual

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• Review:
– Method validation
– Calibration, what a calibrator is
– QC, what a QC sample is, when to run QC

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Next:
– How to plot data on LJ chart
– How to apply Westgard rules
– Identify violations and possible outcomes

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Once LJ chart is established…

• Every subsequent day, QC is run and plotted

• Ideally charts should only be made ~1/year

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Types of error
– Random: (unpredictable)
• higher or lower than
expected
• source cannot be
reliably predicted

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Types of error
– Systemic: produce
consistent effects on
controls
• in one direction and
magnitude predictable
• Trend or Shift

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• False Rejection
– If using only + 2SD, 5% of the data falls outside the
range (false rejection)
– If using 2 controls = 10% false rejection
– If using 3 controls = 15% false rejection
• Westgard Multirules allows for false rejections to be
minimized
• Still maintain high level of error detection

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Rules can be described as being violated in both runs and

control levels
– Within run:
– Across runs:
– Within control:
– Across controls:
• Rules are identified by the control limit that is exceeded
and the number of values that exceed that limit (NL)
12s = one control has exceeded + 2SD

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control
• LJ Chart requirements:
– Mean, SD, CV and 3-4 standard deviations
– Analyte name
– Identification of instrument or method
– Identification of QC level with lot# and expiration date
– Calibrator and reagent lot #
– Date for each data point
– Record of intervention actions
– Identification of MLT assessing control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control Sequence
• Quality control is run
– Minimum of 2 levels once every 24hrs
• Values are documented on LJ chart
– Regardless of whether they are in control or out of control
– LIS often used to do this
– If manually, record in ink
• Out of Control values must be investigated (troubleshoot)
before patients can be released
– Evidence of corrective action must be documented
– This may also trigger an “action log” to be completed
• Once issue is resolved and QC is acceptable, results can be
reported

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.
Quality Control

• Read Chapter 3
• Review Figure 3.19 ”QC limits vs. QC goals” (p.66-67)
• Review Definitions Boxes in book
• Check out links on Moodle
• Problem Set 2

*Make sure you understand the difference between a

calibrator and control

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning company.