01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Quality Control: Ensuring the Validity of Calibration

Results
Dr. Irene Flouda December 2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

1
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

INTERNATIONAL
KNOWLEDGE
TRANSFER
ISO/IEC 17025:2017 clause 7.7
• 3 sub-clauses:

7.7.1 Monitoring (i.e., quality control activities)

7.7.2 Proficiency testing/ interlaboratory

comparisons

7.7.3 Analysis of data from monitoring activities

2
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Sub-clause 7.7.1 TRANSFER

Requires a procedure

Record data

• Detect trends
• Apply statistical techniques, where practicable

Review and monitoring

• Monitoring shall include, where appropriate: a) to k)

INTERNATIONAL
KNOWLEDGE
TRANSFER
Sub-clause 7.7.1

Monitoring activities for ensuring validity of results

• a) use of reference materials or quality control materials;
• b) use of alternative instrumentation that has been calibrated to
provide traceable results;
• c) functional check(s) of measuring and testing equipment;
• d) use of check or working standards with control charts, where
applicable;
• e) intermediate checks on measuring equipment;
• f) replicate tests or calibrations using the same or different methods;
• g) retesting or recalibration of retained items;
• h) correlation of results for different characteristics of an item;
• i) review of reported results;
• j) intralaboratory comparisons;
• k) testing of blind sample(s)

3
 01/12/2020

INTERNATIONAL
KNOWLEDGE

Sub-clause 7.7.2 TRANSFER

Monitor
performance by
comparison • Proficiency testing (PT)
with results of • Interlaboratory comparisons (ILCs)
other
laboratories

Benefits of participation:

Supplements internal QC activities

Added confidence

Comparison of performance

INTERNATIONAL
KNOWLEDGE
TRANSFER
PTs- ILAC P9
P9: ILAC Policy for Participation in Proficiency Testing Activities

Minimum PT activity for labs:

Further and ongoing activity that is

Evidence of satisfactory
appropriate to the scope of
participation prior to gaining
accreditation and consistent with
accreditation
the PT participation plan

… And the requirements of

the Accreditation Body

4
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Most crucial points of ABs’ requirements

Reference to the importance of PTs as a tool

to demonstrate technical competence

Requirements concerning i) the minimum

level and frequency of participation and ii)
need for planning participation

How PT participation and performance will

be reviewed and used during assessments.

INTERNATIONAL
KNOWLEDGE
TRANSFER
PTs- ILAC P9

PT Participation Plan
Major
field/discipline

Previous
Status
participation

Plan or current
participation
10

5
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Sub-clause 7.7.3

Analyse data Take appropriate Use to correct

actions for results and improve
outside the pre- laboratory
defined criteria activities

• Predefined • Prevent
criteria incorrect
results from
being
reported

11

INTERNATIONAL
KNOWLEDGE
TRANSFER
Sub-clause 7.7.3- Recording and Analysing Data

Graphs, Plots Uncertainty

and Charts calculations
and budgets

Spreadsheets,
LIMS
logs

12

6
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

13

INTERNATIONAL
KNOWLEDGE
TRANSFER

Foundations of a Quality Control Programme

Internal Quality • Based on day-to-day and item to

Control item monitoring of performance

• Based on the laboratories ’

External Quality performance in interlaboratory
Control comparisons and proficiency
testing programs

14

7
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Evaluation of internally obtained measurement TRANSFER

assurance data
These data can be obtained through:

15

INTERNATIONAL
KNOWLEDGE
Evaluation of internally obtained measurement TRANSFER

assurance data (cont’d)

Each measurement parameter in the laboratory’s scope of

activities must be reviewed and analyzed to determine the validity
of the measurement process.

• The standards and the measurement process for each parameter must
be in a state of statistical control.
• Statistical control means that the variability of the measurement
process is known, stable and observed values are adequately close to
reference values, within the chosen statistical limits.
• When a process is in statistical control and the reference values are
within suitable limits, we can assume that the reported measurement
uncertainties are valid.

16

8
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Evaluation of internally obtained measurement TRANSFER

assurance data
Minimizing risks of measurement errors includes all the following
laboratory functions:

a. Training staff and evaluating effectiveness and proficiency

b. Monitoring the laboratory environment to minimize potential errors or

excess variation
c. Maintaining suitable equipment (including installation, monitoring,
approvals, and integrated software)

d. Selecting and calibrating standards

e. Ensuring suitable suppliers for materials and calibrations

17

INTERNATIONAL
KNOWLEDGE
Evaluation of internally obtained measurement TRANSFER

assurance data (cont’d)

Minimizing risks of measurement errors includes all the following
laboratory functions:
f. Selecting and validating procedures with evaluation of accuracy/bias and
precision

g. Ensuring proper care and handling of laboratory standards, equipment,

and items submitted for calibration

h. Accurately and effectively calculating, evaluating, and reporting

measurement uncertainty

i. Participating in inter- and intra-laboratory comparisons

j. Creating and reviewing calibration certificates to ensure accuracy of

measurement results and the effective communication of results

k. Controlling data – information management (including software and

information technology controls).
18

9
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Evaluation of externally obtained measurement TRANSFER

assurance data
These data can be obtained through:

a. Evaluation of the calibration history of reference standards, working standards,

and check standards
b. Evaluation of before/after calibrations within a laboratory to compare and
evaluate results obtained from an external calibration provider
c. Review of historical calibration data for items calibrated having demonstrated
stability
d. Comparison of all calibration results and calibration history on control charts for
working standards and check standards with results from external calibration sources
e. Participation in proficiency testing using the same procedures and handling
methods used for routine laboratory calibrations
f. Use of externally obtained data from calibrations, PTs, and ILCs in the
assessment of errors and bias in measurement results
g. Participation in ILCs other than proficiency testing (e.g., for method validation
or as a training activity)
19

INTERNATIONAL
KNOWLEDGE
TRANSFER

Combining Data from Multiple Sources

Measurement results • Any observed problems or changes in
collected over several the measurement results are
years may be investigated- corrective action taken
statistically evaluated
with current results • Ongoing monitoring establishes a
being compared to continuous and comprehensive internal
results from previous laboratory measurement assurance
years. program

Data from internal measurement assurance programs should be

compared to the results of calibration history assessments,
interlaboratory comparisons or proficiency tests, and other external
sources of data

20

10
 01/12/2020

INTERNATIONAL
Analysis of results KNOWLEDGE
TRANSFER

Objectives to evaluate the entire measurement process, as a system, will consider:

a. Personnel;
b. Facility/accommodations;
c. Procedures and method validation;
d. Equipment; and
e. Standards.

21

INTERNATIONAL
KNOWLEDGE
TRANSFER

Analysis of results

Definition of the criteria against which the

measurement assurance results are assessed
and evaluated.

Data from all the measurement assurance

activities are analysed and monitored to both
control and improve the laboratory's activities.

When the results of the data analyses from

monitoring activities are found to be outside the
pre‐defined criteria, appropriate actions are taken
to prevent incorrect results from being reported to
the customer.
22

11
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

23

INTERNATIONAL
KNOWLEDGE
TRANSFER
Internal QC- Use of Check Standards- Summary

Regular checks of check standards (where possible), in order to ensure the

reliability of the process of calibration according to the calibration procedures.

An appropriate check standard (or control standard) is incorporated into the

measurement process and measured at established intervals.

The check standard can be either substituted for, or measured as, the item
being calibrated.

The results of the check standard measurements are recorded, charted, and
analysed to establish the measurement capability and to set process control
and warning limits.

The limits are used to establish process uncertainties and to control future
measurement performance.

24

12
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Internal QC- Use of Check Standards- Summary
A check standard must be stable and is normally comparable to the reference
standard or to the typical item submitted for calibration, depending on what
is being monitored (standards or process).

For lower order calibrations, it should simulate the laboratory's reference or

working standards (to the extent feasible).

It should be calibrated using a better procedure than the one being

monitored to ensure that the expanded uncertainty is equal to or better than
the uncertainty achievable with the process being monitored.

All check standards should be cared for in the same way as reference
standards to prevent their damage or deterioration.

Lower order check standards should be recalibrated at regular intervals,

according to the laboratory’s policy on re-calibration intervals.

25

INTERNATIONAL
KNOWLEDGE
Internal QC- Use of Check Standards- Summary TRANSFER

• Control measurements of the check standard are graphed on

control charts ( 𝑥ҧ and standard deviation charts) for visual
examination of process performance and are evaluated against
statistical control limits.
• Monitors the process with respect to both the standard and
the variability
• A central line is drawn, indicating the mean (𝑥ҧ ) of the
measured values and control limits are indicated within
𝑥ҧ control
which the results of measurements are expected to be
chart
randomly distributed, based on statistical considerations.
• The control limits consist of the “action” and “warning” limits
that represent probabilistic limits for the distribution of
results around the central line.

standard
deviation • monitors the short term precision of the process
chart
26

13
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Internal QC- Use of Check Standards- Summary

27

INTERNATIONAL
KNOWLEDGE
TRANSFER

Internal QC- Use of Check Standards- Summary

The system is in statistical control The system is out of control if:

when the individual values are within • values are present outside
the designated statistical limits. established limits for which no
• The statistical information on reasonable and correctable cause
which the control limits are based have been determined and
can be used to calculate corrected
confidence limits for • unusual trends are observed
measurements made while the • the mean exceeds the control
system is demonstrated to be limits
stable and in a state of statistical
control.
• The standard deviation of the
measurement process may be
included in uncertainty calculations

28

14
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Internal QC- Use of Check Standards
Requirements
Check Standards:

Must represent the standard and/or the items to be calibrated.

Must be stable, and their values should be established with accuracy, since
they will be used to control the uncertainty in the calibration process

Their reference values and uncertainty must be determined by a higher

level of calibration than the procedure being monitored, and preferably
using an independent standard, process, or laboratory competent to work
at that level
As a minimum, they may be calibrated within the laboratory only if qualified
to work at the next higher level and using a procedure one level higher
than the calibration process to be controlled

29

INTERNATIONAL
KNOWLEDGE
TRANSFER
Internal QC- Use of Check Standards
Additional Requirements

Experienced operators

A calculating system for statistical control that

calculates standard deviations, control limits, F-tests,
t-tests, root-sum-of-the-squares (RSS), and creates
control charts

30

15
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Terms used in Control Charts
• bias - expresses the difference between the mean value of the
measured quantity and the “known" value (the result of the most
recent calibration).
• short-term variability- often referred to by the term repeatability
and expresses the ability of the device/system to produce under
clearly defined measuring conditions the same measurement
result.
✓ Repeatability is determined by the standard deviation of a
series of repeated measurements.
✓ This standard deviation is a measure of the variability of the
measurement process over a short period of time, usually the
time necessary to complete one calibration using a particular
sequence of measurements called a statistical design. It is
called the "within standard deviation” sw.
31

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Terms used in Control Charts

• long-term variability - associated with changes in existing

environmental conditions of the laboratory or in the
measurement performance.
✓ Often referred to by the term reproducibility and expresses
the ability of a process to produce under non-identical
measurement conditions the same result.
✓ It is also referred to as a "between" standard deviation, sb,
meaning between calibrations, and is attributed to changes in
the calibration process from day-to-day (environmental
changes that are not accounted for by modelling, changes in
artifact alignment relative to the standard , and other
fluctuations that are not reflected in the within standard
deviation.
32

16
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Terms used in Control Charts

• Total standard deviation- includes both the "within" and

"between" components of variability

s c = s w2 + sb2

✓ It reflects both the short-term and long-term random errors

that affect the measurement process.

33

INTERNATIONAL
KNOWLEDGE
Check Standards- Objectives of QC TRANSFER

• QC is used for defining, measuring, analysing, improving, and

controlling the measurement process and uncertainty as the
calibration is performed through the use of suitable check
standards.
• The uncertainty includes effects of the measurement instrument,
the operator, the procedure, the standards, and the environment
over time.
• Each process is modelled to determine and control:
✓ 1) the measurement process;

✓ 2) the calibration and check standards; or

✓ 3) a balance of both the process and the standards.

34

17
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Check Standards- Objectives of QC TRANSFER

• The objectives will establish the value of the check standard, the
check standard measurement procedure, and influence the
frequency of control measurements of the check standard.
• Objectives may be:
✓ 1) Determine the standard deviation of the process (sp);
✓ 2) Determine the expanded uncertainty, U;

✓ 3) Measure the value of the calibration standard uncertainty

(us).
The model may allow any one objective or a combination of the
objectives to be established

35

INTERNATIONAL
KNOWLEDGE
Check Standards- Methodology TRANSFER

• The check standard is selected to evaluate the standard deviation

of the process, sp, other process uncertainties, uo, and possible
bias, ud of the process.
• Initial measurements for the check standard should be performed
immediately after calibrating the reference or working standards
and after the servicing of the measurement instrument.

Symbol Description
Sc Control measurement of check standard
Scs Accepted value of check standard
U Expanded Uncertainty (of the process)
uc Combined standard uncertainty
us Standard uncertainty of the standard
uo Standard uncertainty of other factors
ud Standard uncertainty of differences
sp Standard deviation of the process
k Coverage factor
36

18
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Check Standards- Obtain Initial Measurements to TRANSFER

Establish Control Charts

• To establish a new control chart, a minimum of 12 or more, independent
measurements of the check standard should be made, using the same
standards, equipment, procedure, and under the same conditions that
will be used to make routine measurements.
• A calibration is defined as the result of replicate measurements as
required by the respective SOP (i.e., a complete test consists of the
number of runs specified in the SOP).
• No two measurements may be made on the same day.
• The time of day should be varied as would be typical during routine
laboratory operations. This is necessary to estimate the long-term
standard deviation to the extent feasible. To make statistically valid
decisions or calculate uncertainties based on this data, 25 to 30 points
are necessary.

37

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Obtain Initial Measurements to
Establish Control Charts

a) A database is created. This database is the statistical population

from which the quality parameters of the measuring process
(bias, long and short-term variability) can be identified and
quantified.
b) Using statistical criteria, the compatibility of the latest
measurements with the defined quality parameters of the
process is examined ( process monitoring )
• The quality parameters are adjusted at regular intervals (every 12
measurements ) to reflect the most recent state of the measuring
processes.

38

19
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Check Standards- Obtain Initial Measurements to TRANSFER

Establish Control Charts

• Conditions:
a) data belong to the statistical distribution (population).
b) the distribution is normal
c) the measuring errors are uncorrelated over time.
• n daily (n≥4) iterations during m days (n is the same for all the
daily repetitions).
• Each point in the database is denoted by Χij (i=1,…m; j=1,…n) The
average of the n iterations of the i-day equals:

𝑛
1
𝑋𝑖 = ෍ 𝑋𝑖,𝑗
𝑛
𝑗=1

39

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Calculations for Construction of Control
Charts
• A standard deviation may be calculated for each set of runs
according to the appropriate SOP.
• Standard deviation of the n repetitions for v=n-1 degrees of
freedom (short-term variability of the measuring system):
𝑛
2 1
𝑠𝑖 = 𝑠𝐶 = ෍ 𝑋𝑖,𝑗 − 𝑋𝑖
𝑛−1
𝑗=1

• A pooled standard deviation is then determined for the

measurement process.
• Pooled standard deviation: a better estimation of the
repeatability of the measurement process and requires the
combination of the m daily standard deviations (v= m(n-1)
degrees of freedom):
𝑚
1
𝑠𝑝 = ෍ 𝑠𝑖2
𝑚
𝑖=1
40

20
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Check Standards- Calculations Related to Control Charts

• Standard deviation of the m values of the check standard:

expresses the long-term variability of the process (v= m-1 degrees
of freedom):
𝑚
1 2
𝑠𝑐ℎ𝑘𝑠𝑡𝑑 = ෍ 𝑋𝑖 − 𝑋
𝑚−1
𝑖=1

• General mean value of all the measurements that refer to the

check standard
𝑚
1
𝑋 = ෍ 𝑋𝑖
𝑚
𝑖=1

41

INTERNATIONAL
KNOWLEDGE

Check Standards- Calculations Related to Control Charts TRANSFER

• Control of the long-term variability of the process: t-test

• Checks whether the measurement difference Δycheck from the
latest quality control measurement is not essentially different
from the overall average of the differences from the previous
control measurements.
• Α test statistic is computed from the current check standard
measurement, the accepted value of the check standard, and the
total standard deviation.
y check −  y check y: measurand
t=
s chkstd
The calibration process is under statistical control if:
t ≤critical value of t-distribution with n degrees of freedom
42

21
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Calculations Related to Control Charts
• Control of the long-term variability of the process: t-test
• Critical value tα/2(ν)- v Dof: depends on v and on α/2 , the
significance level (probability of mistakenly flagging a check
standard measurement as out-of-control).
• Critical values for a two-sided t test with α=0,05:
Degrees of Critical value Degrees of Critical value
freedom freedom
1 12,706 14 2,145
2 4,303 15 2,131
3 3,182 19 2,093
4 2,776 24 2,064
5 2,571 29 2,045
6 2,447 34 2,032
7 2,365 39 2,023
8 2,306 44 2,015
9 2,262 49 2,010
43 10 2,228 50 2,009

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Calculations Related to Control Charts

• Control of the short-term variability of the process: F-test

• If the measurement sequence allows for a within standard
deviation, the ratio of this within standard deviation to the pooled
standard deviation sp is compared to F(critical value).

𝑛 1 m 2
 si
2
s
F = 2i 𝑠𝑖 = 𝑠𝐶 = ෍ 𝑋𝑖,𝑗 − 𝑋𝑖
2 1 sp =
𝑛−1 m i =1
sp 𝑗=1

• The repeatability is under statistical control when:

F ≤critical value of F-distribution with v degrees of freedom for si
and m.ν for sp

44

22
 01/12/2020

INTERNATIONAL
KNOWLEDGE

Check Standards- Calculations Related to Control Charts TRANSFER

• Control of the short-term variability of the process: F-test

Critical values of F-distribution so that snew(ν)  sp(mν) for one-sided test with
α=0,05:
mν ν=1 2 3 4 5 6 7 8 9 10
1 161,45 199,50 215,71 224,58 230,16 233,99 236,77 238,88 240,54 241,88
10 4,965 4,103 3,708 3,478 3,326 3,217 3,135 3,072 3,020 2,978
16 4,494 3,634 3,239 3,007 2,852 2,741 2,657 2,591 2,538 2,494
20 4,351 3,493 3,098 2,866 2,711 2,599 2,514 2,447 2,393 2,348
26 4,225 3,369 2,975 2,743 2,587 2,474 2,388 2,321 2,265 2,220
30 4,171 3,316 2,922 2,690 2,534 2,421 2,334 2,266 2,211 2,165
36 4,113 3,259 2,866 2,634 2,477 2,364 2,277 2,209 2,153 2,106
40 4,085 3,232 2,839 2,606 2,449 2,336 2,249 2,180 2,124 2,077
45 4,057 3,204 2,812 2,579 2,422 2,308 2,221 2,152 2,096 2,049
50 4,034 3,183 2,790 2,557 2,400 2,286 2,199 2,130 2,073 2,026
60 4,001 3,150 2,758 2,525 2,368 2,254 2,167 2,097 2,040 1,993
70 3,978 3,128 2,736 2,503 2,346 2,231 2,143 2,074 2,017 1,969
80 3,960 3,111 2,719 2,486 2,329 2,214 2,126 2,056 1,999 1,951
90 3,947 3,098 2,706 2,473 2,316 2,201 2,113 2,043 1,986 1,938
100 3,936 3,087 2,696 2,463 2,305 2,191 2,103 2,032 1,975 1,927
45

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Calculations Related to Control Charts
• Shewhart Control Charts: Bias and long-term variability
• Control chart parameters: are evaluated based on a reasonable
number of initial measurements and updated as additional
measurement data are accumulated.
• A known value is based on a higher-level calibration of the check
standard that is preferably independent of the measurement
system being monitored.
• The central line is established by the mean of measurements.
• If no higher-level calibration is available, the central line may be
used as the known value, but this is not recommended since it
will allow no evaluation of measurement accuracy or bias.
• Upper and lower control limits should be fixed, and adjusted after
periodic evaluation when/if appropriate.
46

23
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Calculations Related to Control Charts
Shewhart Control Charts: Bias and long-term variability (𝑋𝑖 vs time)
Construction of ഥ𝒙 control chart:
• Calculate the mean (central line), 𝑋, and 𝑠 = 𝑠𝑐ℎ𝑘𝑠𝑡𝑑
𝑚
1 1 𝑚 2
𝑋= ෍ 𝑋𝑖 𝑠𝑐ℎ𝑘𝑠𝑡𝑑 = ෍ 𝑋 −𝑋
𝑚 𝑚 − 1 𝑖=1 𝑖
𝑖=1
• Establish the control chart parameters as follows:
Control Chart Parameter Value
Central Line 𝑥ҧ
Upper Control/Action Limit (UCL) 𝑥ҧ + 3𝑠
Upper Warning Limit (UWL) 𝑥ҧ + 2𝑠
Lower Warning Limit (LWL) 𝑥ҧ − 2𝑠
Lower Control/Action Limit (LCL) 𝑥ҧ − 3𝑠
𝑠 = 𝑠𝑐ℎ𝑘𝑠𝑡𝑑
47

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts
Standard Deviation Control Charts: Short-term variability (𝒔𝒊 vs time)
• This chart is characterized by the central reference line and the upper
control limit.
• This is due to the fact that the deterioration of the short-term
variability of the measurement process is expressed only by the
increase of the standard deviation of repeated measurements.
• The central/base line is the pooled standard deviation sp

• Upper control limit: 𝑈𝐶𝐿 = 𝑠𝑝 𝐹𝑎 𝑛 − 1; 𝑚 𝑛 − 1

• α is chosen based on the desired limits (e.g., 0.05 for 95 %).

• n-1 represents the degrees of freedom from the replicates.
• m(n-1) represents the DoF corresponding to sp.
48

24
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

Check Standards- Analyse Data of Control Charts

Flowchart
Definition of the criteria against which the measurement assurance
results are assessed and evaluated

Data from all the measurement assurance activities are analysed and
monitored to both control and improve the laboratory's activities

Inside the predefined criteria? Outside the predefined criteria?

Take appropriate actions to

Any trends? prevent incorrect results from
being reported to the customer
49

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts:
Outline

In case of normal
If plotted points are stable and process statistic, this
randomly distributed within the means that 99.73 %
warning limits, the system is in of the points will fall
control. within the
control/action limits.

50

25
 01/12/2020

INTERNATIONAL
KNOWLEDGE

Check Standards- Analyse Data of Control Charts: TRANSFER

Outline
Observations outside the limits, or systematic patterns within, should
be investigated and corrected as appropriate or possible

If a plotted point is
outside the warning The re-measured value
limits but within the If none were made, re- must be within the
control limits, measure the check warning limits to merit
investigate the standard the decision of "in
presence of calculation control".
errors.

Keep the “out of

If the re-measured Reject all data control” data in the
results are not within obtained since the last chart and flag it as
limits, consider the "in-control" problematic. In control
measurement process measurement and take again: 2 successive
"out of control" corrective action. measurements within
the control limits
51

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts:
Outline
If a plotted point is outside the control limits and arithmetically correct,
the system is out of control.

Re-attainment of
statistical control must
Corrective actions must
Data are rejected be demonstrated
be taken
before data are
accepted.

52

26
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts

• Additional guidelines for the evaluation of control charts based on

probability statistics should be used to evaluate the presence of
drift, shifts, and possible bias. Examples:
✓ Any single point or series of points outside of 3s (keeping in mind
the probability that a 3s limit could reasonably expect to allow 3
points out of 1000 to be outside these limits);
✓ 2 of the last 3 points are above (or below) 2s;
✓ 4 of the last 5 points are above (or below) 1s;
✓ 8 consecutive points are on one side of the mean or reference
value;
✓ 6 points in a row are trending up (or down); and
✓ 14 points are alternating up and down (sawtooth pattern) about the
mean or reference value.

53

INTERNATIONAL
KNOWLEDGE
TRANSFER

Check Standards- Analyse Data of Control Charts

Nelson Rules

Rule 1 One point is more than 3 standard deviations from the mean.
Rule 2 Nine (or more) points in a row are on the same side of the mean.
Rule 3 Six (or more) points in a row are continually increasing (or decreasing).
Rule 4 Fourteen (or more) points in a row alternate in direction, increasing
then decreasing.
Rule 5 Two (or three) out of three points in a row are more than 2 standard
deviations from the mean in the same direction.
Rule 6 Four (or five) out of five points in a row are more than 1 standard
deviation from the mean in the same direction.
Rule 7 Fifteen points in a row are all within 1 standard deviation of the mean
on either side of the mean.
Rule 8 Eight points in a row exist, but none within 1 standard deviation of the
mean, and the points are in both directions from the mean.
54

27
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts
Nelson Rules

Rule 1 One point is more Tests for stability. This is

than 3 standard the strongest evidence of
deviations from the lack of control. One
sample (two shown in this
mean.
case) is grossly out of
control.
Rule 2 Nine (or more) Tests for stability. This can
points in a row are be used to supplement
on the same side of rule 1. Some
prolonged bias exists.
the mean.

Rule 3 Six (or more) points A trend exists

in a row are
continually
increasing (or
decreasing).
55

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts
Nelson Rules

Rule 4 Fourteen (or more) Tests for a systematic

points in a row variable. The pattern of
alternate in variation should be random,
direction, increasing but is predictable if failing
then decreasing. rule 4. This much oscillation is
beyond noise. Note that the
rule is concerned with
directionality only. The
position of the mean and the
size of the standard deviation
have no bearing.
Rule 5 Two (or three) out of Tests for small shifts in the
three points in a row data. There is a medium
are more than 2 tendency for samples to be
standard deviations out of control. The side of the
from the mean in mean for the third point is
the same direction. unspecified.
56

28
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts
Nelson Rules

Rule 6 Four (or five) out Tests for small shifts in

of five points in a the data. There is a
row are more strong tendency for
samples to be slightly
than 1 standard
out of control. The side
deviation from of the mean for the fifth
the mean in the point is unspecified.
same direction.

Rule 7 Fifteen points in a Tests for stratification,

row are all within which can be
1 standard misinterpreted as good
process control. Within
deviation of the
1 standard deviation,
mean on either greater variation would
side of the mean. be expected.

57

INTERNATIONAL
KNOWLEDGE
TRANSFER

Check Standards- Analyse Data of Control Charts

Nelson Rules

Rule 8 Eight points in a Tests for mixture,

row exist, but which is when the
none within 1 data avoids the
centre line and lies
standard
near the control
deviation of the limits. Jumping from
mean, and the above to below
points are in both whilst missing the
directions from first standard
the mean. deviation band is
rarely random.

http://asq.org/learn-about-quality/data-collection-analysis-tools/overview/control-chart.html
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc3.htm

58

29
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts

Even while the system is in an apparent state of

control, incipient troubles may be indicated
when the control data show short or long term
trends, shifts, or runs.(Use F and t-tests to
estimate their significance)

If the values plotted on the standard deviation

chart fall outside of the control limits, a
decrease in precision is indicated. Problems
with the standards or process will need to be
investigated
If the plotted value of 𝑥ҧ lies outside of the
control limits and the corresponding value on
the standard deviation chart is within the
control limits, a source of systematic error is
suspected.
59

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Data of Control Charts

• Demonstration of "in control" indicates that the calibration

process is consistent with the past experience of the laboratory.
There is no reason to believe that excessive changes in precision
have occurred.
• To the extent appropriate, the indicated measurement precision
of the calibration of the measured type of standards may be
extended to the calibration of other standards of similar type,
capacity and design.

60

30
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Reference Values

When the system remains in

Μay be considered
control: Absence of a
evidence of insignificant
significant difference
bias at the level of
between the central line
confidence of the statistical
and the accepted reference
test used.
value for the check standard

On occasion, small differences (less than

one standard deviation) from unknown
sources will become obvious over time
and the value observed for the bias may
be incorporated into the uncertainty.

61

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Analyse Reference Values
When a Reference Value for it may necessitate obtaining an
the check standard is more updated calibration or evaluating the
than one standard deviation bias or deviation further to
from the mean value: determine the cause and correct it.

The deviation or offset must be

considered with respect to the reported
uncertainty for the reference value as
well as the measurement process being
used to evaluate the value.
Where tolerances are very large, a
the offset can be used as an
measurement process offset might
uncorrected systematic error in
be quite small compared to the
the uncertainty calculations
tolerance

Where tolerances are small or updated calibrations may be

uncertainty requirements stringent required
62

31
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Check Standards- Analyse Reference Values TRANSFER

• The Normalized Error, En, may be used to compare the mean

value of the check standard, Sc to the calibrated reference value,
Scs (once adequate metrological traceability for the reference
value is ensured).
• Expanded uncertainty of the reference value from the calibration
certificate.

• Uncertainty of the mean value of the check standard :

2
𝑠𝑝
𝑢𝑐 = + 𝑢𝑠2 + 𝑢02 𝑈𝑆𝑐 = 𝑘 × 𝑢𝑐
𝑛

• sp: standard deviation of the process from the control chart,

• n the number of relevant data points;
• us:standard uncertainty of the standard
• uo :any other critical components to be considered.
63

INTERNATIONAL
KNOWLEDGE
Check Standards- Analyse Reference Values TRANSFER

𝑆ഥ𝑐 − 𝑆𝑐𝑠
𝐸𝑛 =
𝑈𝑆2𝑐 + 𝑈𝑆2𝐶𝑆

• 𝐸𝑛 ≤ 1 to pass
• If 𝐸𝑛 > 1 , corrective action is required.

64

32
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Improve the Process

During analysis of the measurement process or the reference and

check standards, opportunities for corrective action and
improvement action may be identified.

Control charts or values that do not have normal distributions or

which have significant differences between observed measurement
results and reference values are cause for action of some type.

During review of the charts, the first step is to identify the

source/cause of the concern, and the associated actions will
generally follow.

65

INTERNATIONAL
KNOWLEDGE
Check Standards- Improve the Process TRANSFER

• Examples:
Assignable Cause – Source of Problem Example Action Item
Standard deviation has increased after new staff Staff might need training, instruction, or oversight;
member is hired uncertainties may need to be increased
Standard deviation has increased over the past Devices may need service; uncertainties may need
year to be increased
Standard deviation has gotten smaller over the Uncertainties may benefit from being reduced
past few months (due to staff training or devices (may also not matter if tolerances are large
service) enough)
Observed values of the check standards suddenly
changed. Possibilities might be standards were Identify root cause – take appropriate action.
switched, standards were damaged, values of For example, replace standards, update values of
standards after calibration were not updated standards, validate software, contact calibration
properly, new software was implemented without provider. Evaluate whether the shift corresponds in
proper validation, standards recently returned direction and magnitude to changes in the
from calibration and the provider cleaned them calibration value of standards used.
(weights).
Standard values are demonstrating a drift over
time. Possibilities may be that standards were not
Allow standards to equilibrate longer; replace
equilibrated long enough before being placed into
unstable standards or check standards.
service; standard or check standard type and
66design might be inherently unstable.

33
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Updating Control Chart Parameters

• Update control chart parameters when:

✓ a significant amount of additional data is available, or
✓ the previously determined parameters are no longer relevant
due to changes in the system.

67

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Updating Control Chart Parameters
IF: THEN:
Determine the reason for the difference,
The tests fail and results are significantly
if possible, and decide whether
different
corrective action is required.

Establish new parameters and limits

using the most recent data and note the
Data do not agree within statistical limits
reasons for not using previous data or
correct the causes of variation.

Be sure to note the degrees of freedom

Portions of the process or standard
to support uncertainty analyses and
variations pass
coverage factors.

Pool all data and calculate new control

No significant differences between the
chart parameters based on all existing
data sets are found
data.
68

34
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Control the Process

• How frequently should the check standard be measured and

plotted?
✓ Sufficient frequency to minimize the risk of loss of data during
the period from last-known-in to first-known-out of control
condition.
• Good practice: measure the check standard at least once during
each period when a set of calibration measurements is made.
• For critical calibrations or those of highest accuracy, it is desirable
to alternate measurements of calibration items and check
standards.
• For real-time evaluation it is preferable to incorporate the check
standard in the calibration design (SOP).
69

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Control the Process

• Whenever there has been a long period of inactivity, it is good

practice to make a series of measurements of the check standard
and to plot the results on a control chart to demonstrate
attainment of statistical control prior to resuming measurements
with that specific calibration system.
• Check standard measurements should be plotted in control charts
as close to real time as feasible to effectively monitor the
measurement process and to prevent the possible release of
questionable data that may result in the recall of laboratory work.

70

35
 01/12/2020

INTERNATIONAL
KNOWLEDGE
Check Standards- Checklist for Creation or Evaluation of Control TRANSFER

Charts
Control Charts cover entire scope (are available for each measurement parameter).

Control charts have titles (or otherwise include)

Laboratory name or other identifying information
SOP(s) used to generate measurement result
Equipment
Standard Identification, Check standard Identification
Nominal value
Dates/chronology/time periods if not used on the x axis
Legends if multiple series and extra information are plotted (to avoid confusion )

Control charts have x and y axis with labels and

All measurement values have units of measure associated with them

Control charts have

Mean value (and units)
Standard deviation (and units)
Degrees of freedom or number of points noted (if not obvious or if small number)
Alternative summaries of this information and suitable references (e.g., tables)
Printed summary reports of control data with the charts
Reference values and source and bias if appropriate/available
71

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Checklist for Creation or Evaluation of Control
Charts
Control charts have limits that are based on
Statistical controls of:
Warning limits (i.e., two standard deviations) and
Action/control limits (i.e., three standard deviations)
Or, specification limits (e.g., tolerances or smaller ratios of tolerances)

Good Items (on chart or in spreadsheet or database table summaries). Control charts
have (when applicable and meaningful if not otherwise noted, e.g., in a table)
Tolerances: when applicable
Uncertainties: for the reference value, check standard, and the process output
Equipment information: device readability, configuration (stability
settings/timing)
Standard information: calibration date and interval information
Responsible staff: need on chart or in database
Status of control: in control, out of control with latest date of review
History: previous limits and history of the chart/data with F-test and/or t-test
results

72

36
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Transfer of Statistics

• The estimate of the standard deviation of the process, sp, used to

establish the control limits may be used to calculate confidence
intervals for all related measurements made while the system is in
control.
• The value of the item being calibrated is said to be within the
limits if the combined mean of the measurements on the
calibration item and the expanded uncertainty are within limits.

73

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- Control Charts-Further Thoughts

Shewhart control chart:

the decision regarding the

easy to be constructed state of control of the
and applied process at any time depends
solely on the most recent
ability to detect large measurement from the
changes process and the degree of
"trueness" of the estimates
of the control limits from
historical data
limited ability to detect
small changes (at the level of
½ to 1 st. dev.)

74

37
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- EWMA Control Charts

• Exponentially Weighted Moving Average (EWMA) is a statistic for

monitoring the process that averages the data in a way that gives
less and less weight to data as they are further removed in time.
• The decision depends on the EWMA statistic, which is an
exponentially weighted average of all prior data, including the
most recent measurement.
• By the choice of weighting factor, λ, the EWMA control procedure
can be made sensitive to a small or gradual drift in the process,
whereas the Shewhart control procedure can only react when the
last data point is outside a control limit.

Lucas, J. M. and Saccucci, M. S. (1990). "Exponentially weighted moving average control schemes:
Properties and enhancements", Technometrics 32, 1-29.
75

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- EWMA Control Charts

• The EWMA control chart is characterized by a central reference

line that results from the average of all values in the control
template database and from two bounding lines, UCL and LCL:
• 𝐸𝑊𝑀𝐴𝑡 = 𝜆𝑌𝑡 + 1 − 𝜆 𝐸𝑊𝑀𝐴𝑡−1 𝑓𝑜𝑟 𝑡 = 1,2 … , 𝑛
2 𝜆
• 𝑠𝐸𝑊𝑀𝐴 = 2−𝜆 𝑠2 • EWMA0 is the mean of historical data
(target)
• 𝑈𝐶𝐿 = 𝐸𝑊𝑀𝐴0 + 𝑘𝑠𝐸𝑊𝑀𝐴 • Yt is the observation at time t
• n is the number of observations to be
• 𝐿𝐶𝐿 = 𝐸𝑊𝑀𝐴0 − 𝑘𝑠𝐸𝑊𝑀𝐴 monitored including EWMA0
• k usually =3 • 0 < λ ≤ 1 is a constant that determines
the depth of memory of the EWMA
• s is standard deviation calculated from
the historical data

76

38
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Check Standards- EWMA Control Charts

77

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

78

39
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Internal QC- Intra-laboratory Comparisons

• An intra-laboratory comparison is conducted when:

✓ Several technicians within an organization perform
calibrations on the same or similar artifact, using the same
method, under specified, controlled conditions.
✓ The same artifact is measured by using different methods
or different calibration systems.
• The data resulting from these activities shall be analysed for
statistical validity

79

INTERNATIONAL
KNOWLEDGE
TRANSFER

Contents
• ISO/IEC 17025:2017 Requirements
• Foundations of a Quality Control Programme
• Internal QC:
✓ Use of Check Standards

✓ Control Charts

✓ Intra-laboratory Comparisons

• External QC:
✓ PTs/ILCs

80

40
 01/12/2020

INTERNATIONAL
KNOWLEDGE
External QC- Participation in PTs/ILCs TRANSFER

Use of PTs/ILCs:
a. Evaluating laboratory performance of specific measurement scope
capabilities and monitoring continuing performance
b. Identifying problems and initiating corrective action.
c. Establishing the effectiveness and comparability of measurement
methods (e.g., method validation).
d. Evaluating method performance characteristics (e.g., method validation).

e. Providing additional confidence to laboratory customers

f. Identifying of differences among laboratories

g. Educating participating laboratories based on comparison outcomes

h. Validating uncertainty claims.

81

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs

PT plans and schedules shall be developed and

updated at least annually to determine if the
participation frequency is appropriate and
adequate.

Usual Minimum requirements: Each laboratory

shall participate in one PT per major sub-area of
their accredited or recognized scope every five
years.

Recommendation: organizations participate in PT

activities for each specific measurement parameter
and calibration method that the laboratory employs
as a part of ongoing QC program.

82

41
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

External QC- Participation in PTs/ILCs- Assessment of Results

Evaluation of PT/ILC results for calibration laboratories:
• Bias (Difference): 𝒙𝒍𝒂𝒃 − 𝑿𝒓𝒆𝒇
This value is not used as a pass/fail statistic, but is used in the initial
assessment of data by the PT provider to review the overall data for
obvious blunders and outliers.
The laboratory may use this value as a part of its follow-up assessments
of laboratory bias, accuracy goals, and plans for recalibration limits. (For
precision calibrations, a laboratory might want to set recalibration goals
such that whenever the bias/offset exceeds some ratio of its reported
uncertainty, a recalibration or interim assessment of metrological
traceability is conducted).

83

INTERNATIONAL
KNOWLEDGE
TRANSFER

External QC- Participation in PTs/ILCs- Assessment of Results

Evaluation of PT/ILC results for calibration laboratories:

𝑥𝑙𝑎𝑏 −𝑋𝑟𝑒𝑓
• Normalized Error 𝐸𝑛 : 𝐸𝑛 = < 1 to pass
2 ±𝑈 2
𝑈𝑙𝑎𝑏 𝑟𝑒𝑓

When + ?
When - ?

84

42
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER

External QC- Participation in PTs/ILCs- Assessment of Results

Interpretation of En

0.5

1.25

85 Source: NIST

INTERNATIONAL
KNOWLEDGE
TRANSFER

External QC- Participation in PTs/ILCs- Assessment of Results

• Lab A:
✓ The value submitted is outside the uncertainty of
the reference value
✓ Its uncertainty overlaps the reference value.
✓ Visually, there is a good amount of overlap of
the uncertainty bars.
✓ En (=0.689) < 1 Pass.
• Interpretation:
An En value of 0.689 might still justify further assessment of the
laboratory accuracy by determining if the Bias (Difference) that is shown
has been consistent in previous PTs or is observed in a laboratory control
chart. Further evaluation would depend on the applicable tolerances for
the application and the desired level of accuracy needed by the
laboratory
86

43
 01/12/2020

INTERNATIONAL
KNOWLEDGE
External QC- Participation in PTs/ILCs- Assessment of Results TRANSFER

• Lab B:
✓ The value submitted is identical to the value from
lab A, thus the Bias (Difference) is also identical.
✓ UB<UA and UB< Uref
✓ While the uncertainties still overlap slightly,
UB does not overlap the reference value, Uref
does not overlap the submitted lab B value
✓ En (=1.115) > 1 Fail.
• Interpretation:
The Bias (Difference) for both labs A and B are identical, but UB does not
support this level of bias. Either the uncertainty is too small if all other
laboratories performed the same procedure and submitted uncertainties
comparable to Labs A and C (likely) or the laboratory needs to identify the
root cause of this failure (e.g., a systematic error of some type or the need
for recalibration of standards to bring values closer to the reference value).
87

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Assessment of Results
• Lab C:
✓ The value submitted is not inside reference
value uncertainty.
✓ UC=UA
✓ There is very minor overlap of uncertainty
values, but the overlap is not enough.
✓ En (=1.325) > 1 Fail.
• Interpretation:
Some laboratories working with larger tolerances might suggest that the
offset does not matter and the failure is not significant, which is counter
to the purpose of PTs. If the tolerances are significantly larger than the
offset shown, a larger uncertainty to cover the gap and pass the En
assessment is likely warranted.

88

44
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions

• Pass/fail status of each standard evaluated in the PT is not the

only thing a laboratory should consider when participating in a
PT.
• Recommendation: Prepare a thorough follow up assessment
that creates an Executive Summary that can be used in a
Management Review as well as guiding the laboratory in
performing a thorough assessment of the report, even when
all indicators were successful.

89

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

Results and
Assessment
Evidence
1. Executive Summary
2. PT Failures
3. Analysis and Action Plan
4. Deadlines
5. Uncertainties (7.6., 7.8.6).
6. Uncertainty versus Applicable
Tolerances (7.8.6).
7. Offset/Bias Assessment (7.7)
8. Records (7.5, 7.8, 8.4).
9. Non-Measurement Result
Observations or Failures

90

45
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:
Include summary and highlight the PT findings that can be
used in the Management Review. (For example, total
1.Executive number of points, number passing/failing percentages,
Summary: lessons learned, opportunities, corrective action already
taken, etc.)

2. PT Failures: Describe all laboratory failures that were identified in the

final PT report (or additional failures or concerns identified
outside the report).

3.Analysis and Describe the analysis and investigation of Root Cause

Action Plan: Analysis, Risk/Opportunities, Improvement Action(s), and
Corrective Action(s). (Section 8.5, 8.6, 8.7)

91

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

4. Deadlines: List the deadlines for the completion of each action item
and identify the personnel responsible for implementing
and monitoring the results of each action.

5. Uncertainties
(7.6., 7.8.6):
Describe the uncertainty assessment.

6. Uncertainty
versus Applicable Describe the Precision assessment results for this
Tolerances (7.8.6).: proficiency test. This assessment reviews the laboratory
Uncertainty compared to the applicable Tolerances.

92

46
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

5. Uncertainties (7.6., 7.8.6). Questions to consider include:

How did the reported uncertainty compare to other participating laboratory

values?
Was the correct k factor used?

If the laboratory (or laboratory participant(s)) uncertainty value(s) were at the

high end of the uncertainties, explain why. Could a better procedure or
instrument have been used?

If at the low end, was the value calculated correctly? Why is it smaller than the
values reported by other laboratories?

Explain if all appropriate uncertainty components were included (or why they
were not included). Describe the planned measurement process and/or actual
procedure used for the PT (higher/lower level procedure).

93

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

6. Uncertainty versus Applicable Tolerances (7.8.6). Questions to consider include:

Was a precision test conducted as a part of the analysis? If yes, explain why there
any ranges with unacceptable results. If no, conduct the precision analysis now.

The calculation evaluates the reported uncertainty(expanded at k=2) against the

tolerances required for the equipment with any uncertainty to tolerance ratios
considered. Was the uncertainty reported acceptable/appropriate for the level of
work? Could the uncertainty be improved with different equipment or
procedures?

94

47
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

7. Offset/Bias Was bias observed in the PT also observed in other types of

Assessment (7.7).: measurement assurance charts in the laboratory? Describe
or summarize the bias and offset of the laboratory PT.
Describe the assessment in place to track PT data over time
within the laboratory and evaluate the data against previous
8. Records (7.5, results and other data. Ensure that the final results were
7.8, 8.4): entered in laboratory PT log and identify the summary data
that will be included in the Management Review (8.9).
9. Non-
Measurement Describe any additional feedback related to the PT
Result planning, scheduling, evaluation (e.g., delays) or reporting
Observations or results (e.g., calibration certificate review, 7.8) that were
Failures: provided as a part of this PT.

95

INTERNATIONAL
KNOWLEDGE
TRANSFER
External QC- Participation in PTs/ILCs- Follow-up Actions
• Example of Executive Summary Report:

7. Offset/Bias Assessment (7.7). Questions to consider include:

Was the En > 1? If En > 1, a measurement bias was indicated. If a bias was
present, are there any overriding reasons for it?

An investigation generally needs to be conducted looking for common errors and

problems: e.g., incorrect values for the standard used, errors in software used for
calculations, deviations from SOPs, need for calibration of standards.

Conduct an investigation of bias (even if values passed the En analyses) against

internal calibrations, control charts, previous PT results or recent calibrations to
find out if there is correlation of the ILC data with internal laboratory data.

Was the bias for all laboratory participants similar? If not, describe why.

96

48
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
Conclusions
• The major advantage of a successful system of Quality Control is
the ability to evaluate the reference standards and the
measurement process over time, providing ongoing assurance
regarding accuracy and traceability of the reference standards for
both the laboratory and its customers.
• When a process is in statistical control and the reference values
are within suitable limits, we can assume that the reported
measurement uncertainties are valid.
• Ongoing evaluation of the measurement process provides the
laboratory with data that can be used to establish or adjust
calibration intervals for reference standards.
• The measurement assurance program is also critical for defining
and reporting realistic uncertainties.

97

INTERNATIONAL
KNOWLEDGE
TRANSFER
References
• ISO/IEC 17025:2017: General requirements for the competence of
testing and calibration laboratories.
• JCGM 100:2008 “Guide to the expression of uncertainty in
measurement”
• M. Grabe, “Principles of Metrological Statistics”, Metrologia 23,
(1986/1987), 213.
• K.Weise and W. Woeger, “A Bayesian Theory of Measurement
Uncertainty”, Meas. Sci. Tech. 3, (1992), 11.
• C. Eisenhart, “Realistic Evaluation of the Precision and Accuracy of
Instrument Calibration Systems”, NBS Special Publication 300, vol.
I, H.H. Ku editor, Washington DC 1969, pp. 21-48.

98

49
 01/12/2020

INTERNATIONAL
KNOWLEDGE
TRANSFER
References
• A.J. Duncan, "Quality Control and Industrial Statistics", 5th ed.,
Irwin, Homewood, IL, 1986.
• Bucher, J.L., “Quality Calibration Handbook—Developing and
Managing a Calibration Program”, ISA Publication, ASQ Quality
Press, 2007.
• J.K. Taylor and H.V. Oppermann, "Handbook for the Quality
Assurance of Metrological Measurements", NBS Handbook 145,
Gaithersburg, 1986.
• NIST/SEMATECH e-Handbook of Statistical Methods,
http://www.itl.nist.gov/div898/handbook/, January 19, 2019

99

INTERNATIONAL
KNOWLEDGE
TRANSFER

Thank you for your attention!

[email protected]

100

50