STATISTICAL PROCEDURES FOR MEASUREMENT SYSTEMS

VERIFICATION AND VALIDATION

GENERAL
Specific details of calculations and interpretations can be found in the references listed at
the end of this guidance document.
All measurements will be randomized and will require independent set-ups for repeated
measurements. Whenever possible, the test operator will not be aware of the part number
or serial number to avoid operator bias.
All experimental structures should ensure that no primary causal categories are
confounded.
Functional, non-repeatable or destruct testing methods will require thoughtful
determination of how the matched pairs will be created or selected for testing to
minimize the effect of confounding product variation or use conditions with the
measurement error.
Typical approaches are:
Create large homogenous master samples for bulk materials (wet or
dry)
Take a larger sample and separate it into smaller samples
Select sequential parts made under relatively the same conditions
Functional tests are performed under constant conditions of use.
This may require the consultation of a subject matter expert in measurement systems
analysis and/or testing methods. This structure will be recorded in the validation plan
and final report.
Requirements for sensitivity, specificity (or false positive & false negative rates) and
equivalence must be determined prior to running the testing and analyzing the results.
These requirements are necessary for determining the appropriate sample size and
experimental structure.
Repeatability and reproducibility results are intended to be used as a guide in determining
the final measurement protocol; there are no specific results that dictate usability. If a
specific system requires a specific level of performance (e.g. inspector qualification) that
requirement will be documented in the governing work instruction or SOP for the process.
Any screening test for input factors that is initiated in response to a functional failure must
include correlation of the input factor to the functional failure mode.
Only invalid test results may be excluded from any analysis. They must still be recorded in
the final validation report along with the rationale for claiming that the test was in fact
invalid. (instrument error, impossible value obtained, known error in sample prep, etc.)
All results must be reported with quantification of statistical significance and practical
importance. All p values must be described as one tailed or two tailed with appropriate
scientific rationale for either.
CONTINUOUS DATA
Repeatability
The standard sample size is 30 units measured twice randomly for each operator.
Expensive, difficult or destruct testing may require smaller sample sizes and the rational for
smaller sample sizes is to be documented in the validation plan.
Selected samples should span the full range of expected variation.
Repeatability of each the test equipment should be established using a single operator.
Results are to be plotted on a Youden Plot and the measurement error,
e
, the Intraclass
Correlation Coefficient, (
2
T
2
e ICC
- 1 =
) and the Discrimination Ratio, D
R
(
( ) ( ) + 1 / 1 D
R
) are to be reported.
Specification Limits will be displayed on the plot to provide context for the measurement
error and total observed variation in the study.
Youden Plot: The Youden plot is a square scatter diagram with equal X and Y axes; the
axes are the same physical length and the same scale. The Youden plot also has a 45 line
originating at zero. Measurement error,
e
, is the scatter in the direction that is
perpendicular to a 45 line. When used for graphical display of continuous measurement
data, the plot may be specifically referred to as a
Measurement Discrimination Plot.
The ideal measurement system has a small
measurement error in relation to the specification
range, if the units assessed in the study span the full
range of the specification range, the Discrimination
Ratio will be fairly large (D
R
> 7). However, some
measurement systems can still be valuable with
lower discrimination ratios. Depending on the
criticality of the system guard banding may be
necessary. In these cases, consult a statistician or
member of the Operational Excellence team to
determine the best approach.
Measurement Stability
Each Youden Plot will include also a Measurement Stability analysis to determine if the
repeated measurements were statistically stable. Unstable results are an indication that one
or more test pairs may have an invalid reading causing a large difference between the pairs.
These values should be investigated for possible exclusion and investigation of the
measurement system.

Measurement Discrimination Plot
0.0150
0.0200
0.0250
0.0300
0.0350
0.0150 0.0200 0.0250 0.0300 0.0350
Weight of Fluid Filled in Tube
W
e
i
g
h
t

o
f

F
l
u
i
d

R
e
m
o
v
e
d

f
r
o
m

T
u
b
e
Example Youden Plot
Measurement stability is assessed with a basic Range Control chart assessing only the
upper limit: R D UCL
R 4
where D
4
= 3.267 (for n = 2)
Any values outside the upper control limit would indicate a statistically significant
difference in measurement error for that pair. If any extreme values can be verified as
being invalid (typographical error, physically impossible, or an invalid test condition) the
pair may be removed from the analysis. Pairs may not be removed simply because they
are outside of the control limits (extreme values); pairs may only be removed if they are
verified as being invalid results.
Example of Youden Plot and Range Chart for Detecting Extreme Values:
Measurement Discrimination Plot
60
65
70
75
80
60 65 70 75 80
Measurement Range Chart
0.00
0.50
1.00
1.50
2.00
2.50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Reproducibility
Reproducibility is assessed when more than one operator will be performing the test or
when a new operator is assigned to the test as a qualification for performing the test.
Reproducibility analysis will only be performed after each operator demonstrates adequate
repeatability.
If the original R&R sample set is not available for future operators, reproducibility can be
established by having one or more of the current operators included in the study to enable
the necessary operator to operator comparison.
Reproducibility analysis will include a Youden plot with the first result for each operator.
An Analysis Means (ANOM) approach will be taken to assess the statistical significance of
operator to operator differences. The practical importance of any reproducibility
difference is assessed qualitatively by comparison of the difference to the specification
limits on the Youden Plot.
Example of Youden Approach for Assessing Reproducibility:
Measurement Reproducibility
Operator-Operator
0.1310
0.1320
0.1330
0.1340
0.1350
0.1360
0.1370
0.1380
0.1390
0.1400
0.1410
0.1310 0.1320 0.1330 0.1340 0.1350 0.1360 0.1370 0.1380 0.1390 0.1400 0.1410
Measurement 1
M
e
a
s
u
r
e
m
e
n
t

2
Operator 1
Operator 2
Operator 3
Operator 4
45-degree line
Spcification Limits
Operator Averages
0.1340
0.1345
0.1350
0.1355
0.1360
0.1365
Operator 1 Operator 2 Operator 3 Operator 4
A
v
e
r
a
g
e
Method Comparison
Method comparison is performed when assessing a replicate, alternate or replacement
measurement system. Method comparison is also appropriate when assessing the
difference between systems or methods in different locations such as a vendor & IQA or
IQA & Manufacturing.
If method comparison is required, repeatability and reproducibility will be established for
each method individually then the two methods will be compared. Analysis will include
Youden plots for repeatability of each system and either a Bias Youden or a Bland-Altman
plot to display the size of the bias between the methods. In a Bias Youden plot the bias is
the difference between the 45 degree line and a median line through the data cloud that is
parallel with the 45 degree line. In a Bland Altman plot the average difference is plotted as
the center line. Tolerance limits are then plotted about the average difference line.
Typically the tolerance limits are + 2
d
, where
d
is the standard deviation of the
differences of each of the pairs. The plotted points are the differences of the individual
pairs.
Example of Youden and Bland Altman Analyses for Method Comparison:
Statistical significance of the bias will be assessed by a paired t-test. The required sample
size will take into account the size of allowable bias or disagreement. The calculations
performed to determine sample size will be included in the plan and final report.
Measurement System 1 vs. Measurement System 2
Method Comparison
-0.005
-0.004
-0.003
-0.002
-0.001
0
1.171 1.1715 1.172 1.1725 1.173 1.1735 1.174
Average Value Between Systems
D
i
f
f
e
r
e
n
c
e

(
S
y
s
t
e
m

2

-

S
y
s
t
e
m

1
)
Difference
Center
UTL
LTL
Measurement Systems Comparison
1.166
1.168
1.170
1.172
1.174
1.176
1.178
1.166 1.168 1.170 1.172 1.174 1.176 1.178
System 1
S
y
s
t
e
m

2
Bias
Bias = Center Line
Youden Bias Plot Bland Altman Method Comparison Bias Plot
CATEGORICAL DATA
Design and verification testing may be performed with a non-representative distribution of
units, particularly marginally passing and failing units to optimize the test and set the cutoff
limit(s).
All final validation testing for repeatability, reproducibility and method comparison must
have a sample distribution of passing and failing units that is representative of the actual
distribution of the process stream that is being tested. This is particularly important when
assessing methods that are highly operator sensitive such as a visual inspection. If this
distribution is not known the technique or assumptions utilized to estimate the distribution
are to be documented in the validation plan and final report.
False Acceptance & False Rejection Rates
Each test must have a stated goal for acceptable (not to exceed) false acceptance and false
rejection rates. These rates will be dependent on the purpose of the test and the
consequences of a false result.
False acceptance and false rejection rates are determined by testing units with known
pass/fail status (truth). If truth is unknown every effort should be made to establish truth.
This may be done by utilizing a known gold standard method, functional testing, or
repeated testing to determine scientific and statistical consensus (most probable truth). In
the case where human judgment is involved, especially when the criteria utilizes an
operational definition and not a measureable physical characteristic, an expert panel may be
utilized to determine truth. (e.g. visual inspection of cosmetic defects, functional
anomalies or binning of customer complaints). The method for determining truth will be
documented in the validation plan and final report.
The required sample size will take into account the necessary accuracy of the estimate of
the false results. The calculations performed to determine the sample size will be included
in the plan and final report.
The general format for the analysis is a 2
x
2 table. The critical statistics are the false
acceptance rate and the false rejection rate.
Example Analysis of False Acceptance and Rejection Rates:
False Acceptance
Pass Fail
Pass 96 2 98 4%
Fail 4 98 102 2%
100 100 200
False Rejection
Truth
False Rejection Rate
False Acceptance Rate
Observed
Confidence intervals are to be calculated and reported against the goals for false results.
Typically this calculation will require the Exact Binomial Confidence Limits as the rates
will be fairly low and the Normal approximation to the Binomial will not be appropriate.
A final assessment of the false acceptance and rejection rates given the actual process
distribution is to be included in the final report. This is particularly important if the
validation test samples were not representatively distributed.
The estimated false rates that will be experienced are calculated by:
Expected False Acceptance Rate = Validation False Acceptance Rate x Proportion
of Failing units in the Process Stream
Expected False Rejection Rate = Validation False Rejection Rate x Proportion of
Acceptable units in the Process Stream
The false result rates have no predictive value unless the equipment and operators are also
shown to be repeatable and reproducible.
Repeatability & Reproducibility
Each test system and operator for critical tests must be individually qualified by
repeatability testing. Multiple operators and/or replicate test systems must have
reproducibility studies.
The general statistic of interest is the Kappa test which provides a measure of %agreement
adjusted for the distribution of failing and passing units. The Kappa statistic will be
reported with its corresponding confidence limits in the final validation report. In general a
Kappa statistic of > 0.95 is required.
The sample set for the final validation Repeatability and Reproducibility study should be
selected such that the distribution of passing and failing units is representative of the actual
process distribution.
Repeatability and Reproducibility have no useful value if the system is not also accurate to
truth.
Method Comparison
Method comparison is a special form of reproducibility that is performed when assessing a
replicate, alternate or replacement measurement system. Method comparison is also
appropriate when assessing the difference between systems or methods in different
locations such as a vendor & IQA or IQA & Manufacturing.
If method comparison is required, repeatability and reproducibility will be established for
each method individually then the two methods will be compared.
Statistical significance of any difference will be assessed by McNemars test: The Chi
Square test can be used for n > 20. The exact Binomial p value will be used for smaller
sample sizes, n < 20.
Example Analysis of Statistical Significance of a Difference:
Pass Fai l
Pass
a
25
b
5
Fail
c
15
d
55
Original Test Stand
New Test
Stand
If there is a statistically significant difference, the practical importance of the difference
(bias) will be assessed by the estimated difference and its control limits vs. the pre-
established acceptable difference. The required sample sizes, passing and failing unit
distribution and the acceptable bias will be established prior to performing any testing and
will be documented in the validation plan and final report.
Example Analysis of the Size of the Difference:

one tailed two tailed
p
BINOMIAL
0.00591 0.01182
p
2
0.02209 0.04417
95% LCL 95% UCL
0.015 0.1 0.185
Pass Fail
Pass
a
25.0%
b
5.0%
Fail
c
15.0%
d
55.0%
New Test
Stand
Original Test Stand
Method Comparison: Original vs. New Test Stand
0.015
0.1
0.185
Max Allowable
Difference,
0.05
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
P
r
o
p
o
r
t
io
n

D
if
f
e
r
e
n
c
e
Appendix: Formulas
CONTINUOUS DATA
Measurement Error,
e
:
The Range of measurements of each unit, R
i
R
i
= |X
A
X
B
|
The Average Range of the sample set,
R
:
k
R
R
i

The Repeatability or Measurement Error,

e
:
2
e
d
R

Where k = the number of units in the sample set
d
2
= 1.1.28 (for n = 2)
Total Measured Variation,
T
:
The average value of each unit,
i X
:
2
X X
X
B A i i
i
+

The product variation,

P:
( )
1
X X
2

n
i
P

The total variation,

T
:
2 2
2
1
e P T
+
Measurement Discrimination Formulas:
The Intraclass Correlation Coefficient, : 2
T
2
e
ICC
- 1 =

The percentage of the total variation that is due to product variation is x 100.
%
p
= x 100
The percentage of the total variation that is due to measurement error is (1-) x 100
%
e
= (1- ) x 100
The Discrimination Ratio is the number of distinct categories that the observed process
variation can be divided into given the measurement error and the amount of observed
variation.
1
2

- 1
+ 1
D
2
e
2
T
R

Method Comparison Formulas, Continuous Data

Difference between replicates, d
i
:
i i i
a b d
(where a and b are the two respective readings of a single sample)
Mean of replicates,
i
X :
2
) (
i i
i
b a
X
+

Centerline,
d
:
n
d
d
i

Standard deviation of the differences,

d:

1
) (
2

n
d d
i
d

Upper and Lower Limits:

d
d 2 t
The Bland Altman plot is created by plotting the mean of each pair,
i X
, on the X axis and
the corresponding difference, d
i
on the Y axis for each pair of measurements.
CATEGORICAL DATA
False Acceptance & False Rejection Rates
The false rejection rate =
units passing true #
units rejected falsely #

The false acceptance rate =
units failing true #
units accepted falsely #

Kappa Score, single rater repeatability and two raters reproducibility with two
categories:
Pass Fail
Pass 355 0 355
Fail 3 42 45
358 42
PA Pass & B Pass
Pass Fail
Pass 0.888 0.000 0.888 PB Pass
Fail 0.008 0.105 0.113 PB Fail
0.895 0.105
PA Fail & B Fail
PA Pass PA Fail
Measurement A
Measurement
B
Measurement A
Measurement
B
P
Observed
= P
A_Pass &B_Pass
+ P
A_Fail & B_Fail
P
Chance
= (P
A_Pass
x P
B_Pass
) + (P
A_Fail
x P
B_Fail
)
Chance
Chance Observed
P 1
P P
K

Kappa Score, multiple raters and multiple categories:

Overall Kappa Score:
( )

k
j
j j
k
j
ij
n
i
overall
q p m nm
x m n
K
1
1
2
1
2
) 1 (
1
Kappa Score by Category:
( )
( )
j j
n
i
ij ij
category
q p m nm
x m x
K
1
1
1

n = number of samples
m = the number of judgers, raters or inspectors
k = number of categories
x = an individual rating = 1
p = # ratings within a category/(nm)
q = 1- p
McNemars Test for Differences Between Two Correlated (Dependent ) Results
Calculating p values for the statistical significance of the difference (the difference is
between cells b and c)
One Tailed Exact Binomial (in Excel):
p
Binomial
= 1-BINOMDIST(max(b,c),n
b+c
,0.5,True)
max count(b,c) is the larger of the two cell counts in b and c
n
b+c
is the sum of the cell counts in b and c.
0.5 is Binomial probability given that if there were no real difference the count of b will
equal the count of c.
True = cumulative probability
One Tailed Chi-Square Test (in Excel):
p
2
=
2
CHIDIST(
2
,1)

2
=
( )
( ) c b
1 c b
2
+

degrees of freedom (df) = 1
Confidence Intervals for the Size of the Difference of Two Correlated Results
n = a + b + c + d
= |P
A1
P
A2
|

e
=
( ) ( ) ( )
n
P P P P 2 P 1 P P 1 P
c b d a 2 A 2 A 1 A 1 A
+
Confidence Limits =
e
2
z

t

n
) c a (
P
1 A
+

n
) b a (
P
2 A
+

n
a
P
a
,
n
b
P
b
,
n
c
P
c
,
n
d
P
d

Pass Fail
Pass a b
Fail c d
2
nd
Measurement
1st Measurement
SAMPLE SIZE FORMULAS
Continuous Data
2
2
z
n

,
_

or
( )
2
z z
n
1
]
1

: the standard deviation of the population you are trying to estimate
: the amount of accuracy you want; this is the minimum amount of detectable
difference from the true mean.
is always expressed as a number, not a percent.
Note that + = the Confidence Interval.
: alpha is the probability of detecting a difference that doesnt exist, typically .05 or .01
: beta is the probability of not detecting a difference that does exist, typically 0.01
Categorical Data
Categorical data for the proportion of defective events generally follows a Binomial
Distribution.
( )
2
2
p 1 p z
n

or
( ) ( )
2
2 2 1 1
p 1 p z p 1 p z
n
1
1
]
1

Rare events (<5%) will require iteration using confidence intervals for the Exact Binomial.
A starting point for the iteration is to use the Exact Binomial formula:
ln: natural log
P(Detection): the probability of detecting a defect rate that is = p, typically. 95 or .99
p: the expected defect rate or occurrence rate of the process; may be the minimum rate
which we want the system to reliably detect.
Confidence 80% 90% 95% 99.0% 99.9% Power
0.2 0.1 0.05 0.01 0.001
z

0.842 1.282 1.645 2.326 3.090 z

z
/2
1.282 1.645 1.96 2.576 3.291
) 1 ln(
)) ( 1 ln(
p
Detection P
n

Appendix: References
Donald J Wheeler, Craig Award Paper, Problems with Gauge R&R Studies, 46th
Annual Quality Congress, May 1992, Nashville TN, pp. 179-185.
Youden, William John, Graphical Diagnosis of Interlaboratory Test Results,
Industrial Quality Control, May 1959, Vol. 15, No. 11
Donald J Wheeler, Richard W Lyday, Evaluating The Measurement Process, Second
Edition, SPC Press, 1988
Donald S. Ermer and Robin Yang E-Hok, Reliable Data is an Important Commodity,
The Standard, ASQ Measurement Society Newsletter, Winter 1997, pp. 15-30.
Donald J Wheeler, An Honest Gauge R&R Study, Manuscript 189, January 2009.
http://www.spcpress.com/pdf/DJW189.pdf
Donald Wheeler, How to Establish Manufacturing Specifications, ASQ Statistics
Division Special Publication, June 2003, http://www.spcpress.com/pdf/DJW168.pdf
Bland, Martin, J., Altman, Douglas, G., Statistical Methods For Assessing Agreement
Between Two Methods Of Clinical Measurement, The Lancet, February 8, 1986
Dietmar Stockl, Diego Rodrguez Cabaleiro, Katleen Van Uytfanghe, Linda M. Thienpont
Interpreting Method Comparison Studies by Use of the BlandAltman Plot:
Reflecting the Importance of Sample Size by Incorporating Confidence Limits and
Predefined Error Limits in the Graphic, Letter to the Editor of Clinical Chemistry, 50,
No. 11, 2004
Prond, Paul, and Ermer, Donald S., A Geometrical Analysis of Measurement System
Variations, ASQC Quality Congress Transactions Boston, 1993
Morris, Raymond A., and Watson, Edward, F., A Comparison of the Techniques Used
to Evaluate the Measurement Process, Quality Engineering, 11(2), 1998, pp. 213-219
Futrell, David, When Quality is a Matter of Taste, Use Reliability Indexes, Quality
Progress, Vol. 28, No. 5, May 1995, pp. 81-86
Agresti, Alan, An Introduction to Categorical Data Analysis, John Wiley & Sons,
1996