Intermediate Precision Evaluation

Posted on 2021-05-19 In Biometrics , BiomedicalStats

CLSI EP05A3 and EP15A3 as the reference

Definition of Intermediate Precision：

Intermediate precision (also called within-laboratory or within-

device) is a measure of precision under a defined set of conditions:
same measurement procedure, same measuring system, same
location, and replicate measurements on the same or similar objects
over an extended period of time. It may include changes to other
conditions such as new calibrations, operators, or reagent lots. ——
Intermediate precision

Take throwing darts as an example：

Accuracy: The score you get from the target of darts. The higher the
score, the better.

Precision: The distribution of the score. If you get a very close

location, it means your technique is very stable.

Concepts

If you want to estimate the precision for a certain test, these three
indicators are useful to figure out whether it’s good enough for
using.

%CV coefficient of variation expressed as a percentage

%CVR repeatability coefficient of variation

%CVWL within-laboratory coefficient of variation

We all know that it’s impossible to ensure every test is equal as

there are so many factors that would influence our results, such as:

Day

Run

Reagent lot

Calibrator lot

Calibration cycle

Operator

Instrument

Laboratory

The first two of the above are usually the main factors to be
considered.

Design

So There is always some variants in the measured results compared

to real values. It consists of systematic error (bias) and random
error. Precision measures random error.

In a single-site 20x2x2 study with 20 days, with two runs per day,
with two replicates per run. The associated factors including days
and runs will be involved in the statistical analysis, which it used to
estimate the two types of precision: repeatability (within-run
precision) and within-laboratory precision (within-device precision)
Once the source of variation has been identified, ANOVA model can
be used to calculate the SDs and %CVs in the statistical processing
of the data. Usual factor can be divided into three components:

Within-run precision (or repeatability), measures the results from

replicated samples for a given sample, in a single run, with the
essentially constant situation. This variation may be basically
caused by random error happening inside the instrument, such as
variation of pipetted volumes of sample and reagent.

Between-run precision, measures the variation from different runs

(e.g. run1 and run2). This run factor may cause the operation
conditions to change, such as temperature, instrument status etc.

Between-day precision, measures the variation happening between

days, which is easy to understand, such as caused by humidity etc.

Single Site Precision Evaluation Study

This protocol (20x2x2) is to estimate the repeatability (within-run)

and within-laboratory (intermediate precision) following CLSI EP-15.

From the description above, we can find the protocol is a classic

nested (hierarchical) design, where replicates are nested within runs
and runs are nested within days. So in this situation, nested ANOVA
is appropriate. If two factors are involved, corresponding to two-way
nested ANOVA.

To estimate the precision of this single-site 20x2x2 design, we

should follow a nested linear components-of-variance model
involving two factors: “day” and “run”, with “run” nested with “day”.
I think this model can be analyzed using the two-way nested ANOVA.
It should be noted that the design is balanced because it specifies
the same number of runs for each day, and the same number of
replicates for each run.

CLSI_precision

The above screenshot from CLSI EP05-A3 can help us to understand

the nested linear components-of-variance model. We can especially
know that the residual in the model represents the within-run factor.

Fit a random effects model

Nested random effects are when each member of one group is

contained entirely within a single unit of another group. The
canonical example is students in classrooms Crossed random effects
are when this nesting is not true. An example would be different
seeds and different fields used for planting crops. Seeds of the same
type can be planted in different fields, and each field can have
multiple seeds in it.

Whether random effects are nested or crossed is a property of the

data, not the model. In the other word, you should tell the model
which data is nested or crossed.

I don’t describe the experiment and workflow in this section, which

can be found in the CLSI EP05 and EP15 documents clearly.
Data analysis by ANOVA

Let’s talk about how to calculate the %CV and SD that can be
divided into at least two categories based on how many factors are
involved.

The first step, I load a simple design(20x2x2) data from a R package

VCA including 2 replicates, 2 runs and 20 days from a single
sample，where y is the test measurements.

One reagent lot - a single sample

One instrument system

20 test days

Two runs per day

Two replicates measurements per run

library(VCA) data(dataEP05A2_2) > summary(dataEP05A2_2) day

run y

1 : 4 1:40 Min. :68.87

2 : 4 2:40 1st Qu.:73.22

3 : 4 Median :75.39

4 : 4 Mean :75.41

5 : 4 3rd Qu.:77.37

6 : 4 Max. :83.02

(Other):56

The second step, I use the nested ANOVA by aov function in R to fit
a nested linear components-of-variance model. In this situation, runs
are nested within days.

res <- aov(y~day/run, data = dataEP05A2_2)

ss <- summary(res)

> ss

Df Sum Sq Mean Sq F value Pr(>F)

day 19 319.0 16.787 4.512 3e-05 ***

day:run 20 187.4 9.372 2.519 0.00634 **

Residuals 40 148.8 3.720

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The third step, calculate the SD and %CV of the day, run and error
variation following the formula occurred in EP05-A3. By the way, the
error CV(CVerror) is corresding to %CVR, also called within-run or
repeatability precision. And the %CVWL is the within-laboratory
precision.

nrep <- 2

nrun <- 2

nday <- 20

Verror <- ss[[1]]$`Mean Sq`[3]

Vrun <- (ss[[1]]$`Mean Sq`[2] - ss[[1]]$`Mean Sq`[3]) / nrep

Vday <- (ss[[1]]$`Mean Sq`[1] - ss[[1]]$`Mean Sq`[2]) / (nrun *

nrep)

Serror <- sqrt(Verror)

Sday <- sqrt(Vday)

Srun <- sqrt(Vrun)

Swl <- sqrt(Vday + Vrun + Verror)

> print(c(Swl, Sday, Srun, Serror))

[1] 2.898293 1.361533 1.681086 1.928803

CVerror <- Serror / mean(dataEP05A2_2$y) * 100

> CVerror

[1] 2.557875

CVwl <- Swl / mean(dataEP05A2_2$y) * 100

> CVwl

[1] 3.843561

The fourth step, calculate the confidence interval of SD and %CV,

which is relying on the chi-square distribution value for DF
estimates. Use the CV% of error as an example.

CLSI_precision2

alpha <- 0.05

CVCI <- c(CVerror * sqrt(ss[[1]]$Df[3] / qchisq(1-alpha/2, df = 40)),

CVerror * sqrt(ss[[1]]$Df[3] / qchisq(alpha/2, df = 40)))

> CVCI

[1] 2.100049 3.272809

CVCI_oneSide <- c(CVerror * sqrt(ss[[1]]$Df[3] / qchisq(1-alpha, df =

40)), CVerror * sqrt(ss[[1]]$Df[3] / qchisq(alpha, df = 40)))

> CVCI_oneSide

[1] 2.166476 3.142029

Fortunately, above standard calculation steps have been packed
into a R package, that is the VCA package. So we just apply
anovaVCA function to fit the model and summarize the it. For CI
calculation, the VCAinference function could be used. It sounds so
good.

Fit model:

res <- anovaVCA(y~day/run, dataEP05A2_2)

res

> res

Result Variance Component Analysis:

-----------------------------------

Name DF SS MS VC %Total SD CV[%]

1 total 54.78206 8.400103 100 2.898293

3.843561

2 day 19 318.961943 16.787471 1.853772 22.068447

1.361533 1.805592

3 day:run 20 187.447626 9.372381 2.82605 33.643043

1.681086 2.229366

4 error 40 148.811221 3.720281 3.720281 44.288509

1.928803 2.557875

Mean: 75.40645 (N = 80)

Experimental Design: balanced | Method: ANOVA

Calculate CI for SD and %CV:

VCAinference(res)

> VCAinference(res)

Inference from (V)ariance (C)omponent (A)nalysis

------------------------------------------------

> VCA Result:

-------------

Name DF SS MS VC %Total SD CV[%]

1 total 54.7821 8.4001 100 2.8983 3.8436

2 day 19 318.9619 16.7875 1.8538 22.0684 1.3615 1.8056

3 day:run 20 187.4476 9.3724 2.8261 33.643 1.6811 2.2294

4 error 40 148.8112 3.7203 3.7203 44.2885 1.9288 2.5579

Mean: 75.4064 (N = 80)

Experimental Design: balanced | Method: ANOVA

> VC:
-----

Estimate CI LCL CI UCL One-Sided LCL One-Sided UCL

total 8.4001 5.9669 12.7046 6.2987 11.8680

day 1.8538

day:run 2.8261

error 3.7203 2.5077 6.0906 2.6689 5.6135

> SD:

-----

Estimate CI LCL CI UCL One-Sided LCL One-Sided UCL

total 2.8983 2.4427 3.5644 2.5097 3.4450

day 1.3615

day:run 1.6811

error 1.9288 1.5836 2.4679 1.6337 2.3693

> CV[%]:

--------

Estimate CI LCL CI UCL One-Sided LCL One-Sided UCL

total 3.8436 3.2394 4.7269 3.3282 4.5686

day 1.8056

day:run 2.2294

error 2.5579 2.1000 3.2728 2.1665 3.1420

95% Confidence Level

SAS PROC MIXED method used for computing CIs

These functions can be used to handle complicated design, so we

don't need to set up functions or a package any more.

Reference

Visualizing Nested and Cross Random Effects

R-Package VCA for Variance Component Analysis

How to Perform a Nested ANOVA in R (Step-by-Step)

Lab 8 - Nested and Repeated Measures ANOVA

What’s with the precision?

Please indicate the source: http://www.bioinfo-scrounger.com

Related Posts

Conduct an ANCOVA model in R for Drug Trial

Agreement statistics in diagnostic accuracy

Analysis of Reference Value

Bland-Altman Analysis

Calculation of follow-up time

# Biometrics

Definition of ANOVA, MANOVA and ANCOVA

Tips of pie donut chart and barplot