Measurement System Analysis

How-to Guide - Appendices

Version 6.1
August 2013
MSA How-to Guide

© 2013 Rolls-Royce plc

The information in this document is the property of Rolls-Royce plc and may not be
copied or communicated to a third party, or used for any purpose other than that for
which it is supplied without the express written consent of Rolls-Royce plc.
This information is given in good faith based upon the latest information available to
Rolls-Royce plc, no warranty or representation is given concerning such information,
which must not be taken as establishing any contractual or other commitment binding
upon Rolls-Royce plc or any of its subsidiary or associated companies.

2 | © 2013 Rolls-Royce plc

This appendices provides supplementary information on how to carry out analysis
using Minitab statistical software – together with some of the more detailed analysis of
the statistical output.

Step 11
Be Prepared

CONTINUOUS DATA: ATTRIBUTE DATA:

Appendix 1: Step 22 Appendix 6:
Setting up and Plan the Study Setting up and
randomising the randomising the
spreadsheet in Minitab spreadsheet in Minitab
Step 33
CONTINUOUS DATA: Conduct the Study ATTRIBUTE DATA
Appendix 2: Appendix 7:
Entering the data in Entering the data in
Minitab Minitab

Step 44
Type of Study

Appendix 3: Continuous Data: Attribute Data: Appendix 8:

Carrying out Gauge R&R for Attribute agreement Carrying out
Gauge R&R continuous data analysis for Attribute
in Minitab attribute data Agreement
Analysis in
Minitab

Appendix 4: Step 55 Appendix 9:

Supplementary Taking action if the results Supplementary
Information on are unacceptable Information on
Interpreting the Interpreting the Output
Graphical Output from from Attribute Agreement
Gauge R&R in Minitab Analysis
Step 66
Maintaining the improvement

© 2013 Rolls-Royce plc | 3

MSA How-to Guide

In This Section:

Continuous Data
1. Setting-up and Randomising the
Spreadsheet in Minitab

2. Entering the Data in Minitab

3. Carrying out Gauge R&R in

Minitab

4. Supplementary Information on
interpreting the output

5. FAQ for Gauge RR

4 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

Setting up and randomising the worksheet for a Gauge R&R Study for
Variable Data
1. The starting point for setting up the worksheet for a variable data Gauge R&R
study is the same regardless of which randomisation method for the
worksheet is required. To begin go to:
Stat > Quality Tools > Gage Study > Create Gage R&R Study Worksheet

2. Complete the dialogue box for the required detail:

1) Enter the quantity of

parts to be studied

2) Enter the
identity of
the parts to 3) Enter the
be used number of people
in the study

4) Enter the
identities of the
people in the
study

5) Enter the number of times that each person

will check each part © 2013 Rolls-Royce plc | 5
MSA How-to Guide
Continuous Data
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

3. Click on the options box

Options

This gives you 3 options to randomise the worksheet:

At this stage you must decide which randomisation method to use taking into
consideration the practicalities of running the experiment and the most economical use
of people’s time.

The different options are each described below.

6 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

a. Do not randomise: As it states, this option does not randomise the data. This
option will sequence the parts then the people for each part and provide a run
order column as shown below.

1) Use the ‘Options’ to

confirm selection 2) Make the selection and click
OK will then generate a
3) Note the sequence of worksheet for the study
parts and people

© 2013 Rolls-Royce plc | 7

MSA How-to Guide
Continuous Data

Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

b. Randomise all runs: This will completely randomise the order that the
measurements are taken in as shown below. This is useful to prevent the
appraisers from memorising their previous measurements and also to reduce the
impact of time related factors. It does however require all of the appraisers to be
present at once which can be impractical in many situations such as where
different shifts are worked.

As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)

order by selecting the option ‘Store standard run order in worksheet’.

1) Use the ‘Options’ to

2) Check the box to confirm selection
include standard
order column

3) Make the selection and click

4) Note the sequence of
OK will then generate a
parts and people
worksheet for the study

Following data collection and analysis the standard order can be used to re-sort the
recorded data so that the pattern of collection may give an insight into what happened.
This should only be done if the measurement system analysis study is not clearly
acceptable and in this case can be useful in identifying combinations which were
awkward for the appraisers.

8 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

c. Randomise runs within operator: This will prevent memory of measurements by

the people undertaking the study but preserves the appraiser sequence. This
enables a study appraiser’s time to be managed as only one appraiser needs to
be present at specified times. As a facilitator, it can also be useful to preserve
the ‘standard’ (non-randomised) order by selecting the option.

2) Check the box to

include standard order 1) Use the ‘Options’ to
column confirm selection

3) Make the selection and

4) Note the sequence of
click OK will then generate a
parts and people
worksheet for the study

This is the most commonly used option;

On completion of Note 4, Minitab returns you to previous dialogue box (Create Gage
R&R Worksheet) then press OK on this screen.

Minitab will now generate the worksheet, you will need to add in your data column and
collect the data before running the study.

© 2013 Rolls-Royce plc | 9

MSA How-to Guide
Continuous Data

Appendix 2: Entering the Data in Minitab

Maintaining Data Integrity

It is often overlooked that data integrity starts when the data is entered. In statistical
software such as Minitab it is common to see data formatting and entry errors causing
issues.

The two most common issues to be aware of are as follows:

1) Areas of the worksheet have been previously used OR the wrong sort of data
has been entered resulting in the column being in the wrong data format

A ‘space’ was typed in and turns

the column type to text

The wrong type of data format for columns then has the effect of hiding columns that
are expected to be numeric (or vice versa) when conducting an MSA study.

The most common occurrences of this is when a space is added somewhere in the
column or when the letter O is used instead of 0 (zero). In both cases, even if the
typing error is rectified this will change the format of the column from numeric to text
format.

Text format columns can be identified by the addition of a ‘T’ to the column number as
in the example above.

10 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 2: Entering the Data in Minitab

2) The second issue is that of human mistakes when entering the data.
To guard against this, the facilitator of the measurement study must control the
study to maintain the concentration, time, speed and discipline required to
type/record each data point.
In addition to this, it is possible to assist the person entering the data to select the
correct cell by highlighting the line (descriptions and details) of the active entry.

Left click on the

row number will
highlight the
complete row

The example shown has used the option ‘randomise runs within operators’ with the
next entry being from appraiser 2 for part identity 9.
It can also be very beneficial to record comments when entries are typed. This
additional information can be useful for analysis where the Measurement System is
not acceptable and further investigation is needed.

Data Type Considerations

Types of data are very specific to each MSA study. For Gauge R&R studies the data
is variable and has to be the same units of measure as the operating process.
For Attribute Agreement Analysis this is attribute data BUT this can be in the format of
whole (count or scale) numbers or text values.

© 2013 Rolls-Royce plc | 11

MSA How-to Guide
Continuous Data

Appendix 3: Running the Analysis

To run the analysis then use the menu commands:

Stat>
Quality Tools>
Gage Study>
Gage R&R Study (crossed)

12 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 3: Running the Analysis

A dialogue box will appear. Enter the data into the fields as shown below:

1. Minitab will display in 2. Click against ANOVA in “Method of

this window, appropriate Analysis” This should be the default as Minitab
elements of the opens the dialogue box, however should
worksheet for selection, always be checked
transfer the columns for
parts appraisers and
3. Then click OK
measurement data as
shown.

© 2013 Rolls-Royce plc | 13

MSA How-to Guide
Continuous Data

Appendix 3: Running the Analysis

Click on “Options” to enter the tolerance of the characteristic being

measured. For our example this is 0.5mm.

Enter 0.5 into the “Upper spec – Lower spec”

Click on
“Do not display percentage contribution” & “Do not display percentage study
variation”.

Click “OK” only once.

[This simplifies the graphical output to remove graphs not necessary for our
analysis]
14 | © 2013 Rolls-Royce plc
 Continuous Data

Appendix 3: Running the Analysis

Click on “Gage Info” to enter the relevant equipment, team and

information for the study. It is also good practice to record the date of
the study for future reference

Fill out the details requested

Click “OK” then “OK” again.

© 2013 Rolls-Royce plc | 15

MSA How-to Guide
Continuous Data

Appendix 3: The Numerical Output

The Graphical output will appear as below.

Click on Show sessions folder icon to review the numerical

output”

Gage R&R (ANOVA) for Measurement

Reported by : HA S F A D
G age name: v ernier caliper Tolerance:
Date of study : 16th A ug 2006 M isc: M easurement S y stem A naly sis

Components of Variation Measurement by Parts

% Tolerance
200 2.00
Percent

1.75
100
1.50
0
Gage R&R Repeat Reprod Part-to-Part 1 2 3 4 5 6 7 8 9 10
Parts
R Chart by Appraiser
Measurement by Appraiser
1 2 3
Sample Range

2.00
0.10
UCL=0.0789
0.05 _ 1.75
R=0.0307
0.00 LCL=0
1.50
1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Parts 1 2 3
Appraiser
Xbar Chart by Appraiser
1 2 3 Parts * Appraiser Interaction
2.00
Sample Mean

_
_ 2.00 Appraiser
UCL=1.8504
X=1.819 1
LCL=1.7876
Average

1.75 2
1.75 3
1.50

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
1.50
Parts 1 2 3 4 5 6 7 8 9 10
Parts

16 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

In the following appendix the results and interpretation is explained for each of the six
graphs in the Minitab Graphical Output. Please note that all of the statistical analysis and
graphs should be considered before making conclusions for the study and for the potential
actions required. Note also that ‘no action required’ is a possible and valid conclusion.

The first graph to look at is the “Components of Variation” on the top right
of the graphical output.

This graph shows where most of the variation in the study came from.
The Gauge R&R column shows the % Tolerance taken up by the
measurement system variation. Remember this was 76.99%

If the Part-to-Part columns are high (or very high compared to the others)

Components of Variation
this tells us that most of the variation in the study was due to the fact that
the parts being measured were not identical (which we would expect).

If the Repeat columns are high compared to the others, this indicates that
there is a problem with Repeatability (i.e. one or more of the appraisers is
inconsistent with themselves). The remaining graphs will help us
investigate this further.

If the Reprod columns are high compared to the others, this indicates that
there is a problem with Reproducibility (i.e. some of the appraisers are
inconsistent with each other). The remaining graphs will help us
investigate this further.

The Gauge R&R column is the variation component total for Repeat and
Reprod.

In cases such as this example where a problem is identified with the

repeatability and/or the reproducibility of the measurement system then
the remaining graphs should be examined to investigate further.
Where no problem is identified from the analysis of the components of
variation then there is no need to examine the remaining graphs
© 2013 Rolls-Royce plc | 17
MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – Components of Variation Graph

* 30% is a generally used acceptance criteria, however manufacturing standards may

have tighter requirements. Be sure to consult the relevant measurement standards for
your area. Details are contained within the SABRe Supplier Management System
Requirements document.

18 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Next we will look at the R Chart by Appraiser graph.

This chart shows the Range of the results for each appraiser for each of the 10 parts.

Interpreting the Graphical Data

For a perfectly consistent measurement system, all of the ranges on

the graph would be zero i.e. each part would be measured the same
giving no (zero) range.

Graphical Data
However, it is unlikely that they will all be zero, therefore we use this
chart to help us identify any measurements of concern.

We interpret this graph by saying that any point which is above the
upper red line is worth investigating, as this indicates that the range of
results for that appraiser and part was higher than expected.

So in the case study example we can see that appraiser 2 has a

bigger range than the other appraisers. Julie and the team make note
to ask appraiser 2 if they did anything different from the instructions
given and move on the next graph

Refer to SPC “How to” for more information on R charts.

© 2013 Rolls-Royce plc | 19

MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – R Chart by Appraiser

20 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Next Julie and the team look at the Xbar Chart by Appraiser graph –
this is the graph from the case study.

Interpretation of graphical output

This graph shows the average measurement for each part and
appraiser.

Ideally we want the patterns of the data to be identical for all 3

appraisers. If they are not, we should investigate.

The 2 red lines on the chart are control limits. We would expect all at
least 50% of the points to lie outside the control limits (red lines) on
this chart. This is different and the opposite to the conventional use of
SPC charts.

As we can, even though we do have the majority of parts outside the

control limits we can also see the patterns for each appraiser look
different. This is again an indication of poor Reproducibility.

Julie and the team note the results of this graph and ask each
appraiser what they did to identify differences then move to the next
one.

If you are not sure on interpreting control limits on an Xbar chart then ask a
local Black Belt to help you choose the most appropriate type of MSA.

© 2013 Rolls-Royce plc | 21

MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – Xbar Chart by Appraiser

22 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Next Julie and the team look at the Measure by Part graph – this is the graph from the
case study.

Interpreting the Graphical Data

So in the case study example we can see that:

This graph shows circles for all measured values of each part,
together with the average values for each part (shown by ‘crossed
circles’.)

Interpretation of graphical output

The average values (crossed circles) are connected by the straight
lines

The graph allows us to compare how consistent the measurements

for each of the parts were in the study.

If the measurement system is consistent, there should be very little

scatter between the measurements for each individual part (in other
words, the circles for each part should almost be on top of each other
or overlapping).

We interpret this graph by saying that any part for which there is a
noticeably larger spread in the results, might be worth investigating.
In this case parts 10, 8, 4 & 5 appear to have greater variation than
the rest of the parts. The team need to consider why these parts were
more difficult to measure? Julie and the team note this finding and
move on to the next graph.

© 2013 Rolls-Royce plc | 23

MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – Measurement by Parts

24 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Interpreting the Graphical Data

Next we will look at the Measurement by Appraiser graph:

Interpretation of graphical output

This graph shows a box plot for all measured values of each item,
together with the average measurement for each appraiser (shown by
‘crossed circles’).

The average values (crossed circles) are connected by the straight

lines. If the measurement system is perfectly consistent, we would
expect the average values for the 3 appraisers to be the same – in
which case the connecting lines would be horizontal.

We would also expect the spread of results (boxes and whiskers) for
all 3 appraisers to be the same (however, unlike the previous graph,
we wouldn’t necessarily expect the spread to be small, as the results
for all of the parts are shown against each appraiser).

We interpret this graph by saying that if, for any appraiser, there is a
noticeably larger spread in results, or the average value is noticeably
different from the others, this might be worth investigating.

So in the case study example we can see that appraiser 2 has a

larger spread of result than the other appraisers. We can also see the
average values line is not straight indicating a Reproducibility
problem.

Again Julie and the team note the findings and move onto the final
graph.

© 2013 Rolls-Royce plc | 25

MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – Measure by Appraiser Graph

26 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Interpreting the Graphical Data

Finally, we will look at the Part*Appraiser Interaction graph:

Interpretation of graphical output

This graph overlays the average measurements for each item as
measured by each person.

If the measurement system is perfectly consistent, we would expect

all of the lines to be on top of each other so only one line is seen.

Overlaying lines is the ideal situation here, however there can be

occasions when parallel lines occur. This would indicate that the parts
and appraiser interaction is consistent BUT bias between appraisers
exists.

We interpret this graph by saying that if any of the lines is noticeably

separate from the other 2 lines (for one or more of the parts), this is
worth investigating.

Julie and the team agree that this graph confirms some of their
thoughts from the previous graphs. They are now ready to summarise
their findings.

© 2013 Rolls-Royce plc | 27

MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Summary – Part*Appraiser Interaction

28 | © 2013 Rolls-Royce plc

 Continuous Data

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Use of Gauge Run Charts

One additional graphical tool which can be used to assess differences in
measurements between different operators and different parts is a Gauge Run Chart.
You can use a gauge run chart in combination with a Gage R&R Study to help
determine what is causing the variability in the measuring system.

Create a gauge run chart as follows:

1. To begin go to Stat > Quality Tools > Gage Study > Gage Run Chart

2. Complete the dialogue box for the required details

Click on ‘Gage Info’ to

enter the relevant
equipment and study
references

Enter the columns for

parts appraisers and
measurement data as
shown.

Then click ‘OK’ If known the historical process mean

can be entered and will be plotted as
a reference line on the graph
© 2013 Rolls-Royce plc | 29
MSA How-to Guide
Continuous Data
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies

Use of Gauge Run Charts

The graphical output will appear as shown:

This is a plot of all of the observations by operator (denoted by different colours) and
by part number (each box numbered 1 – 10 represents one of the 10 parts).

The horizontal reference line is the overall mean of the measurements.

The plot allows you to see if any patterns are evident in the data. For instance, you
might see that one operator consistently measures higher than the others or that the
measurements on certain parts vary more when compared to other parts.

Here for example you can see that for part 4, appraiser 1 (in black) has measured
higher than the other two appraisers. You can also see for part 10 noticeable
difference between the measurements of the three appraisers.

Looking at the repeatability within appraisers, for parts 2 and 8 it can be seen that
appraiser 2 (in red) noticeably differs in their three measurements indicating a
repeatability problem.
30 | © 2013 Rolls-Royce plc
 Continuous Data

Appendix 5: FAQ for Gauge R&R (Typical Manufacturing Questions)

Q: “Where can I find the acceptance criteria for MSA in manufacturing?”

A: “The criteria and other information are contained within SABRe Supplier
Management System Requirements document. Further guidance information on
measurement and inspection is available on the supplier portal, specifically in the Guide
to Dimensional Measurement Equipment document.”

Q: “We only have 2 operators using this gauge, shall I take part in the study to
make the numbers up?”
A: “Not unless the operator is trained to use the gauge and familiar with the component
being measured. Having an untrained operator may result in the failure of the study due
to the stability and/or reproducibility of the measurements due to the untrained operator.
It would be far better to compromise on the amount of measurement readings than to
perform a study that is not representative of the way the process works”

Q: “We cannot get access to 10 parts, will 5 do?”

A: “Lowering the sample size will affect the uncertainty of the test. However there are
times when this may be required. In difficult situations a compromise may be required
but the analyst should be mindful of this when interpreting the data. For example if a
gauge R&R returns 19% of tolerance with only 5 parts there is a reasonable case for
either acquiring further parts for study or asking the operators to repeat the
measurements 4 or 5 times rather than the usual 3”

Q: “We can’t get parts that represent the full process variation as the only ones
we have are from the same batch. What should we do?”
A: “If the sample does not represent the true process variation then this will affect the
results of gauge R&R against study variance, the number of distinct categories and the
limits on the X bar chart on the R&R output.
If this is the case 2 options exist. Either proceed and study the % Gauge R&R against
tolerance.”

Q: “Our gauge measures hundreds of features, do I have to run gauge R&R study
on each of them or can I use read-across methodology?”
A: “The Quality Management System requires that all product features/characteristics
are measured using capable measurement systems. That said there are situations
where similar features measured with the same gauge presents an opportunity to
demonstrate capability without direct study of every single feature as long as this is
done robustly. This must however be done in a robust and traceable way. Read across
is not permitted on CMM equipment, not because the CMM’s tend to be incapable but
because of the risk of program errors due to the manual nature of the program
creation.”
© 2013 Rolls-Royce plc | 31
MSA How-to Guide
Continuous Data

Appendix 5: FAQ for Gauge R&R

Q: “The gauge is automated (there is no operator influence)! What do I do?”

A: “First be sure that there really is not any operator influence – for instance if there is
a setup process which is manual then this may lead to reproducibility problems. If the
gauge is completely automated then a study can be performed with only 1 operator.
The gauge R&R study will report only on the repeatability element of the gauge R&R. If
there is the opportunity to study two gauges simultaneously (e.g. 2 CMM’s) these can
be identified within the R&R study to allow the reproducibility due to different
equipment rather than operator.”

Q: “I have a surface finish gauge, and it keeps failing R&R. What do I do?”
A:”Some gauges are notoriously difficult to perform gauge R&R on. Some gauges are
the best available for a given measurement. In this situation contact a measurement
practitioner or Metrologist.”

Q: “Under what circumstances should I repeat the gauge R&R?”

A: “A gauge R&R should be repeated whenever the process (either measurement
process or manufacturing process) changes significantly or the part tolerance is
changed. Also when turnover of labour is high”

Q: “My gauge R&R against tolerance is very good but I only get 1 distinct
category. Is my measurement process good or not?”
A: “It is likely that the parts selected for the study are not representative of the total
process variation. This will affect the %R&R against study variance, the number of
distinct categories and the X bar chart. If the parts are representative then the
measurement process is not adequate for the application of SPC analysis as the
majority of the variation seen will be from the measurement system and not the
underlying process.”

Q: “How many decimal places on my gauge shall I use when conducting the
gauge study?”
A: “As a minimum you should ensure that the study represents the requirements on the
part drawing but the more the better. For instance if the drawing requires measurement
to 2 decimal places, run the gauge R&R to 3.”

Q: “My gauge passes the gauge R&R study. Is this all I need to consider?”
A: “No – gauge R&R will not highlight gauge bias. For instance it is possible to be
repeatibly wrong. Comparison with a known standard will enable you to study the
amount of gauge bias. Calibration is not done on production parts, real parts can
introduce large differences.”

32 | © 2013 Rolls-Royce plc

 In This Section:

Attribute Data
6. Setting-up and Randomising the
Spreadsheet in Minitab

7. Entering the Data in Minitab

8. Carrying out Attribute Agreement

Analysis in Minitab

9. Supplementary Information on
how to interpret the output

© 2013 Rolls-Royce plc | 33

MSA How-to Guide
Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

Attribute Agreement Analysis Worksheet Configurations

1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is

very similar to a Gauge R&R study where, the worksheet construction all start
from an identical point independent of which randomisation for the worksheet is
used. This is found at:
Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet

34 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

2. Complete the dialogue box for the required detail:

1) There are 4 drop down
selections here (see 2) Enter the quantity of
below) parts to be studied

3) Enter the
identity of
the parts to 4) Enter
be used and the
the text number
standard of people
in the
study

5) Enter
the
identities
of the
people in
the study

6) Enter the number of times each person

is to inspect each part
Dropdown Selection
Sample Standard/attribute unknown: This provides a worksheet very similar to that of a
Gauge R&R study but does need to have an additional column manually added so
that a ‘standard’ agreement can be compared to it. This is the most flexible option.
Sample Standard/attribute in text: This constructs the worksheet with the additional
column of text ‘standards’ for comparison, i.e. When comparing judgements such as
good or bad against the standard which is also stated as good or bad. This is the
option used in the following pages.
Sample Standard/attribute in numbers: This constructs the worksheet with the
additional column of numerical ‘standards’ for comparison, i.e. When judgements are
made using a scale (often 1 to 5 or 1 to 10), and the standard should be an exact
match.
Sample Standard/attribute in worksheet: This favours manual lists already in the
worksheet and provides selection of those column references for the study.

© 2013 Rolls-Royce plc | 35

MSA How-to Guide
Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

3. Click on the options box

Options

This gives you 3 options to randomise the worksheet:

36 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

a. Do not randomise: As it states, this option does not randomise the data. This
option will sequence the parts then the people for each part and provide a run
order column as shown below.

1) Use the ‘Options’ to

confirm selection

2) Make the selection and click

3) Note the sequence of OK to generate a worksheet for
parts, people and the the study
‘standard’

© 2013 Rolls-Royce plc | 37

MSA How-to Guide
Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

b. Randomise all runs: This will completely randomise the order that the
measurements are taken in as shown below. This is useful to prevent the
appraisers from memorising their previous measurements and also to reduce the
impact of time related factors. It does however require all of the appraisers to be
present at once which can be impractical in many situations such as where
different shifts are worked.

As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)

order by selecting the option ‘Store standard run order in worksheet’.

2) Check the box to

1) Use the ‘Options’ to
include standard order
confirm selection
column

3) Make the selection

4) Note the sequence of
and click OK to generate
parts, people and the
a worksheet for the study
‘standard’

Following data collection and analysis the standard order can be used to re-sort the
recorded data so that the pattern of collection may give an insight into what happened.
This should only be done if the measurement system analysis study is not clearly
acceptable and in this case can be useful in identifying combinations which were
awkward for the appraisers.

38 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab

c. Randomise runs within operator: This will prevent memory of measurements by

the people undertaking the study but preserves the appraiser sequence. This
enables a study appraiser’s time to be managed as only one appraiser needs to
be present at specified times. As a facilitator, it can also be useful to preserve
the ‘standard’ (non-randomised) order by selecting the option.

2) Check the box to

include standard order 1) Use the ‘Options’ to
column confirm selection

3) Make the selection and

4) Note the sequence of click OK to generate a
parts, people and the worksheet for the study
‘standard’

This is the most commonly used option.

© 2013 Rolls-Royce plc | 39

MSA How-to Guide
Attribute Data

Appendix 7: Entering the Data in Minitab

Maintaining Data Integrity

It is often overlooked that data integrity starts when the data is entered. In statistical
software such as Minitab it is common to see data formatting and entry errors causing
issues.

The two most common issues to be aware of are as follows:

1) Areas of the worksheet have been previously used OR the wrong sort of data
has been entered resulting in the column being in the wrong data format

A ‘space’ was typed in and turns

the column type to text

The wrong type of data format for columns then has the effect of hiding columns that
are expected to be numeric (or vice versa) when conducting an MSA study.

The most common occurrences of this is when a space is added somewhere in the
column or when the letter O is used instead of 0 (zero). In both cases, even if the
typing error is rectified this will change the format of the column from numeric to text
format.

Text format columns can be identified by the addition of a ‘T’ to the column number as
in the example above.

40 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 7: Entering the Data in Minitab

2) The second issue is that of human mistakes when entering the data.
To guard against this, the facilitator of the measurement study must control the
study to maintain the concentration, time, speed and discipline required to
type/record each data point.
In addition to this, it is possible to assist the person entering the data to select the
correct cell by highlighting the line (descriptions and details) of the active entry.

Left click on the

row number will
highlight the
complete row

The example shown has used the option ‘randomise runs within operators’ with the
next entry being from appraiser 2 for part identity 9.
It can also be very beneficial to record comments when entries are typed. This
additional information can be useful for analysis where the Measurement System is
not acceptable and further investigation is needed.

Data Type Considerations

Types of data are very specific to each MSA study. For Gauge R&R studies the data
is variable and has to be the same units of measure as the operating process.
For Attribute Agreement Analysis this is attribute data BUT this can be in the format of
whole (count or scale) numbers or text values.

© 2013 Rolls-Royce plc | 41

MSA How-to Guide
Attribute Data

Appendix 8: Running the Analysis

To run an Attribute Measurement System Analysis (MSA) then use the menu commands:
Stat>
Quality Tools>
Attribute Agreement Analysis

42 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 8: Running the Analysis

A dialogue box will appear. Enter the data into the fields as shown below:

Click to confirm data is listed downwards (stacked)

This is the format Minitab generates for the worksheet

Enter the column containing the

‘standard’ to be compared to

Only check this box when a scale or multiple

class of judgement is used

© 2013 Rolls-Royce plc | 43

MSA How-to Guide
Attribute Data

Appendix 8: Running the Analysis

It is good practice to use the ‘Information’ button to

record details about the MSA study directly into the
displayed results – click on Information button

The information dialogue will appear, so that you can complete the details as
appropriate.

Once completed, click OK button to close the information screen, then click OK again
to close on main window.

44 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 8: The Graphical Output

The Graphical output will appear as below.

Click on Show sessions folder icon to review the

numerical output”

© 2013 Rolls-Royce plc | 45

MSA How-to Guide
Attribute Data

Appendix 8: The Numerical Output

Click on Show graphs folder icon to return to the

graphical output.

46 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 8: Running the Worksheet

Click on Show worksheet folder icon to return to the worksheet

© 2013 Rolls-Royce plc | 47

MSA How-to Guide
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis

Attribute Agreement Analysis Worksheet Configurations

1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is

very similar to a Gauge R&R study where, the worksheet construction all start
from an identical point independent of which randomisation for the worksheet is
used. This is found at:
Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet

48 | © 2013 Rolls-Royce plc

 Attribute Data
Appendix
6 9: Supplementary Information on interpreting the Output of Attribute
Appendix
Agreement Analysis

In addition to the ‘How to guide: Measurement System Analysis’ the following pages
give supplementary information on some of the statistical concepts to deepen your
understanding of how to fully interpret the output

Interpretation
of graphical
output

The ‘blue’ dots have been explained as the actual proportion of agreement within the
sample of parts appraised in the study. Also shown are the ‘red’ lines which end in a
cross. These indicate the confidence intervals for each Appraiser and for each
Appraiser vs. Standard.

The actual numbers for each confidence interval are recorded in the session window
and used in plotting this graph.

These confidence intervals take into account the fact that the actual agreement %
calculated (as represented by the blue dot) is based only on a relatively small sample
of data. If it was possible to know the ‘true’ agreement % of the appraiser (based on
every part they ever inspected) then this % would be likely to be different from the %
seen in the study sample. The confidence interval indicates the possible range of
values that the ‘true’ % agreement could be. Minitab defaults to a confidence level of
95%. This means that we can interpret the confidence interval for Appraiser 1 for
example as saying “I am 95% confident that the true % of within appraiser agreement
for appraiser 1 is between 62.1% and 96.8% (which is the range indicated by the blue
crosses and red line).
© 2013 Rolls-Royce plc | 49
MSA How-to Guide
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis

Each ‘red’ line and two crosses should be as short as possible indicating less potential
error in the ‘blue’ dot point indication, when using 95% confidence (the default level for
most statistical analysis).

These confidence intervals are calculated using the F distribution which also requires
a thing called degrees of freedom. The larger the degrees of freedom, which can be
influenced by using a larger selection of items for inspection (sample size) and the
quantity of matched agreements (such as Pass / Pass and the standard also as Pass)
also contribute to reducing the confidence intervals.

The F distribution is not symmetrical and therefore the confidence intervals can look a
little odd, one (usually the lower) will be longer than the other.

Interpreting the Lower Confidence Interval

It may be desirable in some instances where the quality of the measurement system is
highly critical to use the lower confidence interval (lower red cross) rather than the
observed agreement (blue dot) when assessing the measurement system against the
rules of thumb described on page 46 . This assesses the measurement system based
on the worst case scenario.

You must however take into consideration that the sample size will significantly
influence this. Where the sample size is small (less than 20 parts for example) the
lower confidence intervals will nearly always fall below 70% even when the average
agreement within the sample is fairly good). In these circumstances you should
discuss with a Black Belt an appropriate sample size to use for the assessment of
critical measurement systems. If the measurement needed is highly critical then
consideration should also be given to whether it is possible to redesign the
measurement system to use variable data rather than attribute and assess the system
using Gauge R&R instead.

50 | © 2013 Rolls-Royce plc

 Attribute Data

Appendix 9: Supplementary Information on interpreting the Output of Attribute

Agreement Analysis

Each of the percentage results shown in the session window tables have been
explained in the ‘How to guide: Measurement System Analysis’. This section will
explain the other statistics such as the ‘Kappa’ listed and the P-value. Each of the
tables is constructed with the same format and therefore the ‘Within Appraisers’ table
will be used for these explanations.

Interpretation
of the
numerical
output

The Kappa (correctly quoted as Fleiss Kappa) works on a scale of -1 to 1, where -1

indicates total disagreement between each measurement run and the standard. A
value of 0 (zero) indicates a 50:50 chance of correctly assessing the part which implies
appraisers are ‘guessing’ whether to pass or fail the part. The best possible outcome is
a Kappa value or 1 which indicates total agreement to each round of measures and to
the standard.

Within Appraisers table is shown for a simple Fail or Pass judgement which gives a
Kappa value similar BUT not the same as the percent listing.

Fleiss’ Kappa becomes very useful when a scale or multiple class of judgement is
used which then has a Kappa value indicating agreement for each scale value.
An example of this could be shade judgements, where the ends of the scale, shades A
or B (light), I or J (dark) have very high Kappa values near to 1 and middle shades
such as E or F have lower (approximately 0.778291 say) showing for that person these
were harder to judge as correct. © 2013 Rolls-Royce plc | 51
MSA How-to Guide
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis

P-value Judgements

Interpretation
of the
numerical
output

The P-value (which is short for the probability value) is a back-up statistic describing
the chance of having a Fleiss Kappa near 0 (zero).
As the tabular notation shows, the judgement is made about the ‘chance’ (probability)
where the lower the P (vs. > 0) column p-value shows a low number, the less
statistical chance the Kappa values are 0 (zero) or less.
In other words, the p-values of 0.0000 are very good and what should be seen.

52 | © 2013 Rolls-Royce plc

 Measurement System Analysis
How-to Guide - Appendices

Change History

Revision Date Description of Change Author Owner Approval

Guide reformatted for

V6.1 20/08/2013 D Prodger D Prodger D Prodger
SABRe

Document update policy

This document may be updated periodically. Major amendments will be shown as an update from one
revision number to a higher revision number (e.g. revision 1 to revision 2) and therefore the content of
the higher revision will be regarded as the latest requirements. A minor amendment will be shown as a
number change after a decimal point (e.g. revision 1.1 to revision 1.2) and therefore any of these
revisions may be regarded as the latest requirements until a major amendment is introduced

© Rolls-Royce plc 2013

The information in this document is the property of Rolls-Royce

plc and may not be copied, communicated to a third party or
used for any purpose other than that for which it is supplied,
without the express written consent of Rolls-Royce plc.

While the information is given in good faith based upon the

latest information available to Rolls-Royce plc, no warranty or
representation is given concerning such information, which must
not be taken as establishing any contractual or other
commitment binding upon Rolls-Royce plc or any of its
subsidiary or associated companies.

© 2013 Rolls-Royce plc | 53