Measurement

Total Quality Methodologies in Engineering

INSE 6210

Zachary Patterson, Professor

Concordia Institute for Information Systems Engineering (CIISE)
27 September 2023
 Last Class

• Principles of Six Sigma

• Wrapped-up Six Sigma Metrics
• Six Sigma Problem Solving Methodology
• Lean Six Sigma
• Define
• Project Organization
• Project Selection

2
 Today’s Class
• Wrap up Define
• Project Definition
• Process Measurement
• Identifying and selecting metrics
• Data collection
• Statistical sampling
• Descriptive statistics
• Measurement system evaluation

3
 Overview and Principles of Six Sigma

Six Sigma Methodology

Define Measure Analyze Improve Control

How will you

What problem are Why is the What do you
What is the extent ensure that the
you trying to problem propose to do and
of the problem? problem stays
solve? occurring? why?
fixed?

1. Understand
4. Identify what to 7. Identify performance 9. Generate possible 11. Institutionalize &
customer & business
measure gaps solutions monitor solution(s)
requirements
2. Complete high-level, 10. Prioritize & select 12. Replicate & share
5. Plan and collect data 8. Validate root causes
as-is process map solution(s) best practices
3. Complete project 6. Determine baseline 13. Celebrate &
charter performance recognize success

Additional Topics

Drawn from the “The DMAIC Roadmap” Advanced Innovation Group Pro Excellence 2018
 Overview and Principles of Six Sigma

Six Sigma Methodology

Define Measure Analyze Improve Control

Control How will you ensure the problem stays fixed?

• Project Selection Matrix

• Voice customer (VOC)
Step 1
• Affinity diagram
Understand customer • What does the problem look like from your customer’s
• Kano Analysis
and business perspective?
• Critical to Quality Tree
requirements
• Performance-Importance
Analysis
Step 2
Complete high-level as- • Where is the problem occurring? • SIPOC Diagram
is process map
• What specific measures indicate there is a problem?
Step 3 • What will success look like?
Complete the project • What is the scope?
• Project charter
charter • Who is on the team and what are their roles and
responsibilities?
• What is your project plan?
Tollgate success: The team is equipped and mobilized and fully understands the business areas and problem they are
trying to solve.
Drawn from the “The DMAIC Roadmap” Advanced Innovation Group Pro Excellence 2018
 Define
Project Identification & Selection
Cost of Quality Analysis
• COQ:
• Cost of avoiding (poor quality), or cost
incurred as a result of poor quality
• Translates defects, errors, etc. into the
“language of management” – $$$
• 4 main types of cost to consider

7
Quality Cost Classification
• Prevention Costs
• Costs to keep non-conforming products from
occurring and reaching customers
• Quality planning costs
• Process control costs
• Information systems costs
• Training and management costs

8
Quality Cost Classification
• Appraisal Costs
• Costs associated with ensuring conformance
• Test and inspection costs
• Instrument maintenance costs
• Process measurement and control costs
• Internal Failure Costs
• Costs when non-conformance found before
delivery
• Scrap & rework costs, downgrading costs, etc.

9
Quality Cost Classification
• External Failure Costs
• Costs when poor-quality reaches customer
• Returns
• Product recall
• Product liability

10
 COQ Analysis Tools

• Cost indices and matrices

• Like scoring models
• Sampling and work measurement
• Pareto analysis

11
Cost of Quality Matrix

12
 Work Measurement
Sampling
• How much time a worker spends on quality
• Individual’s salary x worker time

13
 Pareto Analysis
• Pareto analysis:
• Technique to ID the “vital few” from the
“trivial many”
• Pareto distribution:
• 80-20 rule
• Pareto diagram:
• Histogram of the data from the highest to
lowest frequency

14
Pareto Diagram

15
Pareto Diagram

16
 Progressive Pareto
Analysis

17
 Define
Project Definition
 Project Charter
• Project definition culminates with “Project
Charter”:
• Formal mission statement that defines
project, objectives, and deliverables
• “Contract” between project team and
sponsor
• Sets expectations, goals and resources
• Defines problem to be addressed, customer
requirements, benefits, measures,
justification and timeline

19
Project Charter

Figure 3.6 Evans & Lindsay

20
Define Proj. Boundaries
• SIPOC Diagram
• Suppliers, Inputs, Process, Outputs,
and Customers

Suppliers Inputs Process Outputs Customers

Figure 3.7 Evans & Lindsay

• Provides broad overview of process

21
 SIPOC - Diagram
Metal Stamped Body
Fabricators Parts Fabrication

Painting Finished
Dealers
Automobiles

Component Parts &

Assembly
Suppliers Sub-assemblies

S I P O C
Figure 3.7 Evans & Lindsay
 Identifying Customers
• Customer viewed broadly
• Think in terms of customer supplier relations
• What goods or services are produced by my
work?
• Who uses these products and services?
• Who do I call, write to, or answer questions
for?
• Who supplies the inputs to my process?

23
 Customer Requirements
Voice of Customer
• Key approaches:
• Comment cards and formal surveys
• Focus groups
• Direct customer contact
• Field intelligence
• Complaint analysis
• Internet and social media
monitoring

24
Customer Requirements
(Kano Model)
• Dissatisfiers (“must haves”):
• Expected requirements
• Satisfiers (“wants”):
• Expressed requirements
• Considered minimum to stay in business
• Exciters/delighters (“never thought of”):
• Unexpected features (not yet requirements)
• Special attention to develop these

25
Performance-Importance
Analysis
• Also useful for defining projects in context of
different requirements
Performance
Low High

Low Who cares? Overkill

Importance

High Vulnerable Strengths

26
Critical-to-quality (CTQ)
Characteristics
• CTQs:
• Critical to a customer’s perception of quality
• Can be measured
• Specification can be defined so easy to
determine if achieved
• It is useful to identify them to help define a
project

27
 Define
7 Planning & Management Tools
Affinity Diagram

• Organize large numbers

of ideas
• Groups ideas into
smaller categories

29
Interrelationship Digraph

• Identifies & explores

influential relationships
between concepts

30
Tree Diagram

• Breaks problems down

• Maps out tasks
necessary to complete
a project

31
Matrix Diagram

• “Spreadsheets”
• Graphically display
relationship between
ideas, activities, etc.

32
 Matrix Data Analysis

• Arrange data to display quantitative relationships

easily
• Essentially scoring models

33
Process Decision Program
Chart

• PDPC
• Contingency mapping
34
 Arrow Diagram

• Used to sequence and schedule project tasks

35
Define
Project Review
 Define - Project Review
• Team has reached agreement on and clearly
defined the problem or opportunity to address
• Team understands strategic and financial impact
of project
• Team agrees that project can be completed
successfully
• Project plan and timeline have been developed
to guide the entire Six Sigma project
• The right mix of people are on the team

37
 Define - Project Review
• Key stakeholders outside of team identified
• Team members have received necessary “just-in-
time” training
• Voice of the customer and CTQs fully understood
and documented
• Team has developed high-level process map
• Key performance measures identified
• Project charter developed and agreed upon

38
 Overview and Principles of Six Sigma

Six Sigma Methodology

Define Measure Analyze Improve Control

How will you

What problem are Why is the What do you
What is the extent ensure that the
you trying to problem propose to do and
of the problem? problem stays
solve? occurring? why?
fixed?
1. Understand
4. Identify what to 7. Identify performance 9. Generate, prioritize 10. Institutionalize &
customer & business
measure gaps & select solution(s) monitor solution(s)
requirements
2. Complete high-level, 11. Replicate & share
5. Plan and collect data 8. Validate root causes
as-is process map best practices
3. Complete project 6. Determine baseline 12. Celebrate &
charter performance recognize success

Additional Topics
 Overview and Principles of Six Sigma

Six Sigma Methodology

Define Measure Analyze Improve Control

Control How will you ensure the problem stays fixed?

Step 4
• What does the detailed process currently look like?
Identify what to • Functional process map
• What is the output (Y) and primary measures (X's)?
measure
• Data Collection Plan
• Operational Definition
Step 5 • How can we ensure the data collection is robust?
• Sampling
Plan and collected data • What does the data say?
• Measurement System
Analysis
Step 6
Determine baseline • Is the process stable? • Run Chart
performance • Is the process capable of meeting requirements? • Process Capability

Tollgate success: The team understands the current process well and has collected robust data to support detailed analysis.

Drawn from the “The DMAIC Roadmap” Advanced Innovation Group Pro Excellence 2018
 Measure

• Understand current performance of process

selected
• Collect necessary data for Analysis
• Determine right measures (metrics) early in
project planning

41
 Measurement
Identifying & Selecting Process Metrics
 Effective Metrics
• SMART
• Simple
• Measurable
• Actionable
• Provide a basis for decision-making
• Related…
• …to customer requirements and to each
other
• Timely

43
Process Metric ID & Select’n

• Identify all customers and their requirements

(CTQs) and expectations
• Define work processes
• Define value-adding activities and process
outputs

44
Process Metric ID & Select’n

• Develop measures for each key process

• Checkpoints for each customer
• Evaluate measures for their usefulness
• Where does value-addition occur?
• Are measures controllable?
• Are data obtainable?
• Are operational definitions established?

45
High-level Process Mapping

• A process map (flowchart):

• ID’s sequence of activities or flow of materials
& information in a process
• Helps people involved understand process
better
• Helps identify appropriate metrics for process
control

46
 Developing Process Maps
• Best to include all those involved in the process
1. Begin with process output and ask:
• What is the last essential subprocess that
produces the output of the process?
2. For that subprocess, ask:
• What input does it need to produce the
process output?

47
 Developing Process Maps
3. For each input, identify its source
• Often, input will be output of previous
subprocess
• Sometimes, input comes from external
suppliers
4. Continue backward, one subprocess at a time,
until each input comes from an external
supplier

48
 E.g. Pizza Fulfillment

• Example 4.2
• CTQs:
• Easy ordering
• Quick delivery
• Reasonable price

49
 E.g. Pizza Fulfillment

• Note: Taking Order

• Order taker: internal

customer of caller
• Cook: customer of order Assembly
taker
• End customer: customer of
deliverer Delivery
50
 E.g. Pizza Fulfillment

• Note: Taking Order

• Order taker: internal

customer of caller
• Cook: customer of order Assembly
taker
• End customer: customer of
deliverer Delivery
51
 E.g. Pizza Fulfillment

• Note: Taking Order

• Order taker: internal

customer of caller
• Cook: customer of order Assembly
taker
• End customer: customer of
deliverer Delivery
52
 E.g. Pizza Fulfillment

• Note: Taking Order

• Order taker: internal

customer of caller
• Cook: customer of order Assembly
taker
• End customer: customer of
deliverer Delivery
53
 Pizza Fulfillment Metrics
• Possible metrics
• Pizzas per hour
• Order accuracy
• % orders rejected
• Delivery within 20 minutes
• # of errors as % of total
• Raw material inventory levels

54
 Value Stream Maps
• Map of all activities involved in designing,
producing, and delivering goods and services to
customers
• Difference from regular process map:
• Highlight value-added versus non-value-
added activities
• Includes the amount of time activities take
• Important in “Lean Thinking”

55
 Metrics -> CTQ Trees

• Metrics should be clearly aligned with CTQs

• Recall: Y = f(X)
• Y is set of CTQs
• X represents set of critical input variables
affecting Y
• CTQ trees “drill down” from Y to ID critical X-
factors

56
CTQ Trees

Good customer

57
 CTQ Trees

• CTQ trees help:

• Understand relationships btw. Y & Xs
• Define experiments
• Define factors to monitor and control

58
Measurement
Data Collection
 Data Collection
• Key Questions
• What questions are we trying to answer?
• What type of data will we need to answer
the question?
• Where can we find the data?
• Who can provide the data?
• How can we collect the data with minimum
effort and minimum chance of error?

60
Data Collection Techniques

• Data can be collected in a variety of ways…

61
Data Collection Sheets
• Use simple columnar or tabular forms to
record data
• Table
• Spreadsheet
• Naturally, nowadays, these can be replaced
with electronic data collection
• iPads, etc.

62
Data Collection Sheets

63
 Check Sheets

• Integrate data analysis with collection

• Results can be interpreted on the form directly
without additional processing

64
 Data Collection
Check Sheet

65
Check Sheet

66
 Check Sheet

Kaoru Ishikawa relates how this check sheet was used to eliminate bubbles in laminated
automobile windshield glass.

67
Measurement
Statistical Sampling
 Statistical Sampling
• What is it?
• A way to learn about a population using
information from only a sample
• Why use it?
• To save money when trying to learn about a
population
• You don’t have to do a census

69
 Statistical Sampling

• Questions to ask before starting a study

• What is the objective of the study?
• What type of sample should be used?
• What possible error might result from
sampling?
• What will the study cost?

70
 Sampling Methods
• Simple random sampling (SRS)
• Every item same probability of being in
sample
• Stratified sampling
• Population first divided between strata and
SRS used after
• Systematic sampling
• Every nth item selected

71
 Sampling Methods
• Cluster sampling
• Typical group (e.g. department) selected
• SRS within the group
• Judgment sampling
• Expert opinion used to determine location
and characteristics of definable groups
• More subjective, more risky

72
Selecting a Sampling Plan
• You should select a sample:
• With lowest cost that will provide sample:
• Best possible representation of the
population
• consistent with the objectives of precision
and reliability determined for the study
• Trade-off: cost vs. precision & reliability

73
 Error from Sampling
• Any procedure can have two types of error:
• Sampling Error:
• Occurs naturally
• Reduced with larger sample
• Systematic Error:
• Result of poor design
• E.g. error caused by not setting an
instrument to zero prior to its use
• Can be reduced with planning

74
 Choosing Sample Size
• Sample size required for a sampling error
of ± E or less for a confidence level of
100(1 – a):

• E – Maximum sampling error

• σ – standard deviation (typically
estimated)
• zɑ/2 – Critical value for normal distribution
75
 Choosing Sample Size
• Example 4.3
• Managers believe time to process
package ~3 minutes
• Want more accurate estimate
• From pilot sample, estimated standard
deviation 0.16
• Sample size for maximum allowable
sampling error of ± 0.025 minutes with
95 percent confidence

76
 Choosing Sample Size

• Example 4.3

• Need sample size of around 157

77
 Choosing Sample Size

• For a proportion (use sample estimate for

p or 0.5):

78
 Measurement
Descriptive Statistics & Data Summarization
 Descriptive Statistics

• Methods of presenting data visually and

numerically
• Measures of central tendency
• Measures of dispersion

80
Measures of Central
Tendency

81
 Measures of Central
Tendency
• Sample Mean
• x – observation
• n – number of observations
• AVERAGE(data range) in Excel

82
 Measures of Central
Tendency
• Median:
• Middle value (or 50th percentile) when
data are arranged from smallest to largest
• MEDIAN(data range) in Excel
• Mode:
• Observation that occurs most frequently
• MODE.MULT(data range) in Excel

83
Measures of Dispersion

84
 Measures of Dispersion

• Range:
• Difference between maximum value and
minimum value in a data set
• MAX(data range)-MIN(data range) in Excel

85
 Measures of Dispersion
• Sample variance
• VAR.S(data range) in Excel

• Sample standard deviation

• VAR.P(data range)

86
 Proportions
• Proportion:
• Fraction of data that have a certain
characteristic
• Key descriptive statistic for categorical data,
such as defects or errors
• COUNTIF(range, criteria) useful to calculate
this in Excel

87
Measures of Shape

88
 Measures of Shape
• Coefficient of skewness (CS):
• Measure of degree of asymmetry of
observations around mean

89
 Measures of Shape
• Coefficient of kurtosis (CK):
• Measures peakedness (i.e., high, narrow) or
flatness (i.e., short, flat-topped) of a
distribution

90
 Measure
Descriptive Statistics & Excel
Descriptive Stats in Excel

• Analysis ToolPak
• Descriptive Statistics
• Histogram

92
Descriptive Stats in Excel

93
Histogram Tool in Excel

94
 Measure
Measurement System Evaluation
Measurement System Eval.

• Accurately assessing Six Sigma performance

depends on reliable measurement systems
• Measuring quality characteristics generally
requires use of instrument or gauge
• Two categories:
• low technology
• high technology

96
Observed Process Variation
1. Natural variation in process itself
2. Variation from measurement system
• Tools, employees
• Instruments:
• Low-tech
• High-tech
• Want measurement variation to be as low as
possible to capture process variation
• Objective of quality assurance: minimize
measurement error

97
 Metrology
• Science of measurement
• Key characteristics of measures:
• Accuracy (related to average):
• Difference between the true value and
observed average of a measurement
• Precision (related to variance):
• Closeness of repeated measurements to
each other

98
Accuracy vs. Precision

99
 Calibration

• Process of verifying the capability and

performance of an instrument
• Process of testing equipment compared to
traceable measurement standards

100
 R & R Studies
Sources of variation – equipment or workers
• Repeatability (equipment variation - EV):
• Variation in multiple measurements by an
individual using the same instrument
• How consistent a measuring instrument is
• Reproducibility (appraiser variation - AV):
• Variation in the same measuring instrument
used by different individuals
• How consistent workers are in using measuring
instruments
101
 R&R Studies
• Study the variation in a measurement system
using statistical analysis
• Select m operators and n parts
• Calibrate the measuring instrument
• Randomly measure each part by each operator
for r trials
• Compute key statistics to quantify repeatability
and reproducibility

102
 R&R Studies

• Guidelines for evaluating EV, AV in R&R studies

• Under 10%: acceptable
• 10-30%: may be acceptable depending on
importance of the application, etc.
• Greater than 30%: not acceptable and needs to
be improved

103
 Spreadsheet Example

• The Student Companion Site provides an easy-to-

use Excel template, R&R.xlsx, for all the
calculations in an R&R study.
• See Figure 4.14

104
 Readings

• Chapter 3 Evans & Lindsay

• Chapter 4 Evans & Lindsay

105