Engineering Experimental Design

Lecture 1
Course Contents
Design Of Experiment
DOE

Reference Book: Douglas C. Montgomery

Regents’ Professor of Industrial Engineering and
Statistics ASU Foundation Professor of Engineering
Arizona State University

1
 Chapter 1: Introduction and
Fundamental of Design of
Experiment

Chapter 1 2
 Why an Experiment?

Observing a system or process while it is running

provides insights into its current state, behavior, and
outcomes. This form of observation, often termed as
passive observation, allows us to see patterns,
correlations, and performance under typical conditions.

However, observation alone can’t always uncover the

cause-and-effect relationships between variables. For
that, active manipulation of input factors is required,
which is where experimentation comes in.
Observation vs. Experimentation: The Key Difference

Observation is the act of monitoring a system without

interference. It helps identify patterns, outliers, and
potential variables that might be influencing the
outcomes.

Experimentation, on the other hand, involves

deliberately changing one or more input factors
(independent variables) to see how the system responds.
This allows us to draw conclusions about causality—
which inputs cause which effects.

Let’s look at example to illustrate the difference

Manufacturing Process Observation vs. Experimentation

Observation: Imagine a factory producing cars. Engineers may

observe the assembly line to monitor the output, quality control
issues, and throughput rate. By watching, they could notice that
production slows down at a certain point, but they may not know
why. They could hypothesize that it’s due to a bottleneck in a
specific machine or process, but observation alone cannot confirm
this.

Experimentation: To identify the cause of the bottleneck,

engineers could design an experiment. For example, they might
change the speed of a particular machine, add more workers to a
stage of the process, or change the workflow. By analyzing how
each change affects production speed, they can determine which
factor(s) were responsible for the slowdown. This is an experiment
because they're actively changing variables to measure the effect.
 What is DOE?
• DOE refers to the process of:
1. Planning the experiment.
2. Preforming it.
3. Collecting its appropriate
data.
4. Finding the result.
5. Deriving conclusion.
Chapter 1 Design & Analysis of Experiments 8
7E 2009 Montgomery
 1. Introduction to DOE
• An experiment can be thought of as a test or series of
tests in inwhich
tests whichwewemake controlled
make changes
controlled changesto the input
to the
variables of a process
input variables or a system,
of a process in orderintoorder
or a system, determine
to
how they change the output of interest.
determine
how they change the output of interest.

Chapter 1 9
 Count.

A System can be defined

by:
• Output Response (Yield of a Process)
A
• Input  Factors System
• Controlled Factors: set to predefined levels for DOE (i.e. Temp., Fill
Pressure, Material Moisture)

• Uncontrollable Factors: factors that cannot be controlled in

actual operations, but may be controlled during experimentation
(i.e. Humidity, air pollution)

Quantitative Factors: Temperature; pressure, velocity…

Qualitative Factors: Car model, kind of process, gender…

Chapter 1 1
0
 Count.
• Experiments are used widely in the engineering world
• Process characterization & optimization
• Evaluation of material properties
• Product design & development
• Component & system tolerance determination

“All experiments are designed experiments, some are

poorly designed, some are well-designed”

Chapter 1 1
1
 EXAMPLE 1
As an example of an experiment, suppose that a metallurgical engineer is
interested in studying the effect of two different hardening processes, oil
quenching, and saltwater quenching, on an aluminum alloy. Here the
objective of the experimenter (the engineer) is to determine which
quenching solution produces the maximum hardness for this particular
alloy. The engineer decides to subject a number of alloy specimens or test
coupons to each quenching medium and measure the hardness of the
specimens after quenching. The average hardness of the specimens treated
in each quenching solution will be used to determine which solution
is best.

1. Draw the system

2. Show the inputs and outputs of the system
3. How many factors to study in this example?
4. How many levels for each factor?
5. How minimum experiments to do if we decide to make 3
repetitions of each treatment?
Chapter 1 1
2
 As we consider this simple experiment, a number of important
questions come to mind:

1. Are these two solutions the only quenching media of

potential
interest?
2. Are there any other factors that might affect hardness that
should be investigated or controlled in this experiment (such
as, the temperature of the quenching media)?
3. How many coupons of the alloy should be tested in each
quenching solution?
4. How should the test coupons be assigned to the quenching
solutions, and in what order should the data be collected?
5. What method of data analysis should be used?
6. What difference in average observed hardness between the
two quenching media will be considered important?

Chapter 1 13
 EXAMPLE 2
I like to play golf. I am always looking for a simpler solution to lowering my
score. Some of the factors that I think may be important are :
1. The type of driver used (oversized or regular sized)
2. The type of ball used (balata or three-piece)
3. Walking and carrying the golf clubs or riding in a golf cart
4. Drinking water or drinking “something else” while playing
5. Playing in the morning or playing in the afternoon
6. Playing when it is cool or playing when it is hot
7. The type of golf shoe spike worn (metal or soft)
8. Playing on a windy day or playing on a calm day.

Chapter 1 14
 Count.
Now, let’s consider how factors 1 through 4 could be experimentally
tested to determine their effect on my golf score:
Suppose that a maximum of eight rounds of golf can be played over
the course of the experiment. One approach would be to select an
arbitrary combination of these factors, test them, and see what
happens.
• For example, suppose the oversized driver, balata ball, golf cart,
and water combination is selected, and the resulting score is 87.
During the round, however, I noticed several wayward shots with
the big driver, and, as a result,
• I decide to play another round with the regular-sized driver,
holding the other factors at the same levels used previously.
This approach could be continued almost indefinitely, switching the
levels of one or two (or perhaps several) factors for the next test,
based on the outcome of the current test. This strategy of
experimentation, which we call the best-guess approach, is frequently
used in practice by engineers and scientists.
Chapter 1 15
 Count.
Another strategy of experimentation that is used extensively in practice
which is the one-factor-at-a-time (OFAT) approach.
The OFAT method consists of selecting a starting point, or baseline set
of levels, for each factor, and then successively varying each factor
over its range with the other factors held constant at the baseline level.

The major disadvantage of the OFAT strategy is that it fails to consider

any possible interaction between the factors

Chapter 1 16
 Factorial Design
The correct approach to dealing with several factors is to conduct a
factorial experiment. This is an experimental strategy in which
factors are varied together, instead of one at a time.

Chapter 1 17
 Count.

strokes per round

Chapter 1 18
 Count.

Chapter 1 19
 Count.

Chapter 1 20
 Factorial Designs with Several
Factors

Chapter 1 21
Chapter 1 22
 A Fractional Factorial
Designs

Chapter 1 23
PART 2 Guidelines for designing
experiment

Chapter 1 Part II 29
 Introduction
To use the statistical approach in designing and analyzing an
experiment, it is necessary for everyone involved in the
experiment to have a clear idea in advance of exactly what is
to be studied, how the data are to be collected, and at least a
qualitative understanding of how these data is to be analyzed.

Chapter 1 Part II 30
 1. Recognition of and statement of
the problem
It is necessary to develop all ideas about the objectives of the
experiment. Usually, it is important to solicit input from all
concerned parties: engineering, quality assurance, manufacturing,
marketing, management, customer, and operating personnel (who
usually have much insight and who are too often ignored). For this
reason, a team approach to designing experiments is
recommended.
It is usually helpful to prepare a list of specific problems or
questions that are to be addressed by the experiment. A clear
statement of the problem often contributes substantially to better
understanding of the phenomenon being studied and the final
solution of the problem.
Chapter 1 Part II 31
 Count.

Some of the reasons for running the experiments

include:
A. Factor screening or characterization.
When a system or process is new, it is usually important
to learn which factors have the most influence on the
response(s) of interest. Often there are a lot of factors.
This usually indicates that the experimenters do not know
much about the system, so screening is essential if we are
to efficiently get the desired performance from the system.

Chapter 1 Part II 32
 Count.

B. Optimization.
Find the settings or levels of the important factors
that result in desirable values of the response.
For example, if a screening experiment on a
chemical
process in the time and
results identification of the two factors,
temperature experiment
optimization as most
may have
important
as its objective finding
the
the levels of time and temperature that maximize yield,

Chapter 1 Part II 33
 Count.
C. Confirmation.
In a confirmation experiment, the experimenter is trying to verify
that the system behaves in a manner that is consistent with some
theory or past experience.
For example, if theory or experience indicates that a particular
new material is equivalent to the one currently in use and the new
material is desirable (perhaps less expensive, or easier to work
with in some way), then a confirmation experiment would be
conducted to verify that substituting the new material results in no
change in product characteristics that impact its use.

Chapter 1 Part II 34
 Count.
D. Discovery.
In discovery experiments, the experimenters are usually trying to
determine what happens when we explore new materials, or new
factors.
In the pharmaceutical industry, scientists are constantly
conducting discovery experiments to find new materials or
combinations of materials that will be effective in treating disease.
These experiments often address questions such as under what
conditions do the response variables of interest seriously degrade?
Or what conditions would lead to unacceptable variability in the
response variables?

Chapter 1 Part II 35
 2. Selection of the response variable
In selecting the response variable, the experimenter should be
certain that this variable really provides useful information about
the process under study. Multiple responses are not unusual.
The experimenters must decide how each response will be
measured, and address issues such as how will any measurement
system be calibrated and how this calibration will be maintained
during the experiment. The gauge or measurement system
capability (or measurement error) is also an important factor. If
gauge capability is inadequate, only relatively large factor effects
will be detected by the experiment or perhaps additional
replication will be required

Chapter 1 Part II 36
 Count.
It is usually critically important to identify issues related to
defining the responses of interest and how they are to be measured
before conducting the experiment. Sometimes designed
experiments are employed to study and improve the performance
of measurement systems.

Chapter 1 Part II 37
 3. Choice of factors, levels, and
rang the factors that may influence the performance of a process
When considering
or system, the experimenter usually discovers that these factors can be
classified as either potential design factors or nuisance factors.
The potential design factors are those factors that the experimenter may wish to
vary in the experiment. Often, we find that there are a lot of potential design
factors, and some further classification of them is helpful.
Some useful classifications are design factors, held-constant factors, and
allowed-to-vary factors.
The design factors are the factors actually selected for study in the experiment.
Held-constant factors are variables that may exert some effect on the response,
but for purposes of the present experiment these factors are not of interest, so
they will be held at a specific level.
As an example of allowed-to-vary factors, the experimental units or the
“materials” to which the design factors are applied are usually non-
homogeneous, yet we often ignore this unit-to-unit variability and rely on
randomization to balance out any material or experimental unit effect.
We often assume that the effects of held-constant factors and allowed-to-vary
factors are relatively small.
Chapter 1 Part II 38
 Count.
Nuisance factors, may have large effects that must be accounted for. They are
often classified as controllable, uncontrollable, or noise factors.

A controllable nuisance factor is one whose levels may be set by the

experimenter. For example, in a study testing the effectiveness of different
fertilizers on plant growth. Controllable nuisance factor could be watering
schedule. While the focus of the experiment is on the fertilizer type, water is
crucial for plant growth. The amount and frequency of watering could
significantly affect the outcome. To control this, the researcher can ensure that
all plants receive the same amount of water at the same intervals. This way, the
effect of the water is minimized, allowing the experiment to focus on the
fertilizers.

The blocking principle, is often useful in dealing with controllable nuisance

factors.
 Count.
If a nuisance factor is uncontrollable in the experiment, but it can be measured,
an analysis procedure called the analysis of covariance can often be used to
compensate for its effect.

For example, in a factory setting studying the impact of machine settings on

product quality. Uncontrollable nuisance factor could be operator experience.
The skill level of the machine operators can vary, which could impact product
quality, even though the focus of the study is on machine settings. Since
operator skill can’t be controlled perfectly, researchers can record the operator’s
experience and include it as a variable (covariate) in the analysis to account for
its impact.
 Count.
When a factor that varies naturally and uncontrollably in the process, but it can
be controlled for purposes of an experiment, we often call it a noise factor. For
example in a process that involves injection molding to produce plastic parts, the
goal may be to study the effect of machine settings (temperature, pressure) on
product quality. Noise factor could be ambient temperature or humidity in the
factory. The external environment can influence the behavior of materials during
molding. High humidity could affect the cooling of plastic parts, leading to
warping or defects.

In such situations, our objective is usually to find the settings of the

controllable design factors that minimize the variability transmitted from the
noise factors.
Once the experimenter has selected the design factors, he must decide on a
region of interest for each variable (that is, the range over which each
factor will be varied) and on how many levels of each variable to use. Process
knowledge is required to do this.
This process knowledge is usually a combination of practical experience and
theoretical understanding. It is important to investigate all factors that may be
of importance and to be not overly influenced by past experience.
When the objective of the experiment is factor screening or process
characterization, it is usually best to keep the number of factor levels low.
Generally, two levels work very well in factor screening studies. Choosing the
region of interest is also important.
In factor screening, the region of interest should be relatively large— that is,
the range over which the factors are varied should be broad. As we learn more
about which variables are important and which levels produce the best results,
the region of interest in subsequent experiments will usually become narrower.
The cause-and-effect diagram (fishbone diagram) can be a useful technique for
organizing some of the information generated in pre-experimental planning.

Chapter 1 Part II 40
 Example
Suppose that you want to compare the growth of
garden flowers with different conditions of sunlight,
water, fertilizer, and soil conditions. Complete steps
1–3 of the guidelines for designing experiments.

Chapter 1 Design & Analysis of Experiments 41

7E 2009 Montgomery
 4. Choice of experimental
If the abovedesign
pre-experimental planning activities are done correctly,
this step is relatively easy.
Choice of design involves consideration of:
• Sample size (number of replicates),
• Selection of a suitable run order for the
experimental trials,
• Determination of whether blocking or other randomization
restrictions are involved.

Chapter 1 Part II 42
 Design selection also involves thinking about and selecting a
tentative empirical model to describe the results. The model is just
a quantitative relationship (equation) between the response
and the important design factors. In many cases, a low-order
polynomial model will be appropriate.
A first-order model in two variables is

• 𝑦 is the response,
Where,

• 𝑥’s are the design factors,

• 𝛽’s are unknown parameters that will be estimated from

• 𝜀 is a random error term that accounts for the

the data in the experiment,

experimental error in the system that is being studied.

Chapter 1 Part II 43
The first-order model is also sometimes called a main effects
model. First-order models are used extensively in screening or
characterization experiments.
A common extension of the first-order model is to add an
interaction term, say

where the cross-product 𝑥1𝑥2 represents the two-factor

interaction
term between the design factors. Because interactions
between factors is relatively common, the first order model with
interaction is widely used

Chapter 1 Part II 44
 5. Performing the experiment
When running the experiment, it is vital to monitor the process
carefully to ensure that everything is being done according to
plan.
Errors in experimental procedure at this stage will usually destroy
experimental validity. One of the most common mistakes
encountered is that the people conducting the experiment failed to
set the variables to the proper levels on some runs.
Therefore, someone should be assigned to check factor settings
before each run. Up-front planning to prevent mistakes like this is
crucial to success. It is easy to underestimate the logistical and
planning aspects of running a designed experiment in a complex
manufacturing or research and development environment.

Chapter 1 Part II 45
 Count.
Coleman and Montgomery (1993), suggest that prior to
conducting the experiment a few trial runs or pilot runs are often
helpful. These runs provide information about consistency of
experimental material, a check on the measurement system, a
rough idea of experimental error, and a chance to practice the
overall experimental technique.

Chapter 1 Part II 46
 6. Statistical analysis of the
Statistical data
methods should be used to analyze the data so that
results and conclusions are objective rather than judgmental in
nature. If the experiment has been designed correctly and
performed according to the design, the statistical methods
required are not elaborate.
Often, we find that simple graphical methods play an important
role in data analysis and interpretation. Because many of the
questions that the experimenter wants to answer can be cast into a
hypothesis-testing framework, hypothesis testing and confidence
interval estimation procedures are very useful in analyzing data
from a designed experiment.

Chapter 1 Part II 47
It is also usually very helpful to present the results of many
experiments in terms of an empirical model, that is, an equation
derived from the data that express the relationship between the
response and the important design factors. Residual analysis and
model adequacy checking are also important analysis techniques.
We will discuss these issues in detail later.

Relationship

(Response)

Chapter 1 Part II 48
Statistical methods cannot prove that a factor (or factors) has a
particular effect. They only provide guidelines as to the reliability
and validity of results. When properly applied, statistical methods
do not allow anything to be proved experimentally, but they do
allow us to measure the likely error in a conclusion or to attach a
level of confidence to a statement. The primary advantage of
statistical methods is that they add objectivity to the decision-
making process.
Statistical techniques coupled with good engineering or
process knowledge and common sense will usually lead to
sound conclusions.

Chapter 1 Part II 49
 7. Conclusions and recommendations
Once the data have been analyzed, the experimenter must draw
practical conclusions about the results and recommend a course of
action.
Graphical methods are often useful in this stage, particularly in
presenting the results. Follow-up runs and confirmation testing should
also be performed to validate the conclusions from the experiment.
Throughout this entire process, it is important to think that
experimentation is an important part of the learning process, where
we tentatively formulate hypotheses, perform experiments to
investigate these hypotheses, and based on the results formulate new
hypotheses, and so on. This suggests that experimentation is iterative.

50
 Count.
It is usually a mistake to design a single, large, comprehensive
experiment at the start of a study. A successful experiment
requires knowledge of the important factors, the ranges over
which these factors should be varied, the appropriate number of
levels to use, and the proper units of measurement for these
variables.
Generally, we do not perfectly know the answers to these questions,
but we learn about them as we go along. As an experimental program
progresses, we often drop some input variables, add others, change
the region of exploration for some factors, or add new response
variables.
Consequently, we usually experiment sequentially, and as a rule, no
more than about 25 percent of the available resources should be
invested in the first experiment. This will ensure that sufficient
resources are available to perform confirmation runs and ultimately
accomplish the final objective of the experiment. 51
Finally, it is important to recognize that all experiments are
designed experiments. The important issue is whether they are
well designed or not. Good pre-experimental planning will usually
lead to a good, successful experiment. Failure to do such planning
usually leads to wasted time, money, and other resources and
often poor or disappointing results.

Chapter 1 Part II 52
 8. Summary in Using Statistical
Techniques in
Much of the Experimentation
research in engineering, science, and industry is empirical and
makes extensive use of experimentation. Statistical methods can greatly increase
the efficiency of these experiments and often strengthen the conclusions so
obtained. The proper use of statistical techniques in experimentation requires that
the experimenter keep the following points in mind:

1. Use your non-statistical knowledge of the problem. Experimenters are

usually highly knowledgeable in their fields. For example, a civil engineer
working on a problem in hydrology typically has considerable practical
experience and formal academic training in this area. In some fields, there is
a large body of physical theory on which to draw in explaining relationships
between factors and responses. This type of non-statistical knowledge is
invaluable in choosing factors, determining factor levels, deciding how
many replicates to run, interpreting the results of the analysis, and so forth.
Using a designed experiment is no substitute for thinking about the problem.

Chapter 1 Part II 53
2.Keep the design and analysis as simple as possible. Don’t be overzealous in
the use of complex, sophisticated statistical techniques. Relatively simple
design and analysis methods are almost always best. If you do the pre-
experiment planning carefully and select a reasonable design, the analysis will
almost always be relatively straightforward. In fact, a well-designed experiment
will sometimes almost analyze itself

3.Experiments are usually iterative. In most situations it is unwise to design too

comprehensive an experiment at the start of a study. Successful design requires
the knowledge of important factors, the ranges over which these factors are
varied, the appropriate number of levels for each factor, and the proper methods
and units of measurement for each factor and response. Generally, we are not
well equipped to answer these questions at the beginning of the experiment, but
we learn the answers as we go along. Consequently, we usually should not
invest more than about 25 percent of the resources of experimentation (runs,
budget, time, etc.) in the initial experiment. Often these first efforts are just
learning experiences, and some resources must be available to accomplish the
final objectives of the experiment.

Chapter 1 Part II 54