Unit 5

Designing
Experiments

1
Definitions:
Observational Study - observe
outcomes without imposing any
treatment
Experiment - actively impose some
treatment in order to observe the
response
Survey - simply asking respondents for
data, no observation or treatments 2
I’ve developed a new rabbit food, Hippity
Hop, to improve the overall health of
rabbits. Makes fur
soft & shiny!

Increases
energy!
Rabbit Food

100% of daily
vitamins &
essential oils! 3
Can I just make these claims?
NO!
What must I do to make these
claims? Do an experiment

Who (what) should I test this on?

Rabbits
What do I test?
The type of food
4
• Experimental Unit – the single
individual (person, animal, plant,
etc.) to which the different
treatments are assigned
• Factor – is the explanatory
variable – it’s what we test
• Level – a specific value for the
factor 5
• Response Variable – what you
measure
• Extraneous Variable – a variable not
of primary interest, but could
potentially influence the outcome of
the response variable
• Treatment – a specific experimental
condition applied to the units
6
I plan to test my new rabbit food.
What are my experimental units?
Rabbits
What is my factor?
Type of Food
What is the response variable?
Health of the rabbit
(shinier coat, improved energy, etc) 7
 I’ll use my pet rabbit,
Lucky!

Rabbit Food

Since Lucky’s coat is shinier & he has

more energy, then Hippity Hop is a
better rabbit food! 8
• Control group – a group that is
used to compare the factor
against; can be a placebo or the
“old” or current item
• Placebo – a “dummy” treatment
that can have no physical effect

9
Rabbit Food

Now I’ll use Lucky & my

friend’s rabbit, Flash. Lucky
gets Hippity Hop food & Rabbit Food

Flash gets the old rabbit

food.

WOW! Lucky is bigger & shinier so

Hippity Hop is better! 10
 Rabbit Food The first five rabbits that I
catch will get Hippity Hop
food and the remaining
Rabbit Food five will get the old food.

The Hippity Hop rabbits have scored

higher so it’s the better food! 11
 Number the rabbits from 1 – 10.

Place the numbers in a hat and shake well

Rabbit Food

The first five numbers pulled

10 1
from the hat will be the
Rabbit Food
rabbits that get Hippity Hop
6 food.
2
The remaining rabbits 8
4
get the old food. 9
3 7
I evaluated
9 the rabbits & found that the 3
5
rabbits eating 8
Hippity Hop are better than
7 5
the old food! 12
• Blinding - method used so that
units do not know which
treatment they are getting
• Double Blind - neither the units
nor the evaluator know which
treatment a subject received
13
 Hippity Hop
Rabbit Food
makes fur soft
Rabbit Food and shiny, &
increases energy
for ALL types of
rabbits!
Can I make
this claim?

14
Definition:
• Confounding Variable – A variable whose
effect on the response cannot be
separated or distinguished from the
Becauseofobservational
effects the factor onstudies may contain
the response.
confounding variables, their results can NOT
be used to show cause-effect relationships.
• It is a variable that is related to both
group membership and the response
variable of interest in a research study
15
Principles of Experimental Design
• Control - the effects of extraneous
variables on the response
• Randomization – the use of chance to
assign subjects to treatments
• Replication - of the experiment on
many subjects to quantify the natural
variation in the experiment
• Comparison - the experiment must
have two or more treatments
(one of which may be a control or placebo) 16
The ONLY way to show
cause & effect is with a
well-designed,
well-controlled
experiment!!!
17
Example 1: A farm-product manufacturer wants to
determine if the yield of a crop is different when the
soil is treated with three different types of fertilizers.
Sixteen similar plots of land are planted with the same
type of seed but are fertilized differently. At the end of
the growing season, the mean yield from the sample
plots is compared.

Experimental units? Plots of land

Factors? Type of fertilizer
Levels? Fertilizer types A, B, C
Response variable? The amount the crop yield increased
How many treatments? 3 18
 Supposethat
Suppose thethe
researcher decides
sixteen plots to fertilize
of land are notthe 8
similar.
plotsare
Some closest
closetotothe riverwith
a river withtrees
fertilizer A the
along and bank,
the
remaining
while othersplots with B.away
are farther He has a greater
from yield
the river. Seewith
the
fertilizerbelow.
diagram A. Does this mean that A is the better
fertilizer?

A A B B
NO! The amount of
water or soil type is A A B B
confounded with the
type of fertilizer! A A B B
A A B B19
Example 2: A consumer group wants to test
cake pans to see which works the best (bakes
evenly). It will test aluminum, glass, and
plastic pans in both gas and electric ovens.
Experiment units? Cake batter
Factors? Two factors - type of pan & type of oven

Levels? Type of pan has 3 levels (aluminum, glass, & plastic

& type of oven has 2 levels (electric & gas)
Response variable? Does the cake bake evenly or not?
Number of treatments? 6
20
Example 3: A farm-product manufacturer wants to
determine if the yield of a crop is different when the
soil is treated with three different types of fertilizers.
Fifteen similar plots of land are planted with the same
type of seed but are fertilized differently. At the end of
the growing season, the mean yield from the sample
plots is compared.
Why is the same type of seed used on all 15 plots?
It is part of the controls in the experiment.
What are other potential extraneous variables?
Type of soil, amount of water, etc.
Does this experiment have a placebo? Explain
plots is compared.
21
NO – a placebo is not needed in this experiment
 Experimental Designs
Completely Randomized Design
–Experimental units are assigned at random
to treatments

Measure
Treatment
Random Assignment

Response
A for A

Experimental Compare
Units Treatments

Treatment Measure
Response
B
for B
22
 Suppose the researcher decides to randomly
Suppose that the sixteen plots of land are not similar.
assign fertilizer A to 8 plots and the remaining
Some are close to a river with trees along the bank,
plots with B. He has a greater yield with
while others are farther away from the river. See the
fertilizer A. Does this mean that A is the
diagram below.
better fertilizer?

Yes! The effects of B A B B

water and soil type
are evenly spread into B A A A
both treatments.
Thus there is no A B B A
confounding.

A B B A23
 Experimental Designs Continued . . .
Randomized block – units are blocked into
Units should be blocked on a variable that
groups (homogeneous) and then randomly
affects the response!!!
assigned to treatments
Measure
Random Assignment
Treatment A
response
for A

Block Compare
treatments
1 for block 1

Treatment B Measure
response
Experimental

Compare the results

from the 2 blocks
for B
Create blocks
Units

Measure
Random Assignment

Treatment A
response
for A

Block Compare
treatments
2 for block 2

Treatment B Measure
response 24
for B
 Then thethe
Suppose researcher
researcher randomly
decides assigns
to blockfertilizer
the 16 Aplots
to 4
Suppose
plots in that
the pinkthe sixteen
block and toplots
4 of inland
plots the are
blue not similar.
into two blocks based on their proximity to the river.
Some are remaining
block. The close to aplots
riverreceive
with fertilizer
trees along B. Hethehasbank,
while others
a greater yield are
withfarther
fertilizeraway
A. Doesfrom
thisthe
mean river.
that See
A the
is the better
diagram fertilizer?
below.

A B B A
Yes! Blocking is a way
to “control” the
A A B B
effects of extraneous
variables. B B A A
B A B A25
Matched pairs - a special type of
block design
– match up experimental units according
to similar characteristics & randomly
assign one to one treatment & the
other automatically gets the 2nd
treatment
– have each unit do both treatments in
random order
– the assignment of treatments is
dependent
26
 Treatments vs.
Treatment A
Measure
Treatment Groups

Random Assignment
response for A

Experimental Compare
Units treatments
Measure
Treatment B response for B
Completely Randomized

Measure Randomized Block

Random Assignment
Treatment A
response
for A

Block Compare
treatments
1 for block 1

Treatment B Measure

Compare the results

response
Experimental

from the 2 blocks

for B
Create blocks
Units

Measure
Random Assignment

Treatment A
response
for A

Block Compare
treatments
2 for block 2

Treatment B Measure
response 27
for B
Treatment A Treatment B

Pair
Next, experimental
randomly assignunits
according
one unit from atopair
specific
to
Treatmentcharacteristics.
A. The other
unit gets Treatment B.

This is one way to do a matched pairs

design – another way is to have the
individual unit do both treatments (acting
as their own block, as in a taste test).
28
 These headlines imply that spanking is the CAUSE of
Consider
thethe following
observed headlines
difference in IQ. which
occurred
Is this on September
conclusion 25, 2009.
reasonable?

“Spanking lowers a child’s IQ” (Los Angeles Times)

“Do you spank” Studies indicate it could lower your

kids’
In thisIQ.” (SciGuy,
study, Houston
two groups Chronicle)
of children were followed for 4
years; 806 children ages 2 to 4 and 704 children ages 5 to 9.
IQ was measured at the beginning of the study and again
“Spanking can lower IQ” (NBC4i, Columbus, Ohio)
four years later. Researchers found that the average IQ of
children, ages 2 to 4, who were not spanked was 5 points
“Smacking
higher thanhits
thosekids’
whoIQ”
were(newscientist.com)
spanked and 2.8 points higher
for children, ages 5 to 9.

29
Spanking and IQ Continued . . .
Confounding
In this variables
study, two groups mustwere
of children be associated
followed forto both 806
4 years;
children
to theages 2 to 4 and 704
explanatory children
(factor) ages
and 5 to 9. IQvariables.
response was measured
at the beginning of the study and again four years later.
Be sure found
Researchers to describe
that thethose relationships
average IQ of children,when
ages 2asked
to 4, who
to identify
were not spanked was 5 to confounding
points higher thanvariable!
those who were
spanked and 2.8 points higher for children, ages 5 to 9.
These are called
Does spanking “CAUSE”confounding
a decrease in IQ?
Why or why not? variables.

Are there other variables connected to the response

(decreased IQ) and the groups of children?
30
 Treatment B
Treatment A One group is assigned to
treatment A & the other
group to treatment B.

Treatment A Treatment B
Confounding does NOT
occur in a completely
randomized design!

Treatment & group are confounded

31
Example 4: An article from USA Today
reports the number of victims of violent
crimes per 1000 people. 51 victims have
never been married, 42 are divorced or
separated, 13 are married, and 8 are
widowed.
Is this an experiment? Why or why not?
No, no treatment was imposed on people.
What is a potential confounding variable?

Age – younger people are more at risk to be

32
victims of violent crimes
Example 5: Four new word-processing programs
are to be compared by measuring the speed with
which standard tasks can be completed. One
hundred volunteers are randomly assigned to
one of the four programs and their speeds are
measured.
Is this an experiment? Why or why not?
Yes, a treatment is imposed.
What type of design is this? Completely randomized
One factor: word-processing program
Factors? Levels?
with 4 levels
Response variable? Time/speed to complete tasks33
 Example 5: Four new word-processing
programs are to be compared by measuring
the speed with which standard tasks can be
completed. One hundred volunteers are
randomly designed to one of the four
You could do a matched
programs and their speeds are measured.
pairs design where each
Is thereperson
a potential confounding
uses each program variable?
in random order.
Can this design be
NO, completely randomized improved?
designs have no confounding Explain. 34
Let’s Look at another Example:
You collect data on sunburns and ice cream
consumption. You find that higher ice cream
consumption is associated with a higher
probability of sunburn.
Does that mean ice cream consumption causes
sunburn?
Here, the confounding variable is temperature:
Hot temperatures cause people to both eat
more ice cream and spend more time outdoors
under the sun, resulting in more sunburns.
35
Example 6: Suppose that the manufacturer wants to
test a new fertilizer against the current one on the
market. Ten 2-acre plots of land scattered throughout
the county are used. Each plot is subdivided into two
subplots, one of which is treated with the current
fertilizer, and the other with the new fertilizer. Wheat
is planted and the crop yields are measured.

What type of design is

this? Why use this Matched - pairs design
method?
When does randomization Randomly assigned
treatment to first acre
occur?
of each two-acre plot36
Randomization reduces bias by spreading
Is thereconfounding
any uncontrolled another wayvariables
to
reduce variability?
evenly throughout the treatment groups.

Blocking also helps reduce variability.

Bias is a systematic error in measuring
Variability is controlled by sample size.
the estimate
Larger samples produce statistics with less
variability.
37
 Bias vs. Variability

High bias & high variability High bias & low variability

Low bias & high variability Low bias & low variability
38