CHAPTER 12

Experiments and
Observational Studies

October 25, 2024

October 28, 2024
 Observational studies:
• Conducted by observing choices the
participants made and recording the
results.
• Choices are not assigned by the
researcher.
• Can be either retrospective (subjects are
selected, then historical data are
compiled) or prospective (subjects are
identified, then data are collected as
events occur)
 Observational Studies:
The Good
• Useful for discovering trends (used in the
areas of marketing and public health),
especially retrospective studies.
• May be easier to generalize results more
broadly due to a higher likelihood of getting
a representative group
• Easier to conduct when large sample
sizes/groups are needed.
 Issues with observational studies
• The likelihood of a lurking variable (or
variables) being present is high.
• EX) A comparison of music students to non-
music students at a CA high school showed
that the music students had a much higher
GPA than non-music students, 3.59 to 2.91.
As a result, parents and educators pushed
for more music programs.
Issues with observational studies
• Problem: This study is being used to try
and establish a cause-and-effect
relationship, when in fact there are likely
other factors involved in a student’s
academic achievement…enrollment in a
music class being merely one of them.
Demonstrating causation: A matter
of degree and efficiency
• Cause and effect relationships are much more
difficult to establish with observational studies,
and are not as efficiently found, which is why we
use other methods to accomplish this more
directly…
• There is a link between smoking and lung cancer,
for example, but the only way this could be
established is through long-term observation
and careful selection of participants to study to
establish this link, among other aspects.
 How can we more reliably establish
a cause-and-effect relationship?
• EXPERIMENT: A study intended to establish cause-
and-effect via random assignment of subjects to
treatments.
• Example: In order to test the effectiveness of a
company’s new fertilizer compared to its current
fertilizer, 50 tomato plants are obtained. Half of
them are randomly assigned to the new fertilizer,
half to the current fertilizer. Results are analyzed
after a given time frame.
 Experiment
• An experiment studies the relationship
between two or more variables.

• At least one explanatory (independent)

variable must be identified…this is
called a factor. These factors are
manipulated by the researcher.

• The quantity being measured is called

the response variable.
 Basic breakdown of an
experiment:
• Researcher intentionally manipulates the
factors of any treatments
• Subjects are assigned treatments at
random
• Response variable is observed and results
for different groups are compared.
 Levels
• Specific values that are chosen for a given factor
• EX) If I wanted to see how a half-dose of
fertilizer worked compared to a full dose, that
would be an additional level of the chosen factor
(fertilizer)

Treatment
• The combination of specific levels from all the
factors that a subject (or experimental unit)
receives.
 EXAMPLE:
• A biologist is interested in studying the effect of
growth-enhancing nutrients and different
salinity (salt) levels in water on the growth of
shrimps. The experiment is to be conducted in a
laboratory where 10 tiger shrimps are placed
randomly into each of 12 similar tanks in a
controlled environment. The biologist is
planning to use 3 different growth-enhancing
nutrients (A, B, and C) and two different salinity
levels (low and high).
 EXAMPLE CONTINUED:
• List the treatments that the biologist plans to use
in this experiment.
• How many are there?
• SIX
• Nutrient A, low salinity Nutrient A, high salinity
• Nutrient B, low salinity Nutrient B, high salinity
• Nutrient C, low salinity Nutrient C, high salinity
• NOTE: Just saying “Nutrients A, B and C with low
and high salinity” is insufficient.
Key Elements of Experimental Design

• Control
• Randomization
• Replication
• Blocking (or not…more on that later)
 Control
• Outside of the factor being tested, all other
sources of variation are kept as similar as
possible for all groups.
• Tiger shrimp example:
• Other than the nutrient and salinity level,
items like the size of the tank, the amount of
water in the tank and temperature of the
water would ideally be identical or as close
to identical as possible.
 Control ≠ control group
 Randomization
Treatment(s) should be assigned
randomly to create groups that are as
equivalent as possible…it also reduces
confounding (more on what that is
later…).
 Replication
• Treatments should be applied to the
largest number of subjects possible.
• EX) Two tanks getting each
combination of nutrient and salinity
level is replication.
• NOTE: Replication can also refer to the
ability to repeat an experiment under a
different set of circumstances.
 Blocking
• A block is a group of experimental units or
subjects that are similar in ways that are
expected to affect the response to the
treatments. Random assignment of treatments
is carried out separately within each block.

• Blocking is not required in an experimental

design. There are times when it may be very
advantageous to utilize blocking, though.
 OK, how do we know if the
treatments made the difference?
• Differences are considered
statistically significant if they are
bigger than what might be
expected from randomization
alone.
If we don’t believe that something
occurred only by chance, we’ll
consider it statistically significant.
EX) If the tiger shrimp with nutrient A
put on twice as much weight compared
to the shrimp which got nutrients B
and C, we would likely attribute the
difference in weight to nutrient A and
not to another unmeasured
characteristic.
Pick up from here next class day.
 Differences between sample
surveys & experiments
• Both randomize, but for different reasons
• Surveys measure a parameter by
randomly selecting people that are
representative of a population
• Experiments do not randomly choose
subjects…the randomization occurs with
treatments
 Control Group
• A group that gets a baseline treatment
(“nothing happens”). This baseline
measurement is called a control
treatment.

Blinding
• Knowledge of treatments is unknown.
This is done to reduce, if not eliminate,
personal biases from influencing an
outcome.
 Blinding continued…
• Groups which can affect the outcome of
an experiment:
1. Those who influence the results
(subjects, administrators)
2. Those who evaluate the results (judges,
treating physicians)
3. The subjects themselves, if they are
aware of their particular treatment.
• Single-blind: the experimenters are aware of
which subjects are receiving the treatment,
but the participants of the study are not.
• EX) An investigation into whether people can
determine a taste difference between low-fat
yogurt and high-fat yogurt is being carried out.
The two types of yogurt are given to
participants in identical containers,
participants try both and are then asked which
one is low-fat. Those investigating know which
is which, but the participants do not.
• Double-blind: both the experimenters and
participants are unaware of who is receiving
the treatments and who is not
• EX) Patients are randomly assigned to one of two groups
- one taking an experimental medication, the other
taking a pill identical in color, taste, size, etc., but has
none of the experimental medication in it. None of the
patients are aware of the group to which they've been
allocated.
• Doctors recording the symptoms reported by the
patients and taking measurements don't know which
patients are in which group.
• This is double-blind. (Researchers who did the assigning
are aware but are outside of the experimental process.)
 Placebo
• A “fake” treatment that mimics the
treatment being tested. Some examples
of placebos (not limited to these…)
– Sugar pills that mimic an experimental
medication
– Saline solution that mimics an
experimental vaccine.
 Types of experimental design
• Completely randomized design: all
subjects have an equal chance of
receiving any treatment.

• Randomized block design:

Randomization of treatments only
occurs within blocks.
 Completely randomized
• All subjects have an equal chance of
receiving any treatment.

• EX) You want to study the effect of

using differing doses of fertilizer on 24
tomatoes that came from the same
nursery. 8 get no fertilizer, 8 get a half-
dose, 8 get a full dose.
 Randomized block design
• Randomization of treatments only occurs
within blocks.
• EX) If the 24 tomatoes came from different
nurseries (say 12 came from Nursery A and
12 from Nursery B), we could block the
tomatoes by nursery, then randomly assign
fertilizer treatments within each block.
 The term you use matters…
• Experimental design vocabulary and sample
survey vocabulary have similar sounding
terms, but they must be used correctly.
Sample survey term Experimental design equivalent
Simple random sample Completely randomized
Stratified random sample Randomized block

• Stratify is to sampling, as block is to

experimental design.
 Matched pairs:
• Subjects in the experiment are grouped into
pairs based on some variable they “match”
on. Then, within each pair, subjects are
randomly assigned to different treatments.
• EX) Suppose researchers want to know how a
new diet affects weight loss compared to a
standard diet. Subjects could be paired by
gender, weight and age as best as possible,
then assign one of the pair treatment A, and
the other treatment B.
 Confounded factors
• outside influence(s) that interact with the
explanatory variable and response variable in a
way that makes it difficult to determine cause
and effect
• EX) If you are researching whether a lack of exercise
has an effect on weight gain, diet would likely be a
confounding factor that would impede determining
the impact of lack of exercise on weight gain (one’s
diet could impact exercise frequency, and diet could
impact weight as well).
 Lurking vs. Confounding:
• Lurking: an association is created by an
outside variable that gives the appearance
that x causes y (remember the shoe size and
reading ability example from a while back... )

• Confounding: a variable associated in a non-

causal way with a factor and affects a
response (is someone’s weight gain due to
lack of exercise or diet?)