Designing Studies

ILO(Intended Learning Outcomes)

This section covers…

• Sampling and Survey

• Experiments

• Application of statistics
1. Sampling and Surveys

• A census collects data from every individual in the population.

• A survey collects sample data from the population of all individuals

about which we desire information.

• The goal of a sample survey is inference.

1. Sampling and Surveys -Typical bad sampling:

Convenience sample:

• Choosing individuals from the population who are easy to reach results
in a convenience sample.

Voluntary response sample

• A voluntary response sample consists of people who choose themselves

by responding to a general invitation.
1. Sampling and Surveys -Typical bad sampling:
1. Sampling and Surveys -Typical bad sampling:

• How to get a good sample?

① Unbiased- reliable
② Accurate- useful
③ Time saving- economical
1. Sampling and Surveys -How to Sample Well

Random Sampling

• Random sampling involves using a chance process to determine which

members of a population are included in the sample.

Simple Random Sample (SRS)

• A simple random sample (SRS) of size n is chosen in such a way that

every group of n individuals in the population has an equal chance to be
selected as the sample.
1. Sampling and Surveys -How to Sample Well

How to Choose a Simple Random Sample?

By technology: a random number generator

a calculator

By hand: Table D
1. Sampling and Surveys -Other Random Sampling Methods
1. Sampling and Surveys

Stratified random sample and strata( 分层随机样本 )

For example, in a mean weight of a group of 60 men and 40 women.

Stratified random sampling works best when the individuals within each stratum
are similar with respect to what is being measured and when there are large
differences between strata. Because it makes the sample more representative.
1. Sampling and

• Surveys Cluster sample and clusters( 聚类随机样本 )

• When populations are large and spread out over a wide area. Clusters is
time-saving and inexpensive.

Example: the mean height of

16-year-old teens in Australia.

 1. Sampling and Surveys -Other Random Sampling Methods

Stratified random sample and strata( 分层随机样本 )

• start by classifying the population into groups of similar individuals,

called strata. Then choose a separate SRS in each stratum. similar
within, different between

Cluster sample and clusters( 聚类随机样本 )

• start by classifying the population into groups of individuals that are

located near each other, called clusters. Then choose an SRS of the
clusters. Just like the population.
1. Sampling and Surveys -Other Random Sampling Methods
1. Sampling and

A systematic random sample is a method in which sample members from

a population are selected according to a random starting point and a
fixed, periodic interval
1. Sampling and Surveys -Other Random Sampling Methods

 Describe how you would select 80 students

to complete the survey
1. Sampling and Surveys -Inference for Sampling

Key to infer population from sampling

• Larger samples size generally provides better information

• Avoid bias in choosing a sample.

1. Sampling and Surveys -Inference for Sampling

What Can Go Wrong?

Undercoverage

• Undercoverage occurs when some members of the population cannot

be chosen in a sample.

Nonresponse

• Nonresponse occurs when an individual chosen for the sample can’t be

contacted or refuses to participate.
1. Sampling and Surveys

These scenarios cause bias ：

• Convenience sample

• Voluntary response sample

• Undercoverage

• Nonresponse

• Question wording
1. Sampling and Surveys –How to design a survey*

Respondents

• Engaged?(Do they care?)

• Well-informed(Do they know enough information?)

• Opinions fixed?(Will they change their minds later?)

• Give honest answers?(sensitive questions may not receive a

completely honest answer)
1. Sampling and Surveys- Practice
1. Sampling and Surveys -Inference for Sampling
 2. Experiments
Observational study and experiment
An observational study observes individuals and measures variables of
interest but does not attempt to influence the responses.
An experiment deliberately imposes some treatment on individuals to
measure their responses.

When our goal is to understand cause and effect,

experiments are the only source of fully convincing
data.

observational study experiment

2. Experiments
2. Experiments-Practice
• Does reducing screen brightness increase battery life in laptop
computers? To find out, researchers obtained 30 new laptops of the
same brand. They chose 15 of the computers at random and adjusted
their screens to the brightest setting.

• The other 15 laptop screens were left at the default setting—moderate

brightness. Researchers then measured how long each machine’s bat-
tery lasted.
Was this an observational study or an experiment? Justify your answer.
2. Experiments

• In a cause-and-effect study, explanatory variable can explain the

response of response variable .

• Confounding occurs when two variables are associated in such a way

that their effects on a response variable cannot be distinguished from
each other.

• Observational studies of the effect of an explanatory variable on a

response variable often fail because of confounding between the
explanatory variable and one or more other variables
 2. Experiments-Practice
• Does eating dinner with their families improve students’ academic
performance? According to an ABC News article, “Teenagers who eat
with their families at least five times a week are more likely to get better
grades in school. This finding was based on a sample survey conducted
by researchers at Columbia University.
I. Was this an observational study or an experiment? Justify your
answer.
II. What are the explanatory and response variables?
III. Explain clearly why such a study cannot establish a cause-and-effect
relationship. Suggest a variable that may be confounded with whether
families eat dinner together.
2. Experiments

• An experiment is a statistical study in which we actually do something (a

treatment) to people, animals, or objects (the experimental units) to
observe the response. Here is the basic vocabulary of experiments.
2. Experiments

• Experiments can give good evidence for causation.

 2. Experiments
Researchers at the University of North Carolina were concerned about the
increasing dropout rate in the state’s high schools, especially for low-income
students.
To change this perception, researchers developed a program called CareerStart.
The big idea of the program is that teachers show students how the topics they
learn get used in specific careers.
To test the effectiveness of CareerStart, the researchers recruited 14 middle
schools in Forsyth County to participate in an experiment. Seven of the schools,
determined at random, used CareerStart along with the district’s standard
curriculum. The other seven schools just followed the standard curriculum.
Researchers followed both groups of students for several years, collecting data
on students’ attendance, behavior, standardized test scores, level of
engagement in school, and whether or not the students graduated from high
2. Experiments
What are the effects of repeated exposure to an advertising message?
The answer may depend on both the length of the ad and on how
often it is repeated.
An experiment investigated this question using 120 undergraduate
students who volunteered to participate. All subjects viewed a 40-
minute television program that included ads for a digital camera.
Some subjects saw a 30-second commercial; others, a 90-second
version. The same commercial was shown either 1, 3, or 5 times
during the program. After viewing, all the subjects answered
questions about their recall of the ad, their attitude toward the
camera, and their intention to purchase it.
2. Experiments
How to Experiment Badly
2. Experiments

• Outside the lab, badly designed experiments often yield

worthless results because of confounding.
2. Experiments

How to Experiment Well

• The remedy for the confounding is to do a comparative experiment.

• Comparison alone isn’t enough to produce results we can trust.

• The solution to the problem of bias in sampling is random selection.

2. Experiments
2. Experiments

• We should control for the effects of some variables by keeping them the
same for both groups.

• Replication: use enough experimental units to distinguish a difference in

the effects of the treatments from chance variation due to the random
assignment.
2. Experiments

Effectiveness of control

Assigned group Subjects choose group

2. Experiments

• Does regularly taking aspirin help protect people against heart attacks?

• Does beta-carotene help prevent some forms of cancer?

 2. Experiments

1. The subjects in this experiment were 21,996 male physicians

2. One- fourth of the subjects were assigned at random to each of these

treatments.
Identify what each principle of
3. experimental design was used:
 Comparison
 Random assignment
 Control
 Replication
 2. Experiments

• Notice that a CDR does not require that each treatment be assigned to an equal number of
experimental units.
• It does specify that the assignment of treatments must occur completely at random
2. Experiments
• Many utility companies have introduced programs to encourage energy
conservation among their customers. An electric company considers
placing small digital displays in households to show current electricity use
and what the cost would be if this use continued for a month. One
cheaper approach is to give customers a chart and information about
monitoring their electricity use from their outside meter.
Describe a completely randomized design involving 60 single-family
residences in the same city whose owners are willing to participate in
such an experiment.
Write a few sentences explaining how you would implement your design.
 2. Experiments- What can go wrong
The response to a dummy treatment is
called the placebo effect
Knowing that they are getting “just a
placebo” might weaken the placebo
effect and bias the experiment in favor
of the other treatments.

• Whenever possible, experiments with human subjects

should be double-blind.
 2. Experiments

• The idea of a double-blind design is simple. Until the experiment ends and the
results are in, only the study’s statistician knows for sure which treatment a subject
is receiving.
2. Experiments

In some cases, a single-blind experiment is inevitable.

 2. Experiments

• In an experiment, researchers usually hope to see a difference in the

responses so large that it is unlikely to happen just because of chance
variation.

• If they are, we call them statistically significant( 统计显著 ).

2. Experiments

• The great advantage of randomized comparative experiments is that

they can produce data that give good evidence for a cause-and-effect
relationship between the explanatory and response variables.

• Although in general a strong association does not imply causation.

• A statistically significant association in data from a well-designed

experiment does imply causation
2. Experiments

CRD

RBD(randomized block design)

2. Experiments

• When blocks are formed wisely, it is easier to find convincing evidence

that one treatment is more effective than another.

• Control what you can, block on what you can’t control, and randomize to
create comparable groups.
2. Experiments
 2. Experiments

① This ignores the differences between men and women, which would
probably result in a great deal of variability in responses to the advertising
in all three groups. Blocking will account for the variability in responses to
advertising due to gender.

② We randomly assign the 120 women into three groups of 40, one for each of
the advertising treatments.

③ Write the women’s names on 120 identical slips of paper, place the slips in a
hat, and mix them well. Pull out 40 slips to determine which women will
view Ad 1. Then….
2. Experiments

• A common type of randomized block design for comparing two

treatments is a matched pairs design.
2. Experiments

Keys of matched pairs designs

① Sometimes each “pair” in a matched pairs design consists of just one

experimental unit that gets both treatments one after the other.

② In that case, each experimental unit serves as its own control.

③ The order of the treatments can influence the response, so we

randomize the order for each experimental unit.

④ the design of the study determines the appropriate method of analysis.

2. Experiments

• A Completely randomized design:

• A matched pairs design:

3. Using study wisely

Scope of Inference

• What type of inference can be made from a particular study? The answer
depends on the design of the study.
3. Using study wisely
 3. Using study wisely
A small-town dentist wants to know if a daily dose of 500 milligrams (mg) of vitamin C will
result in fewer canker sores in the mouth than taking no vitamin C.
Design 1: Get all dental patients in town with appointments in the next two weeks to take
part in a study. Give each patient a survey with two questions: (1) Do you take at least 500
mg of vitamin C each day? (2) Do you frequently have canker sores? Based on patients’
answers to Question 1, divide them into two groups: those who take at least 500 mg of
vitamin C daily and those who don’t.
Design 2: Get all dental patients in town with appointments in the next two weeks to take
part in a study. Randomly assign half of them to take 500 mg of vitamin C each day and the
other half to abstain from taking vitamin C for three months.
Design 3: Select a random sample of dental patients in town and get them to take part in a
study. Divide the patients into two groups as in Design 1.
Design 4: Select a random sample of dental patients in town and get them to take part in a
study. Randomly assign half of them to take 500 mg of vitamin C each day and the other half
to abstain from taking vitamin C for three months.
 3. Using study wisely

• A well-designed experiment tells us that changes in the explanatory variable

cause changes in the response variable.

• But in some cases, it isn’t practical or even ethical to do an experiment.

Consider these important questions:

① Does texting while driving increase the risk of having an accident?

② Does going to church regularly help people live longer?

③ Does smoking cause lung cancer?

• Lack of realism in an experiment can prevent us from generalizing its results.

3. Using study wisely

What are the criteria for establishing causation when we can’t do an

experiment?

① The association is strong.

② The association is consistent

③ Larger values of the explanatory variable are associated with stronger

responses.

④ The alleged cause precedes the effect in time.

⑤ The alleged cause is plausible.

3. Using study wisely
• End of chapter