SIT

TOGETHER
W I T H YO U R
P RO J E C T
G RO U P S
T O DAY ! ! !
 Ways to improve

I N T E R N A L A N D E X T E R N A L VA L I D I T Y
AG E N DA
• Internal validity and control variables
• Randomization and blinding
3
• External validity
• Group activity: analyze a real study
I N T E R N A L VA L I D I T Y

❖ Internal validity is the degree to which we can appropriately conclude that the
changes in X caused the changes in Y. Causality is the state of nature in which an
active change in one variable directly changes the probability distribution of
another variable (Seltman, 2018)
• Association (non-zero correlation) does not necessarily imply causation. 4

❖ When 2 variables X and Y are found to be associated/correlated → some

possibilities:
• X causes Y
• Y causes X
• Another variable Z causes both X and Y
• You made some mistake(s), they are not even correlate.
I M P RO V I N G I N T E R N A L VA L I D I T Y
❖The no1 threat to internal validity is confounding, i.e. more than one thing (this
one thing is the independent variable that researchers want to study) is different on
average between the groups to be compared.
❖Main approaches to improve internal validity:

Randomization
Have more controlled 5
(including block-
variables
randomization)

Blinding: Single -
Control group
Double - Triple
 I N T E R N A L VA L I D I T Y – B E T T E R
C O N T RO L O F VA R I A B L E S
❖ “Falling objects” experiment
• When done in air, more than one
thing is different between the
6
subjects: mass and air resistance
→ air resistant is a confounding
variable → invalid conclusion
that heavier objects fall faster.
• Done in vacuum: remove the
confounding → improve internal
validity.
I N T E R N A L VA L I D I T Y – B E T T E R
C O N T RO L O F VA R I A B L E S
In this report, the phrase “after
controlling for other factors” appears
8 times!
❖ The more controlled variables you
can identify, the better. 7

❖ Identify potential influencing factors

→ design experiment that can
control them
❖ Be aware:
• There might be factors we did not
think about
• We know about some factors but
cannot control them
AG E N DA
• Internal validity and control
• Randomization and blinding
8
• External validity
• Group activity: analyze a real
study
R A N D O M I Z AT I O N
❖ Randomization can help to reduce confounding and improve internal validity.
• Good for when you don’t know all the details of the relevant factors that
might confound your results, randomization aims to average out these
differences between groups.
9
❖ Performed at which step? Whenever the question “ which subject….” comes up.
In general, 2 key steps: sampling and treatment assignment.
• Sampling: which subjects in the total population to involve in the
experiment? (if sample size = population size then ignore this step); difficult
when we do not have a complete list of the population.
• Treatment assignment : which subjects get which treatment?
R A N D O M I Z AT I O N
❖ Commonly by computerized-random number generators: identify the subjects
→ assign each with a number → let computer pick number randomly

Food
A 10

Randomized! Randomized! Food

A
sampling assignment
Food
B

Food
B
Population Sample Units
B L O C K R A N D O M I Z AT I O N
• Block randomiz ation is a sub-class of randomization, done when we know or have good
reasons to believe a certain factor has significant impact on the result: separate the
subjects into blocks based on the factor → then randomize within the blocks.

11
Consider
2 blocks
separately

Breed X Breed Y Randomization Sampling & Randomization

assignment
N O T E S A B O U T R A N D O M I Z AT I O N
❖ Only in truly homogeneous population can we get “truly random” sampling or
assignment, i.e., the sample is a perfect representation of the population, the units
are perfect representations of the sample.
• (we’ll take “homogenous” here to
mean that the subjects in the 12
population are similar enough that
there are no reasons to believe they
will respond differently)
❖Most populations are heterogenous, thus even when we use methods to randomize our
sampling and assignment, there is always a chance that the resulted grouping would be
biased.
• Example: in a company with both male and female employees, a study (on sales tactics for
example) using randomized assignment could ends up with one group mostly male and
other group mostly female.
I N T E R N A L VA L I D I T Y – B L I N D I N G

❖ With human subjects, blinding is useful, since knowing about the

treatments can (and most of the time, do) affect human’s response,
subconsciously or consciously (i.e. they’re biased)
❖ (single) Blinding: the subjects do not know which treatments they
are getting.
• Example: patients do not know if they’re getting placebo; in 13

psychology: clever design that camouflage the treatments

❖ Double blinding: both the subjects and the experimenter (who
handles the actual treatments) do not know which treatment they
are involved with → prevent potential biases in administering the
treatment.
• Example: knowing that a drug in testing is much more expensive
than another might cause the nurses to be extra careful when
giving the drug to patients → double blind (patients and nurse)
I N T E R N A L VA L I D I T Y – B L I N D I N G
❖ Triple blinding (not used as often): subjects, experimenters, and analysts don’t know about
the treatment → prevent potential bias in analysis as well.
• Example: an analyst might make judgement (explicitly or implicitly) based on his/her
existing knowledge about similar scenarios and think “this cannot be right…”. Such
thoughts could influence their actions during the subsequent steps.
• (note: double and triple blinding also help preventing conflict of interest issue, more on this 14
next time).
C O N T RO L G RO U P
• Note: potential confusing usages of the word “control”: controlled variables & control
group
• A control group is a treatment group that represents some appropriate baseline treatment.
It is hard to describe exactly what “appropriate baseline treatment” means, and this often
requires knowledge of the subject area and good judgment. (Seltman, 2018)
• The focus is the independent variable: the control group would either not get this 15

treatment or have the independent variable fixed at a benchmark level.

• Example: placebo control group in drug testing phase 3 (as the paper on previous slide)
In our example it could be the pigs fed with the current type of food the
farmer is using.
• Useful , but not always needed (meaning it’s necessary in some cases, not all).
• When used, the subjects involved in the control group should also be randomly assigned.
AG E N DA
• Internal validity and control
• Randomization and blinding
16
• External validity
• Group activity: analyze a real
study
E X T E R N A L VA L I D I T Y
❖ External validity is synonymous with generalizability: how applicable is the result of the
experiment outside of the studied sample.
❖ The researchers should consider external validity of their results when concluding
about the applicability/scope of the study:
• Too narrow (only applicable to the sample): not really valuable to scientific
knowledge or of interest to other people. 17

• Too broad (claim that results would be true for a large population beyond the
sample) → risk of over-generalization.
• Depends on sample size and how representative it is of the population
SAMPLE SIZE
❖ Havinga large sample size is the first step in improving external validity. A larger
sample would more likely to represent the population.
• Example: population: FUV students.
• Randomly sample 3 students (i.e. sample size = 3) vs randomly sample 30 vs 300
(more when we learn statistics)
18
❖ So how large is “large enough”? There is no hard-and-fast rules:
• For some specific case such as the FDA drug trial there’re standards.
• Generally speaking, the smaller the population, the larger the sampling ratio needed.
• For populations under 1000, a minimum ratio of 30% is advisable to ensure
representativeness of the sample.
• For larger populations, such as a population of 10,000, a comparatively small
minimum ratio of 10 percent of individuals is required to ensure representativeness of
the sample.
 E X T E R N A L VA L I D I T Y
• High internal validity is a requirement for every good experiment, but the requirement for
external validity is a bit more case-by-case, depending on the goal of the study/the standard
practice of that research area (paradigm strikes again!)
• Example:
Study on a new drug’s effect on patients → high Study on how to MAKE that drug through chemical
requirement for external validity due to synthesis → we synthetic chemists normally don’t
heterogenous population & high-stake even think about external validity (reliability is a 19
much more important requirement here)
consequence (human health)
 C A S E S O F C AU T I O N S
❖3 cases of low generaliz ation worth talking about more (Seltman, 2018):
• Non-participation
• Drop-out
• Environment and treatment application variability
❖ Non-participation: People who agree to take part in a study might share something in 20
common, which may be a factor influencing the results → sample less representative of
broader population.
• Example 1: A study about effect of coffee, people who
agree to participate might be those who like coffee and
have been consuming it regularly for a long time.
• Example 2: Paid survey about consumer habit, those
who agree might have a certain view on money (that
influences them to spend the time to go through the
survey with that amount of payment)
C A S E S O F C A U T I O N S – D RO P - O U T
❖ Drop-out: people who start the study but did not complete it.
• This reduces the sample size → in general that’s bad for external validity
• The people who drop-out might share some common characters which
affect the randomization, this can lower both external and internal validity.
Example:
• A study on impacts of different genres of music on students’ learning effectiveness: groups 21
of 20 participants each, there’s one group assigned to listen to Heavy metal.
• After 1 week, 15 participants in this group dropped out.
• Sample size much smaller: ¼ of the original → external validity drops
• Those dropped out might be people who can’t stand this genre, leaving 5 Heavy metal
fans in the study group, thus any conclusion from this group is not representative of
broader student body → external validity drops
• Comparison of impacts of different genres on students are not conclusive, for example if
these 5 students out-performed other groups, it might be because they get the genre they
like → internal validity drops.
C A S E S O F C A U T I O N S – E N V I RO N M E N T
A N D T R E AT M E N T A P P L I C AT I O N
❖ Environment and treatment application variability: this is not about the subjects, but
the quality requirements of the environment: the lower these requirement, the more
the research result can be applied to environment other than the one in the study.
❖ Examples:
22
• A new chemical reaction that could be use with many chemicals, but requires very
high pressure;
• A clinical treatment that requires lab-like controlled which is difficult to obtain in a
normal doctor’s office;
• A drug that requires special storage;
• A study on effect of online learning that uses stable, high speed internet connection
…..
AG E N DA
• Internal validity and control
• Randomization and blinding
23
• External validity
• Group activity: analyze a real
study
L E T ’ S A N A LY Z E A
PA P E R

In groups, let’s analyze the paper “The

Effect of Listening to Music On
Concentration and Academic 24
Performance of the Student: Cross-
Sectional Study On Medical
Undergraduate Students”, with a focus
on design and execution, especially the
validity of the research.
Try to identify the weak points in this
study and AVOID making such mistakes in
your projects.