QUANTITATIVE RESEARCH METHODS

2. Sampling mechanisms

In experiments, it´s essential to select a sample from the population, because it´s impossible to
work with all the elements about our study is. Firstly, we must make a concrete definition of the
population under study, which will have consequences for the generalizability of the results
(external validity). And then we must draw (extraer) a sample from the population, bearing in mind
their size and what mechanisms is better to use for sampling.

To study the population, we must choose the research elements, for which we have to think who
the research is about. Population may pertain to individuals, pairs, groups, collectivities.
Also, is necessary to create an operational definition of the population where we can motive our
choice. That implies a reduction of the population and the goals (funnel metaphor) and has
consequences for our conclusion (external validity, that’s only valid for the defined population
under study).
The operational definition includes different aspects:
• Context. Time and location of the research has to be specified as far as it influences the
process under study. For example, a learning performance for school children would be
schools in the city or at the countryside.
• Characteristics of the research elements that should be observable/measurable.
- Inclusion criteria. Characteristic should be present. For example, if we are studying
dyslexia and we obtain a low score on a reading test, we must know what test it´s
used.
- Exclusion criteria. Characteristic should not be present. For example, not having a
low IQ for dyslexia too.
• Validity of these characteristics. In the dyslexia example we have to know when a person
has dyslexia and find arguments/features in the literature.
• Population. Set of research elements to which the theory/statements of the researcher
pertain to (cf. inclusion and exclusion criteria).
• Sample frame. Set of elements that has a non-zero probability of being selected.
• Sample. Subset (subconjunto) of elements that have been selected.

It is often impossible to work with whole population, except for census. For this reason, we use a
sample, which is a selected set of elements from de population.

13
 QUANTITATIVE RESEARCH METHODS

N: total number of elements in population (sample frame)

n: number of selected elements in the sample
Sample fraction: n / N (always < 1) shows the relative
size of the sample. For example, it says if n = 100 is a
large sample, paying attention on the value of N (no
when N = 11.000.000).
The goal of sampling is to make inferences of
population characteristics based on it that help us to
solve the researching problem.
When the sample is representative for the population, sample characteristics (e.g., means,
differences in means between groups) are informative for population characteristics.
Population Theory

Sample Experiment

2.1. HOW TO GET A REPRESENTATIVE SAMPLE?

To get a representative sample we must limit the influence of randomness (e.g., selecting a sample
of only tall individuals). Aselect sampling mechanisms (see later): who is selected is determined at
random and every element has the same probability of being selected.
Also, we must know that larger samples are more
representative (n/N) and estimates based on the
sample will differ to a smaller extent over
samples. It´s very important because makes the
study more efficient and precise and confer a
smaller variance of the sampling distribution.
For example, if we are estimating the average length of a group of people and we take a small
sample, there will be large variability in mean height (over samples) and the same is true when
not taking a representative sample (e.g., only tall people).
Another example will be the Referendum Catalonia-Spain, where 89,3% said yes and 42,6% voted.
Hence, we know for sure that 38% of all potential voters (hence population) said yes, but it´s
difficult to infer conclusion because we don’t know if there are no votes. Contrary to, in Brexit,
51,9% said yes and 72,2% voted. Hence, 37,5% surely said yes.

2.2. SAMPLING MECHANISMS

Simple random sampling

Systematic sampling
a) Aselect sampling procedures
(equal selection Stratified sampling
probabilities)
Multi-stage sampling
Cluster sample

b) Select sample procedures

c) Single-case studies

14
 QUANTITATIVE RESEARCH METHODS

Use for instance random generator to

select 8 items.

Simple random e.g., 2d, 7th, 13th, 14th, 27th, 29th, 31st,
sampling (SRS) 41st
Distribution of certain characteristics in
population can be not represented well
(especially with small n), because there
are many A’s, just one B.
Start with for example the second (SRS)
and go further in steps of 6.

Systematic sampling Distribution of certain characteristics in

population can be not represented well
(especially with small n), because more
C’s than A’s & B’s.

Group the items (= forming strata). Take

3 of each, e.g., by means of simple
random sample. In example equal
number of items in each strata, is not
always the case.
Proportional Stratified sampling and
Stratified sampling
sample size of 10:
A’s = 40%, B’s = 40%, C’s = 20%
4 A’s, 4 B’s and 2 C’s
Non-proportional Stratified sampling
and sample size of 12:
4 A’s, 4 B’s, 4 C’s
Do SRS to decide which line (e.g.
Cluster sampling class/school/…) you select (e.g. 2).
Select everyone from this line.

3.1. SIMPLE RANDOM SAMPLING

Most statistical tests assume this type of sampling: t-test (difference of 2 means), F-test (ANOVA)
or regression. In these mechanisms, every element of the population has the same selection
probability (= probability to be sampled).
For example, if we take balls from an urn we have to distinguish:
a) Sampling without replacement. Same element can be selected only once.
b) Sampling with replacement. Same element can be selected twice (or three times).

15
 QUANTITATIVE RESEARCH METHODS

Advantages Disadvantages
Quite easy to carry out (realizar). Possibility that certain characteristics in
population are not represented well
It is a prerequisite for most standard statistical
(especially with small n), at random selecting
tests (selected elements are independent): t-
many rich people.
test, ANOVA, linear regression.
Not efficient in fieldwork
It doesn’t assume knowledge about the
population of interest (only n and N should be
known in advance), so no assumption needed
for characteristics of population.

3.2. SYSTEMATIC RANDOM SAMPLING

We use it when we have an ordered list of all elements in the population, ordered along one or
more characteristics like age, social status, diploma or IQ.
We need to know N and n (e.g., N = 100, n = 5), the sample fraction: n/N (e.g., 5/100 = .05), the
interval length f: f = N/n (e.g., f = 100/5 = 20) and an ordered list of all elements in the population.
Procedure
1. Draw a random number g from interval [1 … f] (e.g., a number between 1 and 20).
2. Select corresponding element on the list (e.g., g = 8), starting from g (e.g., 8) take every
fth element (e.g., 28, 48, 68, 88), and selected elements are: 8, 28, 48, 68, 88.

Advantages Disadvantages
For the characteristic(s) that is represented by Watch out for periodicity, sampling every 7th
the list, a representative sample is selected. day: always on the same day.
But some (extreme) samples become
Not representative for some small
impossible to select, for example, only rich
subpopulations. For example, not enough rich
people.
people in sample (because not many in
Will influence the precision when the population). A solution for this may be
characteristic is related to the variable of stratification (non-proportional).
interest because it reduces negative effect of
A bad list may lead to a lower precision.
randomness in comparison to simple random
sampling. But many extreme values might
have a negative effect.
Still aselect sample (equal selection
probabilities): standard statistical techniques
can be applied.

16
 QUANTITATIVE RESEARCH METHODS

Good list. Monotonically related to the Bad list. Cyclic behaviour, studying suicide
process, size of the house for estimating and sampling every 7th day, and yields a
income, and yields a heterogeneous sample homogeneous sample (not efficient/low
(efficient/high precision). precision).

3.3. STRATIFIED SAMPLING

Procedure
Divide population in strata (=subpopulations). For example, old and young people; men and
woman; social status groups = take a simple random sample (without replacement) from each
stratum.
➔ Why does it work?
Creation of homogeneous strata (elements from the same stratum
are more alike than elements from different strata), stratification
variable should be related to the process (e.g., social classes to study
income).
Sampling from each stratum yields a heterogeneous sample, extreme
samples become impossible, increased statistical precision.
Two variants
a) Proportional. Take the same % of people from each stratum (more people are sampled
from large strata).
b) Non-proportional. Take the same number of people from each stratum

Proportional
Advantages Disadvantages
Stratification characteristic(s) will be well Choice of stratification characteristic depends
represented in sample. on theory. It should be related to variable of
interest and watch out for overstratification
There are more difficult to compare groups
because group sizes differ in the sample.
Non-proportional
Advantages Disadvantages
Easy to compare groups (they have equal Representativity of sample for population
number of elements in the sample). have unequal selection probabilities: results in
bias (sesgos) which you can solve by weighting
oversampled elements less than
undersampled elements (i.e., weighted mean).
However, leads to a decrease in precision.

17
 QUANTITATIVE RESEARCH METHODS

Remark
Post-stratification is often used reweighting (simple random) sample based on difference in
sample and population distribution for important characteristics (e.g., age: more old people in
sample than expected), give a larger weight to younger people than to older. Also, it is less
efficient than ordinary stratification, but often more efficient than simple random sampling
(without).
Danger post-stratification
Despite falling behind in most national surveys, Mr. Trump has generally led in the U.S.C./LAT poll.
One reason: post-stratification:
– For some states, not much afro-american people in the sample
– Especially the case for Illinois.
– Therefore, they are weighted more.
Consequently, a 19-year-old afro-american Trump supporter in Illinois is weighted as much as 30
times more than the average respondent, and as much as 300 times more than the least-weighted
respondent.
Exercise
A researcher wants to study the motivation of students in the Master of Educational Sciences. This
population consists of 170 students of which 10% are male. The researcher wants to select a
stratified sample (based on gender) of 20 students.
• What is the value for n, N, and the sample fraction?
The sample size (n) equals 20.
The population size (N) is 170.
The sample fraction (f) equals 𝑓=𝑛/𝑁, hence=20/170,hence= 0.1176.
• What will be the number of males and females in (a) aproportional stratified sample, and
(b) a non-proportional stratified sample.
A proportional stratified sample for gender: distribution of men/women in the sample is the same
as the distribution of men/women in the population. As the population contains 10% men, our
sample should also contain 10% men. For a sample of 20 people, we should select 2 males and
18 females.
A non-proportional stratified sample for gender: distribution of men/women in the sample is not
the same as the distribution of men/women in the population. When goal is to compare males
and females, sampling an equal amount of males and females is recommended for statistical
reasons. Hence, 10 men and 10 women.
• Which sampling mechanism do you prefer (proportional or disproportional)? Why?

3.4. CLUSTER SAMPLING

Procedure
Cluster sampling is a special case of multi-stage sampling
which take a simple random sample from the primary elements
for each selected primary unit: select all secondary elements,
difference with multi-stage sampling: only a simple random
sample from secondary elements is selected.

18
 QUANTITATIVE RESEARCH METHODS

Advantages Disadvantages
Efficient for fieldwork. Elements within clusters are correlated, very
homogeneous samples decrease precision.
E.g., only a few schools should be visited.
Individuals within clusters do not add much
information, statistical techniques for
correlated data are needed.

3.5. MULTI-STAGE SAMPLING

Procedure
a) Existing groups in the population yield nested data (proporcionan datos anidados,
incluidos dentro de los otros): students are nested in classes; individuals are nested in
households and those are nested in cities.
b) Select an initial sample (1st stage) that is based on groups, which are called the primary
sampling units (PSUS).
c) Create a 2nd stage sample by drawing a subsample from each selected PSU in the 1st stage
sample.
d) By repeating this operation, you can select a higher-stage sample.

At each level choice between proportional

and non-proportional has consequence for
the selection probabilities and account for it
in the analysis.

Advantages Disadvantages
Very efficient for fieldwork. Only a few schools Selected elements are correlated. Two people
(classes) should be visited. from the same group (e.g., school) have more
in common than two people from different
The correlation (intra cluster) between the
groups (reason: they share the group effect
different elements belonging to the same
and associated common influences).
cluster can be computed and studied. Gives an
idea about how much is lost in statistical Correlated elements do not contribute much
efficiency. extra information (yields homogeneous
samples).
Standard statistical techniques cannot be
used.
Muestreo estratificado: un tipo de muestreo de probabilidad (questionpro.com)

19
 QUANTITATIVE RESEARCH METHODS

4.1. NOT ASELECT

Selection probabilities differ across elements in the population, problem of (self)-selectivity (not
determined at random) and is a severe threat for generalization (external validity), watch out when
selectivity is related to the process under study.
Convenience sample: take a sample that is easy to reach, but may yield a bias (puede producer un
sesgo).

Examples
• Studying IQ and creativity and taking parents from children at sailing camps.
• Mainly rich people go to sailing camps.
• Bias because relation between IQ and creativity may be different for rich and poor
families.
• Not a problem when investigating shoe size and taking a…
• Snowball sample (for populations difficult to reach, addicted people): based on one
element, selecting the next element.

a) Single-subject research
• Quantitative research is possible when the person is measured multiple times
over time (longitudinal data).
• Can for instance be used to study processes that are rare
b) Qualitative case studies
• Studying a single or a few cases (rare phenomena)
• More profound and/or explorative. Studying a patient with a rare brain
disease/injury.
• Less control here (versus an experiment).

It depends on:
1. Does the sampling mechanism used yields a bias?
• Operational definition of research elements: clearly defined (time period, context,
inclusion and exclusion criteria).
• Type of sampling: aselect or select.
• Size of the sample.

20
 QUANTITATIVE RESEARCH METHODS

• Non-response (selected but not possible to contact or refusals): is the non-

response systematic/selective? Is it related to the process under study?
- Studying intervention effects for behavioural problems.
- 40% non-response because it was too difficult to participate (because of
severe behavioural problems).
- When an intervention effect is observed: it is only true for families with small
and moderate problem behaviour.
- It is not clear whether the effect may be generalized to families encountering
large behavioural problems.
- Is a problem of external validity (generalization).

6.1. FACTORS THAT INFLUENCE IN SAMPLE SIZE

a) Population size (N).

b) Refusals in selected sample elements:
• Problematic if related to research question. Example: especially rich people do not
respond to income questions.
• Problematic for external validity (e.g., does not pertain to rich people)
• How to detect? Check whether refusal rate is different across groups and whether
this is related to the process under study.
c) Variability of the characteristic under study in the population: large variability, larger
sample needed (increase n).
d) Confidence in the sample estimate: more certainty requires a larger sample size.
e) Organization of the fieldwork: balancing statistical efficiency (precision) with fieldwork
efficiency (and cost to collect data).

Fisher (1938) → To consult the statistician after an experiment is finished is often merely to ask
him to conduct a postmortem examination. He can perhaps say what the experiment died of.
Also true for sample size is that it doesn´t say afterwards “I should have had more participants”
and doesn’t ask afterwards “how can I get nice/good/significant results with this (too) small
number of participants?”.

21
 QUANTITATIVE RESEARCH METHODS

7.1. SOME TOO SIMPLE RULES OF THUMB

VanCoorhis and Morgan (2007)

Green (1991): Regresión N > 50 + 8m

m: número de VIs, que es igual a 5 y significa que hay 90 participantes
104 + m si queremos saber los predictores individuales = 109
In general, THE MORE PARTICIPANTS THE BETTER
But more participants mean more money and don’t mean more representative.

7.2. THE EXTREME MALE BRAIN THEORY OF AUTISM

Greenberg et al., 2018, PNAS develop a controversial theory with new data made with an
impressive amount of participants.
Our initial analyses were restricted to participants who indicated they were males or females
(672,279). Thus, we removed 22,887 participants who indicated "other" or "prefer not to say" when
asked to indicate their sex. Finally, we applied an age cutoff from 16 to 89 y old, to be consistent
with other research in the field (14) and removed participants who did not provide their age, leaving
671,606 participants for analysis. Of those who

In Spring 2017, Channel 4 TV developed a website for a documentary later entitled "Are you
autistic?" (15). As part of this website, users were able to take several scientific measures and find
out how their scores compared with the general population.
Participants were asked to provide demographic information and asked to click a checkbox
indicating that they would allow their results to be used for scientific research.
Only the results of those who checked the box were recorded for the dataset. The website

they had an autism diagnosis. In total, there were 36,648 autistic individuals (cases)
(18,188 males, 18,460 females). This equates to 5.45% of the sample, which is higher than the
population prevalence of autism (1%) (13, 46), possibly due to the nature of the TV.

For an experiment, there are also two factors: reality and 2 sorts of hypotheses.

22
 QUANTITATIVE RESEARCH METHODS

1. Null hypothesis. Typically states that there is no relationship between the response and
explanatory variable(s). It’s the hypothesis you want to nullify.
2. Alternative hypothesis. Typically states that there is a relationship between the response
and explanatory variable(s).

1-β is the power:

• = chance of detecting an effect, given that effect indeed exists
• High power is desirable

Null hypothesis in TRUE Null hypothesis in FALSE

(no effect) (effect)
Reject null-hypothesis Type I error Correct
You conclude: effect False positive True positive
α 1- β
Accept null-hypothesis Correct Type II error
You conclude: No effect True negative False negative
1- α β

The “α” is the significance level:

• You want it to be small, hence p < .05
• The probability of this mistake is less than 5%

8.1. COHEN ON THE RELATION BETWEEN ALPHA AND BETHA

In scientific research, it is typically more serious to make a false positive claim (Type I error) than
a false negative one (Type II error) because the implicit convention for significance is α = .05, the
use of the .80 convention for desired power (hence, β =.20) makes the Type II error 4 times as
likely as the Type I error, an arbitrary but reasonable reflection of their relative importance.

8.2. HOW CAN I GET THIS “HIGH” POWER?

Power is influenced by 3 factors (more or less) fixed: .08.

1. Sample size: this is the only factor you can influence.
– Simple rule = the more participants, the higher the power.
– Sample size and power are positively correlated.
– More participants mean more like population (in general, not always true).
2. Type I error: (more or less) fixed: .05.
– As α decreases, power decreases.
– We want α to be small, typically .05.
3. Effect size: (more or less) fixed: previous data or “wish”.
– = the degree to which the null hypothesis is false (Cohen, 1977).
– = the measure of the strength of the phenomenon.
– As the effect size increases in magnitude, power increases.
– We can know/calculate/guess/... this from previous studies.

23
 QUANTITATIVE RESEARCH METHODS

8.3. STEPS TOWARDS DECISION ABOUT SAMPLE SIZE

1. Decide the α (almost always .05).

2. Select the desired power = 1 – β (almost always power = .80).
3. Find/calculate/… the effect size.
4. Find the sample size from the combined information.

8.4. A FEW WORDS ON EFFECT SIZE (ES)

Sometimes one can read that the “p” gives you statistical significance and the ES gives you
“practical” significance, but it´s not really true. ES comes only “closer” to practical significance, less
influenced by variables like sample size, variance, etc.
Another “warning”: in and of itself, the effect size is not an indication of causality. When you obtain
a significant difference, you have to ask if there is a meaningful difference. For example, asking
how much more effective the new teaching method is compared to the old one. One does not
know, especially because researchers often work with measurements, they made themselves
(number of words, outcome on an achievement test, speed of reading, …). And it´s especially
difficult when one has two independent experiments:
• One showing new teaching method X is sign better than old teaching method.
• Other showing new teaching method Y is sign better than old teaching method.
Therefore, ES is helpful.
An effect size is a standardized, scale-free measure of the relative size of the effect of an
intervention/manipulation. It’s especially useful for quantifying effects measured on unfamiliar or
arbitrary scales and comparing the relative sizes of effects from different studies. It´s very
important for meta-analyse and there are basically, two types of effect sizes:
• ES which describes differences in means relative to the study's variability.
• ES which looks at how much of the variability can be attributed to the treatment
conditions.

24
QUANTITATIVE RESEARCH METHODS

25
 QUANTITATIVE RESEARCH METHODS

• Facebook influenced how happy people would feel.

– By either removing some very positive posts on people's timelines.
– Or by removing very negative posts on people's timelines.
• Measurement: comments people typed on Facebook and how positive or negative were.
• Result: when people saw less positive information, they actually were slightly less positive
in the Facebook messages that they post themselves.
• However, effect size was a Cohen's d of 0.001: very very small.
• Why significant? Because Facebook has a huge amount of data.
• Practical description of effect: after 3,570 words, one more negative word was typed in
the condition where people did not see the positive feedback.
• Not a typical Facebook-situation.

9.1. IS THERE AN EASY WAY TO INTERPRET ES?

Interpretation is not always straightforward:

• Cohen’s d and others have values which can be greater than 1.
• Therefore, some rule of thumbs

Again, these terms are arbitrary and only guidelines. We don’t use the numbers as cut-off values!
Micro Small Medium Large Very big Gigantic

Measures Difference Female vs.

Difference
between the Male height
Time of day between the Biological sex
M vs. F verbal heights of 13-
effects on heights of 15- Learning in affects gender
fluency and 18-year-
school test and 16-year-old school class vs. identity
old girls in the
girls in the USA 1:1 tutoring
Cases USA

Cohen´s d e.g. < 0.03 0.2-0.3 =0.5 0.8 or larger 2 12

R (corelation
0.1-0.3 0.3-0.5 o.5 or larger
coeff.)

R2: % of variance
1% 9% 25%
explained

R2: (==Partial eta-

0.02 0.13 0.26
squared)

Generalized eta- 0.26

0.02 0.13
squared η2p

Partial eta-
0.01 0.06 0.14
squared η2

26
 QUANTITATIVE RESEARCH METHODS

Understanding Statistical Power and Significance Testing — an Interactive Visualization

(rpsychologist.com)

9.2. SOFTWARE AND ES AND SAMPLE SIZE

• Once you know more about ES, you can calculate them “easily”.
• Or you use an only tool. E.g. https://www.uccs.edu/~lbecker/
• Or you use the labels to help you with determining the sample size.
• For sample size, there are also many online tools.

• So, there is a lot of software available!

• Good and free one: G*Power (Erdfelder, et al 1996) – general power analysis program can
compute sample size and power for t-tests,F-tests, chi-square tests
• http://www.gpower.hhu.de

27