Copy of Level 2 Paired Experiments Workbook 2024.Docx
Copy of Level 2 Paired Experiments Workbook 2024.Docx
Experiments
Workbook
Name:
By Liz Sneddon
Causation
If an experiment is well-designed and well conducted study, then causation, or
cause-and-effect claims can be made. However, if it is an observational study, then
causation cannot be claimed.
Page 2
Exercise 1:
7) Is it ethical to do this
experiment? Why/why not?
Page 3
Note about Ethics
In the past, many experiments were conducted that took advantage of vulnerable
people and animals. Most countries now have very complex systems in place to
ensure that if someone wants to carry out an experiment using animals or humans,
there are committees that approve all experiments, to ensure that we can’t take
advantage of others and do harm to them. For example, Hitler is well known for the
many experiments he conducted using identical twins.
Here are some things we need to think about when designing an experiment, and the
ethical considerations.
Page 4
Experimental studies
Our first step is to identify whether the study is an observational study or an
experimental study. The key is whether participants have been randomly
allocated into the groups.
Page 5
Example:
For each of the situations below, identify if an experiment or observational study is
the most appropriate.
1) Study Study which compares pulse rates of students with and without
exercise
Page 6
Exercise 2:
Decide if each of the following studies are likely to be observational or experimental.
Justify your decision.
1) Study Study that compares performance in an IQ test with and
without caffeinated drinks.
Ethical
issues to
consider
Practical
issues to
consider
Justify your
decision.
Page 7
2) Study Impact of smoking during pregnancy on babies’ health.
Ethical
issues to
consider
Practical
issues to
consider
Justify your
decision.
Page 8
3) Study Study of whether food tastes better if it has a brand.
Ethical
issues to
consider
Practical
issues to
consider
Justify your
decision.
Page 9
4) Study Study on driving skills for sleep deprivation.
Ethical
issues to
consider
Practical
issues to
consider
Justify your
decision.
Page 10
Variables and Participants
There are two variables we need to identify, the treatment variable and the response
variable.
We also need to identify the participants (or experimental units as they are
sometimes called).
Page 11
Example 1:
Work out which is the explanatory variable and response variable for the following
numeric variables:
● Amount of water a plant is given,
● The height of the plant.
The first option is correct – we would control or change how much water a plant is
given, and then we can measure the height.
This means:
Explanatory variable = amount of water given to a plant
Response variable = height of the plant.
Example 2:
Identify the response and treatment variables for each of the experiments described.
(Think about what data you can collect).
Control group Dominant hands (as this is the hand they normally
write with).
Page 12
Exercise 3:
Identify the response and treatment variables for each of the experiments described.
(Think about what data you can collect).
Response variable
Treatment variable
Control group
Treatment group
Participants
Response variable
Treatment variable
Control group
Treatment group
Participants
Page 13
3) Study Study of whether food tastes better if it has a brand
using primary school students in NZ.
Response variable
Treatment variable
Control group
Treatment group
Participants
Response variable
Treatment variable
Control group
Treatment group
Participants
Page 14
Problem / Kaupapa
When writing an investigation, you need to remember to include:
● Treatment variable,
● Response variable,
● The word “effects”,
● The participants.
Example:
We want to do an experiment to study whether students at our high school can write
faster with their dominant or non-dominant hand.
Investigation I wonder if writing with your dominant hand has an effect on the
question length of time (seconds) it takes to write a sentence compared to
writing with your non-dominant hand, for students in our Year 12
Statistics class.
Page 15
Exercise 4:
Write an investigation question for each of the situations below.
Investigation
question
Investigation
question
Investigation
question
Investigation
question
Page 16
Plan / Whakamahere
Experimental design
There are two methods that can be used to design experiments.
Random allocation
As all participants are going to carry out both treatments (treatment and control
groups), where possible we are going to randomly allocate the order in which the
participants do the treatments (E.g., whether they do the treatment or control first).
Page 17
Tools
To randomly allocate the order, we want to think about what tools could be used to
ensure the choice is random. Here are some suggestions:
● Tossing a coin,
o if it is heads, allocate to control/treatment first,
o if it is tails, allocate to the treatment/control first.
● Rolling a die,
o if it is a 1, 2, or 3, allocate to control/treatment first,
o if it is a 4, 5, or 6, allocate to the treatment/control first.
● A pack of cards,
o if the card is a red suit, allocate to control/treatment first,
o it the card is a black suit, allocate to the treatment/control first.
● Websites that have a list randomizer, such as www.random.org/lists/. when you
input a list of names it will randomly allocate into 2 groups:
o If the students name is in list 1, they do the control/treatment first.
o If the students name is in list 2, then they do the treatment/control first.
It is useful to have similar numbers of students in each group (treatment and control
groups). One way to do this is to:
● have the same number of cards/pieces of paper which match the total number
of participants,
● have half of the cards in red, and half the cards in black, or have half the pieces
of paper with one group written on it, and the other half of the pieces of paper
with the second group written on it,
● and then each participant will get one card/piece of paper.
Treatment Control
Treatment Control
etc.
Page 18
Example:
Identify the participants, control and treatment groups and then describe the random
allocation process.
Control group Dominant hand (this is the hand students normally write
with).
Page 19
Exercise 5:
Identify the response and treatment variables for each of the experiments described.
(Think about what data you can collect). Then describe the random allocation process
you would use.
Participants
Control group
Treatment group
Random allocation
Page 20
2) Study Study that compares performance in an IQ test of NZ
University students with and without caffeinated
drinks.
Participants
Control group
Treatment group
Random allocation
Page 21
3) Study Study of whether food tastes better if it has a brand
using primary school students in NZ.
Participants
Control group
Treatment group
Random allocation
Page 22
4) Study Study on driving skills for sleep deprived on driving (in
a simulator) for University students in NZ.
Participants
Control group
Treatment group
Random allocation
Page 23
Design principles
There are a number of ideas and concepts
that we need to take into consideration
when designing an experiment. Some of
these may not be able to be used in
different situations, but they should all be
considered where appropriate. We want
the data (response variable) to be
measured as accurately as possible, so
that the results will be reliable and
precise.
Page 24
Example:
Study Study of whether students can write faster with their dominant or
non-dominant hand.
Page 25
Exercise 6:
Identify and describe what experimental design principles can be controlled for in the
following experiments.
Conditions
kept
constant
Repeated
measures
Page 26
2) Study Study that compares performance in an IQ test of NZ
University students with and without caffeinated drinks.
Conditions
kept
constant
Repeated
measures
Page 27
3) Study Study of whether food tastes better if it has a brand using
primary school students in NZ.
Conditions
kept
constant
Repeated
measures
Page 28
4) Study Study on driving skills for sleep deprived on driving (in a
simulator) for University students in NZ.
Conditions
kept
constant
Repeated
measures
Page 29
Instructions
Once you have identified what conditions
need to be managed and how they can be
managed, your next step is to write a set
of instructions.
Example
Investigation I wonder if writing with your dominant
question hand has an effect on the length of time (seconds) it takes to write
a sentence compared to writing with your non-dominant hand, for
students in our Year 12 Statistics class.
Instructions 1) I will find the number of students present, dividing this in
half. This tells me how many students will do the test with
their dominant hand first, and how many will do it second.
2) I will have the same number pieces of paper as the number of
participants, with the words Dominant or Non-dominant (this
is the hand they will use first).
3) Each student will be randomly given one piece of paper. There
are equal or nearly equal numbers in both groups.
4) On the whiteboard a stopwatch will be projected.
5) Students will be asked to get out a pen and move to make
sure that they can see the whiteboard.
6) Students will be given their instructions. As soon as the
sentence is displayed on the whiteboard, the stopwatch will
start, and students will copy down the sentence onto a piece
of paper. When they have finished, they will look at the time
on the stopwatch and record the time on their paper.
7) Students will then switch their pen to the other hand and
repeat the test, recording the results on their paper.
8) Papers will then be collected, and the data will be recorded on
an Excel spreadsheet as follows:
… …
Page 30
Exercise 7:
Write instructions for the study below. Consider the conditions you identified in the
previous exercise that need to be controlled.
Instructions
Page 31
Data / Raraunga
Once the data has been collected (which you will be doing in the assessment in
groups of 3-4 students), the next step is to copy the data from your spreadsheet into
NZGrapher to make displays from.
You will need to enter your data into a spreadsheet first (either Excel or Google
Sheets), including the headings (names of the variables).
Set up the spreadsheet with one column for the data from the control group, and one
column for data from the treatment group, and each row represents another student.
Example:
Here are two variables from the Fitness dataset:
Page 32
Steps for entering data into NZGrapher
Step 1: Step 2:
The quickest way to get data into Next, go to the NZGrapher website.
NZGrapher is to select all of the data in
your spreadsheet and copy this.
Step 3: Step 4:
Click on the Data menu and select NZGrapher will ask you to give your
“Import from Clipboard”. permission to copy information from
the clipboard.
Click “Allow”.
Page 33
Graphs
There are two types of graphs that we need to draw – an Arrow graph, and a dot
plot/box plot of the differences. The reason we are drawing these groups is that the
two groups we are comparing (control and treatment group) are NOT Independent.
This is why we are interested in how much the data has changed by.
Arrow graph
An arrow graph is where we draw a dot plot using the data from both the treatment
and control group and draw an arrow between the matching points. Each arrow
represents one person, and it connects each of the two tests they do (treatment and
control). When we interpret the graph (which we will explore in the Analysis section),
we are interested in the direction of the arrows.
Example
Each arrow
represents a
person, and their
writing time for
their dominant and
non-dominant
hand.
Page 34
Exercise 8:
Draw an arrow graph of the following dataset. This experiment compared how many
times students could hop on one leg in a minute, comparing using their dominant leg
with their non-dominant leg.
1 24 22
2 32 21
3 27 18
4 28 20
5 25 23
6 14 24
7 33 28
8 30 27
9 18 19
10 25 25
First draw the dots for each leg, and then draw an arrow connecting them together.
Page 35
Drawing an arrow graph
Step 1: Step 2:
Go to the bottom of the screen, scroll Select the variables for your graph. Put
down to nearly the bottom of the list, the control group as the first variable, and
and click on the Graph Type menu, the treatment group as the second
and select Paired Experiment Dot variable.
Plot (and Arrows graph).
Step 3: Step 4:
Go to the options section in the bottom Add axes labels, units, and a title to the
middle of the page. Select Arrows, and graph.
this will form the arrow graph.
Also, click on the Summaries option. Once you have entered the labels and
title, then click the Update Graph button
(it’s on the right-hand side at the bottom
of the page).
Page 36
Dot plot of differences
A dot plot of the differences is where we calculate the difference between the
treatment and control group, and then draw a dot plot of these differences. Again, we
are interested in how much the data changes between the two groups.
Example:
Page 37
Exercise 9:
Draw a dot plot of the differences for the following dataset. This experiment
compared how many times students could hop on one leg in a minute, comparing
using their dominant leg with their non-dominant leg.
First you need to calculate the difference between the number of hops.
Hops on Hops on
Difference
Student dominant leg non-dominant leg (Dominant – Non-dominant)
(Control group) (Treatment group)
1 24 22
2 32 21
3 27 18
4 28 20
5 25 23
6 14 24
7 33 28
8 30 27
9 18 19
10 25 25
Page 38
Exercise 10:
Open this link: bit.ly/HandExperiment
Copy the data into NZGrapher and form both an arrows graph and a dot plot of
differences. Include the axes labels and units, and title for the graph.
Observations
During the experiment, the observer (and others) may
notice behaviour or events that could affect the results of
the experiment. It is important that these are noticed,
recorded, and discussed. Improvements for future
experiments may also be noted (for Excellence).
Example:
I noticed that some students were talking while writing the passage. This
means that they may have distracted themselves and the person they were
talking to. This may have led to them taking longer to write the passage.
If I were to do this experiment again, I would give instructions to students to
not talk during the experiment, so that all students have the same conditions,
and therefore the results are likely to be only due to the difference in
treatment, not due to other factors affecting the results.
Page 39
Analysis / Tātari
We need to analyse both the arrow graph and the dot plot of the differences.
Arrow graph
We’ll start with the arrow graph. We need to start by identifying what each arrow
means and how to interpret the arrows.
Example:
Investigation I wonder if writing with your dominant hand has an effect on the
question length of time (seconds) it takes to write a sentence compared to
writing with your non-dominant hand, for students in our Year 12
Statistics class.
Arrow graph
What does each dot Each dot represents the time that students wrote a
represent? sentence eitehr with their dominant hand or their
non-dominant hand.
What does each arrow Each arrow represents one person and how much
represent? more (or less) time they take to they write with their
non-dominant hand compared to their dominant hand.
What does an arrow going An arrow going in a positive direction shows that a
in a positive direction student takes more time to write with their
mean? non-dominant hand than their dominant hand.
Page 40
What does an arrow going An arrow going in a negative direction shows that a
in a negative direction student takes more time to write with their dominant
mean? hand than their non-dominant hand.
What does it mean if lots of It means that some students take longer to write with
arrows are going in both their dominant hand, and some students take more
directions? time to write with their non-dominant hand.
When we interpret the arrow patterns, we want to look at how many arrows are going
in each direction. There are summary statistics on the top right-hand side of the
graph that tells you how many arrows go in the positive direction, stay the same, or
in the negative direction.
Example:
Arrow graph
Interpreting From the arrow graph above we can see that all the arrows are
the arrow pointing in the same direction.
patterns
This suggests that all students took longer to write a sentence with
their non-dominant hand than their dominant hand.
Page 41
Exercise 11:
Interpret the following arrow graphs.
Arrow graph
Interpreting
the arrow
patterns
Page 42
2) Study Study that compares performance in an IQ test of NZ
University students with and without caffeinated drinks.
Arrow graph
Interpreting
the arrow
patterns
Page 43
3) Study Study of whether food tastes better if it has a brand using
primary school students in NZ.
Arrow graph
Interpreting
the arrow
patterns
Page 44
4) Study Study on driving skills for sleep deprived on driving (in a
simulator) for University students in NZ.
Arrow graph
Interpreting
the arrow
patterns
Page 45
Dot plot of the differences
Next we’ll interpret the dot plot of the differences. NZGrapher calculates the
differences for each individual. Check the axis label below the graph to know which
way around the variables have been subtracted.
Example:
Investigation I wonder if writing with your dominant hand has an effect on the
question length of time (seconds) it takes to write a sentence compared to
writing with your non-dominant hand, for students in our Year 12
Statistics class.
What does each dot Each dot represents one person and how much more
represent? (or less) time they take to they write with their
non-dominant hand compared to their dominant hand.
What does a positive data A positive data point shows that a student takes more
point mean? time to write with their non-dominant hand than their
dominant hand.
What does a negative data A negative data point shows that a student takes more
point mean? time to write with their dominant hand than their
non-dominant hand.
Page 46
When we analyse the dot plot,
we want to look at the shape,
centre and spread.
Example:
Dot plot (and
box plot) of
the
differences
Interpreting Shape:
the arrow The shape of the differences between the time it takes students to
patterns write with their dominant and non-dominant hand, is right skewed,
because there is a long tail on the right-hand side, and the data on
the left-hand side is closely packed.
Centre:
I notice that the median difference between the time it takes
students to write with their dominant and non-dominant hand is 56
seconds. Because this difference is positive, this shows that it takes
students a median of around 56 seconds longer to write a sentence
with their non-dominant hand, compared to their dominant hand.
Spread:
IQR (Difference) = 110.5 – 43 = 67.5 seconds
The spread of the middle 50% of differences between the time it takes
students to write with their dominant hand and non-dominant hand has a
spread in the middle 50% of 67.5 seconds. That shows that there is quite a
wide spread in the time differences.
Outliers:
There does not appear to be any outliers.
Page 47
Exercise 12:
Interpret the following graphs.
Interpreting
the features
Page 48
2) Study Study that compares performance in an IQ test of NZ
University students with and without caffeinated drinks.
Interpreting
the features
Page 49
3) Study Study of whether food tastes better if it has a brand using
primary school students in NZ.
Interpreting
the features
Page 50
4) Study Study on driving skills for sleep deprived on driving (in a
simulator) for University students in NZ.
Interpreting
the features
Page 51
Conclusion / Whakatau
In your conclusion you need the following:
● To answer your original question (I wonder …).
● Justify your decision.
● Describe the decision in context, stating specifically who these results
can be applied to.
● Discuss improvements or other investigations that this might lead into.
Page 52
Answering your investigation question
Interpreting the results is different from what we did in Inference. Now the key
idea is to think about:
However, if I say my graph had around 75% (for a sample size of 20) of the
arrows going in one direction (or dots on a dot plot being positive or negative),
Page 53
I can suggest that the treatment causes the difference, for my
participants.
You could equally use this same idea looking at the dot plot, and seeing if the
whole box sits above or below 0. (Remember that the Lower Quartile has 75%
of the data above it, and the Upper Quartile has 75% below it.)
OR
Note: if your sample size is smaller, you need a percentage larger than 75%.
The evidence we are looking for is:
● is 75% (or more) of my arrows going in the same direction,
OR
● is 75% (or more) of my differences positive/negative (from the dot plot).
There are two conditions we need to meet to be able to suggest that one variable
affects the other:
1) The experiment has been well designed, and
2) The difference in medians is large enough (75% of the data in one group
bigger than at least 50% of the data in the other group).
Page 54
Example:
Conclusion:
I have enough evidence that writing with your dominant hand has an effect
on the time it takes to write a passage compared with using your non-dominant
hand, for students in my Year 12 Statistics class.
My evidence is that 100% of the differences are positive. This is more than the
75% required to have enough evidence to make the call.
Since I randomly allocated students to write with their dominant or
non-dominant hand first, this supports my suggestion of an effect, as the
writing times should have been pretty similar if using a dominant hand had no
effect.
Improvements:
I could improve my experiment by getting the students to write the paragraph
5 times, and then finding the average of these measurements. This would
improve the accuracy of the results.
Limitations:
My experiment was done using students at my school. This means that the
results are only applicable to students at my school who are taking Year 12
Statistics-. We cannot assume that these results can be applied to any other
students who are younger than sixteen or adults over the age of eighteen. In
order to show that younger students can write faster with their dominant hand
for example, we would need to run the experiment again with a group of
younger students.
Other investigations:
If I ran the experiment again, I could make a small change in the design. For
example, I could use a different passage of text, or use something written in a
different language (not English).
Page 55
Exercise 14:
Interpret the following graphs.
Arrow graph
Answer the
investigation
question
Page 56
2) Investigation I wonder if having a caffeinated drink affects the performance
Question in an IQ test compared with not having a caffeinated drink, for
NZ University students.
Arrow graph
Answer the
investigation
question
Page 57
3) Investigation I wonder if food gets a better rating if the brand can be seen
Question compared to not seeing the brand for primary school students
in NZ.
Arrow graph
Answer the
investigation
question
Page 58
4) Investigation I wonder if being sleep deprived has an effect on driving skills
Question (in a simulator) compared to not being sleep deprived, for
University students in NZ.
Arrow graph
Answer the
investigation
question
Page 59
Page 60