Evaluating Research Articles: Experimental Designs

Sarah Hanson
In this assignment you will have the opportunity to read an experimental study and
practice your skills at evaluating its methodological features.
Directions:
Read the article entitled What Does Doodling Do? paying close attention to the
methods and procedures.
For each question below, rate the degree to which the information provided in the
research article satisfactorily addresses the question on a 5-point Likert-type scale. In
addition, justify each of your ratings by referring to specific elements mentioned or not
mentioned in the research article that formed the basis of your rating.
Scale:

1
3
5

Poorly addressed
Partially addressed
Extremely well addressed

1. Does the researcher describe the design and procedures used in sufficient detail to
allow for replication of the study, and for analysis of the credibility of the findings?
Rating:

Justification: The researcher provides the sample size in the study, how the
participants were chosen, the location of the study, what the participants listened to
and did during the study, and what materials were used (pencil, lined paper, doodling
for shapes). The researcher also discusses the performance measures used to gather
data for the group with and without the doodling variable. The analysis includes the
instrument used to enter scores for memory (ANOVA). However, it would be difficult to
create a scenario prior to the procedure as done in the study to enhance the boredom
before the study takes place. Also, the population sample is small but has a very large
age range from 18-55.

2. Does the researcher convince you that potential threats to external validity were
minimized, or at least were noted and discussed?(Note: mention at least one specific
potential threat to external validity, and describe how it was dealt with or not dealt with
by the author. You will need to consult supplementary material provided in Unit 2
Week 1, Step 1 Read and Prepare [ Supplement to Plano Clark & Creswell - Internal &
External Validity (Gall, Gall, & Borg, 2005 - Applying Educational Research) ]). Hint
Does this study seem generalizable and applicable to other individuals and settings,
given the makeup and size of the sample and other features of the study?

Rating:

Justification: This study would have to take place in a quiet, plain room in order to
replicate the same procedure and be able to compare results. There are so many
distractions in a real setting that it would be hard to determine whether or not doodling
helps concentration and limits daydreaming. The researcher discusses the actions
that would need to be taken to measure daydreaming in participants, but this was not
used in the study. The population used for this study is also limited. The sample
included people of a particular educational background since they were all pulled from
the University. I am not convinced that this study reduced potential threats to external
validity. When participants are shading in shapes and the researcher is trying showing
results of doodling helping with performance, there is quite a gap and leaves me with
many questions. The researcher herself discusses the difference between doodling
and what was actually done in the study, so perhaps the study should be called What
does coloring in shapes do since what the participants are doing is not the same as
what is being studied. Yes, this rules out the potential for inaccurate results if people
are feeling self-conscious about their doodling instead of paying attention to the
prompt, but the variable is not doodling and thus does not support the hypothesis.

3. Does the researcher convince you that the groups being compared were equivalent
prior to the onset of the treatment, either by describing an appropriate method of
random assignment, matching, and or statistical evidence that the groups were the
same at the onset of the experimental treatment?
Rating:

Justification: Yes, I am convinced that the groups being compared were equivalent
prior to the onset of the treatment. The participants were all members of the MRC
Applied Psychology Unit and were randomly assigned to each group. The same
amount of participants were assigned to both groups with an age range provided for
the whole of the participants and the number of males listed in the control and doodle
group. There are not any red flags in this study that lead me to believe that the
participants in the groups were not equivalent.

4. Does the researcher convince you that all other aspects of the conditions were the
same in the experimental and control groups, except for differences in the treatments?
(e.g., was the experimental treatment more motivating for the participants? If so, the
conditions may not be the same).

Rating:

Justification: The conditions were the same for each group, but the experimental
treatment may have been more motivating for participants. The control group and the
doodling group were given the same prompt in a plain room. However, the doodling
group was given a printed piece of paper with squares and circles to shade in while the
control group was only given plain paper to write their answers down on. The
researcher would have been more convincing if the control group was given the
opportunity to doodle but was not instructed to do so or was given a concurrent task
other than doodling since the measure of daydreaming was not measured in the study.

5. What is your overall assessment of the study with respect to the criteria below?

Are the results at all surprising to you? That is, if you had not known the outcome

of the study, would you have been able to correctly predict the outcome?
Did you learn something new from reading the study?
Is there anything you learned from reading the study that you will apply to your
life or someone elses life?
The results of this study were not surprising to me. The doodling group had to
stay more engaged because of the extra condition to doodle, thus more of a
reason to record the names from the prompt. There was not another task
assigned to the control group. I learned the importance of external validity and
how it can effect a study and the ability to reproduce the study. It seems that this
researcher has quite a few extraneous variables that could be weighing in on the
research. For one, daydreaming was not measured, but the researcher
discussed the importance of measuring this. Also, participants were removed if
they were not doodling but were assigned to the doodling group. This
experimental mortality could have skewed the results greatly because some
people may perform better without doodling. This study seemed muddled to me
and I was left unclear with the purpose of it all. The method seemed clear
enough, but I am not convinced that this research can support doodling while
working as beneficial. I want to make sure that when I am looking at data or
collecting data from my students, I am aware of the purpose of collecting the data
and any internal or external factors that could be effecting the study.