FIELD METHODS IN PSYCHOLOGY 311

REVIEWER|BS PSYCH|FINALS
QUANTITATIVE RESEARCH: MEASUREMENT AND DATA ❖ Interval Data — a kind of variable that is similar
COLLECTION to ordinal data in that both reflect increases in
Measurement: Turning Abstractions into Variables quantity; data, however, the quantity between
❖ Variable — a construct that can take on two or the different responses of the variable is the
more distinct values. same (e.g. grade point ave.)

❖ Data — a collection of variables from a sample o Likert Scale - a type of response alternative in
make up what researchers which participants indicate their degree of
agreement with a stated attitude or judgment.
❖ Operational Definition o Scatterplot – displays interval data; graphically
represent the relationship between two
o defines how a variable will be measured or variables
assessed
o used to capture the meaning of the abstract ❖ Ratio Data — a kind of variable that have an
concept they are trying to measure absolute zero; an absolute lower value that
o may be straightforward (e.g. gender, college corresponds to the absence of the measure;
major) or difficult to define (e.g. aggression) allow numerical values to be placed in ratios

❖ Nominal Data — a kind of variable that divide

responses into two or more distinct categories;
independent variables are often nominal data

o frequency distribution - common way to display

nominal data; shows how often the different
possible values of the variable were selected
o Cross-tabulation - method to show how the
responses of one nominal variable relate to
the responses of another nominal

Characteristics of Good Measures

❖ Truthful (valid)— must accurately reflect the
construct
❖ Ordinal Data — a kind of variable that makes a
❖ Consistent (reliable) — must yield the same
further distinction by the quantity of response
results across time, circumstances, and groups
alternatives; show numerical differences
of people.
between response alternatives
❖ Validity — actually measures what it claims (or is
o often summarized in the same fashion as
intended) to measure
nominal data
o ordinal refers to the ordering of responses o Content Validity – the extent to which the items
or behaviors fully represent the concept being
measured.
o Construct Validity – the extent to which the
measure is on target to measure the construct
being studied.
▪ Convergent Validity - the extent to which
other measures of the same behavior are
similar to your measure.
▪ Discriminant Validity - when the instrument
being examined is uncorrelated with
another measure that is presumably
unrelated
1
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
o Predictive Validity – the extent to which a the fields of organizational behavior and
measure is related to some other measure educational studies
that you would be interested in predicting
o Normative measure - person’s performance
❖ Reliability — extent to which a measure yields the relative to the performance of others.
same scores across different times groups of
people or versions of the instrument. o Criterion measure - how much a student has
learned in a particular subject.
o Cronbach’s Alpha – most common way to
assess the reliability of self-report items; o Achievement measure - the amount of material
measures the degree to which the items in an a person has mastered.
instrument are related.
o Aptitude measure - a person’s potential for
▪ has maximum value of 1.0; values closer to
success in a given area
1.0 reflect a stronger relationship between
the test items.
o Test-retest reliability – measures the similarity
of participants ’ scores at two different times;
the greater the similarity between the two sets
of scores, the higher the test-retest reliability;
often used for measures of achievement and
other types of performance
o Parallel-form reliability - similar but not
identical, versions of the same instrument
have the same measurement characteristics;
solves the problems associated with assessing QUANTITATIVE RESEARCH: DESCRIPTIVE AND
test-retest reliability; often used when you are CORRELATIONAL DESIGNS
trying to determine if a measure changes over Basic Concepts
time ❖ Correlational Research — identifying
relationships between two variables
o Inter-rater reliability - often used for behavioral
▪ Correlation – a statistical measure between
observations; high inter-rater reliability if two
two variables
people who are observing a behavior agree on
the nature of that behavior. o Positive correlation — when, as one variable
increases (or decreases), the other variable
Collecting Data also increases (or decreases)
❖ Self-Report — a technique that social scientists o Negative correlation - when, as the value of
commonly use; collected by asking participants one variable increases, the value of the other
to answer questions on their own by survey or variable decreases (and vice versa).
questionnaire
▪ Self-serving bias – disadvantage where
there's a tendency to report their behaviors
and attitudes in a positive light.

❖ Behavioral Observations — come from the

researchers rather than the participants; provide
a third-party account of people’s activities
▪ Self-presentation bias – when participants
in the study know they are being observed
and behave differently because of it.
o Mutually exclusive - each behavior can be Causation and Prediction
classified into only one category ❖ Prediction — if a researcher knows a person’s
o Mutually exhaustive - every behavior gets score on one measure or variable, the researcher
classified. will have a good guess about what the person’s
score is on another measure
❖ Performance measures — are similar to
physiological measures in that they contain an ❖ Causation — a change in one variable
actual behavior record; also similar to self-report (independent variable) creates a change in
measures in that the participant is the one another variable (dependent variable); three
providing the record of the data; often used in components to infer causation:

2
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
▪ covariation – as one variable changes,
another variable must change.
▪ time order - cause must precede effect.
▪ other plausible causal variables must be
eliminated.

Data Gathering
❖ Surveys — the best way to collect a large amount
of data from a large number of people in a short
amount of time; very robust and can be used in
many life domains.

o Constructing Survey Items — You Only Get One

Chance (crucial to pay attention to detail
before you send out the survey):

▪ Have a symmetric set of response

alternatives (choices of answers on a
survey from which respondents select)

▪ Avoid double-barreled questions (a survey

item that has two distinct components in
the stem of the question)

▪ Make the stem of the question unbiased

o Anchors - two endpoints of a response scale

where descriptors are attached
o Telephone Surveys – very common, higher
participation rate than mail
▪ directory-listed sampling (randomly selects
households with listed telephone numbers)
▪ random-digit-dialing (actual production of ❖ Observations — collect actual occurrences of
these random numbers is a daunting task; behavior
because non-listed numbers are included,
the sample will be more representative; ▪ Time sampling - involves recording behavior
more expensive) by taking systematic observations at preset
o Mail Surveys – paper-and-pencil surveys intervals.
delivered through surface mail; effective for ▪ Event Sampling – involves recording all or a
reaching a large group or geographical area; proportion of the specific instances of the
behavior of interest.
minimal costs for survey production, major
▪ Rating Scale - an observational record in
financial cost in postage which the observer records his or her
▪ Cover letter that’s only one page long judgment about the nature of the
▪ Put a return date in bold aggressive act.
▪ Send a reminder email or postcard
▪ Don’t make your survey too long o Participant Observation - the researcher
▪ Look professional enmeshes himself or herself in the community
▪ Consider putting surveys on heavier-than- under study; tend to be more reliable
usual paper o Nonparticipant Observation - the researchers
▪ Timing is everything (sending surveys out observe a group of people from outside of the
between the middle of November and early group and thus do not embed themselves in
January should be avoided it is the busiest the community under study while they conduct
time) their investigations.
o Web/Email Surveys – extremely popular;
financial savings are compelling, as there are
Data Analysis
no printing or postage costs.
❖ Descriptive Statistics: Central Tendency
o Face-to-face Interviews - to speak with people
in person; allow a wider channel of
o Mean - most common measure of central
communication. tendency; arithmetic average of a set of
3
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
numbers; provide information about the ❖ Pearson correlation - measures the relationship
average participant’s score on a measure. between two interval variables; most social
o Median – middle score of numbers in a scientists use it for ordinal data
dataset. ▪ range is from – 1.0 to +1.0
o Mode - most frequently occurring number in a ▪ the magnitude of a correlation increases as
dataset; the absolute value of the correlation
increases
❖ Descriptive Statistics: Spread - measures how ▪ the closer a correlation is to +1.0 or – 1.0,
much variability there is among the observed the greater its magnitude
scores on a variable
o correlation matrix - set of correlations among
o Range - a straightforward measure of spread; variables in a study
the lowest score in the dataset is subtracted ❖ Chi - Square - a measure of association/
from the highest score. relationship between two nominal variables.
o Variance – more complex; the sum of the ▪ If a chi-square is statistically significant, it
squared difference between each observation means that there is a relationship between
and the mean value, divided by one less than the two nominal variables
the number of observations
o Standard Deviation – more complex; the Example:
square root of the variance; the average
amount any one observation will differ from
the mean.

❖ Inferential Statistics (to draw conclusions about

significant relationships between variables):
Understanding Statistical Significance

o level of significance - the probability that a

relationship between variables is not real, but
rather due to chance factors.
o statistically significant– uses .05 as the level
of significance and deem it significant; cant if
the level of significance surpasses a threshold
such that researchers are willing to conclude
that the finding is a “ real ” relationship rather
than an artifact of chance factors.
▪ p value – likelihood that the difference
between two variables that were found in
the current study would be this large or QUANTITATIVE RESEARCH: BASIC EXPERIMENTAL
larger if you assumed that there was no DESIGNS
difference; the larger the difference, the ❖ Causal inference - to make statements about
smaller the p value, and the smaller the which variable is the cause and which variable is
chance that the researchers are wrong to the effect.
claim that the difference is real.
▪ Type I error - idea of researchers “ being Experimental Validity
wrong ”; believes that a relationship is true ❖ Internal validity - extent to which the claim of
when in fact it is due only to chance factors; changes in the independent variable causing
false positive changes in the dependent variable is accurate
▪ Type II error – opposite of Type I error;
believing there is not a difference between
❖ External validity - extent to which the findings
two variables when there really is one; false
from one investigation will generalize to other
lelenegative
samples, populations, or settings.

Types of Variables
❖ Independent variable – systematically controlled
by the researcher to determine the variables
effect on the outcome

o Factor – used in experimental research as a

term for independent variable

4
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
o Level – distinct amount of independent ▪ treatment group - the group that received
variable/factors the independent variable
o Placebo - nonactive or zero - level version of an
independent variable, often used in drug trials o Simple Random Assignment - randomly
o Individual - difference variable - measure of assigning participants to the different levels of
some inherent trait, disposition, or personality the independent variable.
difference; can be numeric or categorical; e.g. ▪ Random assignment - all participants have
gender, income, and SAT score an equal opportunity to be placed in any of
the conditions of the experiment.
❖ Dependent variable – the outcome measure in ▪ Law of large numbers - as the size of a
which researchers are interested; aka criterion sample increases, the more likely it is that
variable; measured by the observation, test, or the sample will approximate the overall
survey population.
o Stratified Random Assignment - an even more
▪ clear operational definitions of this variable precise way to assign participants to
are valuable because other researchers can experimental groups; people are assigned to
replicate your research conditions based on a preexisting trait.
▪ another important component of ▪ For example: If the researcher wants to
dependent variable measurement is guarantee equal proportion of genders in
consistency in data gathering. each condition

❖ Extraneous variable – a rival explanatory variable o Matching - involves making the experimental
that could also explain the relationship between groups as similar as possible on a potential
the independent and dependent variables rival variable

o Confound - a situation in which it is not known o Balancing Unwanted Variables - equal

whether changes in the dependent variable distribution of nuisance variables makes it less
were caused by the independent variable or by likely that one experimental group will be
an extraneous variable. disproportionately affected by such a variable
o Placebo effect - which arises when participants Types of Experimental Designs
behave differently because they believe that ❖ Random Groups Design – participants are
the independent variable is having an effect randomly assigned to one of the experimental
rather than because the independent variable conditions; most straightforward way
is actually having an effect
❖ Matched - Groups Design - participants are
Characteristics of Experiments placed into groups based on some preexisting
❖ Covariance – inherent in a claim about causality; characteristic
occurs when scores on two variables change at ▪ characteristic should be a variable that you
the same time. as a researcher believe is correlated with
the dependent variable.
❖ Time Order - cause comes before effect;
▪ experiments allow time order to be ❖ Natural – Groups Design - some difference
established because the researcher already present among the participants is used
controls the independent variable and in assigning the groups
purposely puts it before the dependent ▪ researchers use an individual-difference
variable to establish the desired time order variable to make up the levels of the
independent variable e..g. (gender,
❖ Elimination of Rival Hypotheses - causality is race/ethnicity)
difficult to determine in the presence of rival
variables that could also explain the relationship Data Analysis
▪ experiments are able to eliminate rival ❖ t Tests – used to determine if two groups or
variables because researchers are able to levels of an independent variable differ on a
exert control over the experimental dependent variable
situation.
o independent - samples t test - compares mean
o Presence of a Control Group - a group of scores from a study in which each participant
participants in an experiment who receive receives only one level of the independent
either no exposure to the independent variable variable.
or the same exposure that would otherwise
occur in everyday life ▪ If a t test is found to be statistically
significant, we would say that the two
groups differ on the dependent variable.
5
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
❖ Analysis of Variance (ANOVA) - used to determine ❖ Practice Effects – occur when participants ’
if three or more groups or levels of an performance in an experiment changes simply
independent variable differ on a dependent because they have done the experimental task
variable; logical extension of t test multiple times, rather than because of the
experimental manipulation.
▪ If the p value is less than .05, researchers ▪ main disadvantage of repeated-measures
conclude that the ANOVA is statistically
significant and therefore the three (or more) ❖ Counterbalancing - is a method of alternating the
groups differ from each other. order of delivery of the independent variable to
reduce practice effects.
o post - hoc tests - statistical tests performed on
statistically significant ANOVAs to determine o ABBA Design - when one order of the levels of
how the means differ from each other. the independent variable is presented for one
participant, then for the next participant the
QUANTITATIVE RESEARCH: ADVANCED EXPERIMENTAL opposite order is used.
DESIGNS ▪ problem with this approach is each level of
❖ Between-Groups Design - an experiment in which the repeated-measures independent
participants receive only one level of an variable appears in each position.
independent variable
o Latin Square Design - each level of the
❖ Repeated–measures designs - participants in the repeated - measures independent variable
experiment receive all levels of the independent appears in each position.
variable rather than just one level ▪ solves previous approach’s problem by
having each level of the independent
❖ Complex Design - experiments with more than variable appear in each order.
one independent variable; more complex if
conducted outside of a standard laboratory o all -possible - orders counterbalancing - each
setting. level of the independent variable appears in
each position and each level precedes and
Basic Concepts follows every other level equally often.
❖ Main Effects - is a test of whether an ▪ an independent variable with n levels will
independent variable is statistically significant require n! (“n factorial”) orders.
across all of the levels of the other independent
variable(s) Complex Designs
❖ Complex Designs with Between-Groups Factors -
❖ Interactions - the effect of one independent can be extended to any number of independent
variable at a particular level of another variables and any number of levels of those
independent variable in the same study independent variables
o simple main effect - analysis of one o two between - groups factors - each
independent variable at a particular level of participant in the experiment receives
the other independent variable. only one level of each of the
Repeated-Measures Designs independent variables.
▪ would need only one set of participants, o completely - crossed design - which all
who would experience all levels of the the levels of one independent variable
independent variable. are paired with all the levels of the other
independent variable,
❖ Advantages of Repeated-Measures
❖ Mixed Design - a complex experimental design
o Replication - a set of completed data for all
that contains at least one between-groups factor
the levels of the independent variable(s).
and at least one repeated-measures factor.
o Error variance - variation in the scores of the
▪ for the between-groups factor, participants
dependent variable that cannot be accounted
are randomly assigned to the different
for by the independent variable.
levels of that factor.
▪ Within-group error variance - due to random ▪ then assign the levels of the repeated-
fluctuations in the performance of one measures factor within each level of the
group of people due to characteristics of between-groups factor.
the people in the study. ▪ sometimes the between-groups factor is a
natural - groups variable.
▪ Between-group error variance - due to
differences between the groups in the ❖ Solomon Four - Group Design- a special case of
experiment. the mixed design, used when the presence of a
pretest might create a practice effect
6
FIELD METHODS IN PSYCHOLOGY 311
REVIEWER|BS PSYCH|FINALS
▪ considered a quasi-experimental design variable in a repeated-measures design has more
▪ involves four groups than two levels
o paired t test - tests whether two means in a
repeated-measures study are statistically
different
▪ used in a repeated-measures design with
two levels
Quasi-Experimental Designs
❖ Quasi – Experiment - a study that takes place in ❖ ANOVA for Complex Designs
a real - life setting as opposed to a laboratory;
conclusions are not so easily drawn o When Interaction Is Not Significant
▪ we focus on the main effects, treating two-
❖ Difficulties with Experiments in Real - Life level significant effects like a t-test,
Settings ▪ for three or more levels, we conduct post-
hoc tests like the Tukey HSD to determine
▪ sometimes the researchers introduce an which means significantly differ from each
independent variable in a real-life setting. other
▪ however, it could also be the case that an
intervention was created and implemented o When Interaction Is Significant
by someone else and then studied by ▪ we need to analyze simple main effects,
researchers. which involves advanced data analysis
▪ the influence of possible external factors techniques and graphical interpretation of
cannot be estimated or controlled; never complex designs.
achieves the degree of control that
laboratory researchers obtain. o Mixed - Design ANOVA
▪ quasi - experiments usually have a ▪ we first check if the interaction is
comparison group significant, and if not, we analyze the main
effects.
o non-equivalent control group - a comparison ▪ if there are two levels, we compare the
group that is created by some method other means
than traditional randomization ▪ if there are more than two levels, we
conduct post-hoc tests.
❖ Threats to Internal Validity ▪ If the interaction is significant, we analyze
o history - when participants in a study simple main effects using SPSS
experience some common social or cultural ❖ SPSS (Statistical Package for the Social
event, not related to the independent variable, Sciences) - software program widely used in
that could affect the outcome of the study. research for statistical analysis and data
o maturation - when the participants change management; provides a range of tools and
during the course of the experiment. features that allow researchers to analyze data,
generate reports, and visualize results
o instrumentation (aka instrumental bias/decay)
- occurs when the scale, survey, or
performance measure used to measure the
dependent variable changes over time.. Prepared and arranged by:

o statistical regression (aka regression toward

the mean) - tendency for extreme scores on Yuan C. Obal
any measurement to move toward the mean Psychological Society QC, Educational Committee
(regress) when the measurement procedure is
repeated.
▪ that scores that are way above or way
below the mean will, on a subsequent
measure, return or regress back toward the References:
true mean.
Vanderstoep, S. W., & Johnston, D. D. (2009).
o attrition - participant withdrawal from a Research Methods for Everyday Life. Jossey-
research study before it is completed. Bass.

Data Analysis
❖ Repeated - Measures ANOVA - the procedure to P.S. check out the FILD 311 midterm reviewer for
test significant differences If the independent additional information on some of the highlighted aspects
of quantitative research, notably experimental designs :)
7