Comprehensive Exam Review

715: Research Methods and Designs in the Behavioral Sciences

Relevant links
1. https://study.com/academy/lesson/debriefing-in-psychology-research-de
finition-process-quiz.html
2. https://www.stcloudstate.edu/irb/_files/documents/debriefing-example.
pdf
3. https://great-home-decorations.com/intervening-variable-definition-exa
mple/#:~:text=For%20example%2C%20income%20is%20an%20interven
ing%20variable%20that,%28independent%20variable%29%20and%20ca
ncer%20mortality%20rate%20%28dependent%20variable%29.
4. https://www.scribbr.com/methodology/quasi-experimental-design/#:~:t
ext=%20Types%20of%20quasi-experimental%20designs%20%201%20No
nequivalent,and%20field%20experiments%2C%20researchers%20normal
ly%20control...%20More%20
5. https://psych.hanover.edu/classes/ResearchMethods/Readings/Quasi_ex
perimental_designs.pdf

Research purposes:
Theory: Explanation about how things/ how the world are supposed to work
- Formulate hypotheses;

Exploratory Research: When little or nothing is known about the topic

● The purpose/goals: “What”, “How”, or “Where”
● Questions: What is It, How is it done, or Where is it?
Descriptive Research: Describes a topic
● Example: offending among a particular population
Explanatory Research: Provides explanations about a topic and builds off
knowledge gained from exploratory and descriptive research
● Questions: Why is it, What are the effects of it, What causes it?
Evaluation Research: Used to generate knowledge
● systematic assessment of the need for, implementation of, or output of a
program based on objective criteria
● Uses observation, interviews, content analysis, survey research, and
secondary data analysis
● DEMONSTRATING CAUSATION***, a program changing or
causing a change in individuals

Types of data:
 Inductive:
Deductive:
Quantitative: numbers
Qualitative:

Concepts: abstract ideas about things that are only in our mind, such as IQ and
embezzlement

Conceptualization (creating specific definitions) → operationalization (how will we

measure this concept? Based on the conceptual definition) → variables
Independent variable:
Dependent variable:
Confounding variables: variable that can have an effect on both the IV and DV
“Matching” minimizes confounding variables

Independent variable is the variable that will lead to a difference in the dependent
variable
Program is the independent variable bc it is causing the difference
Outcome is dependent variable

Data Design:
Experimental design- to establish a causal relationship; identify the causal
relationship and to prove the variation in the dependent variable is due to the
independent variable
- Purpose: to identify causation
- Causation exists: when one variables influences the other
Before we conduct the experiment, 3 criteria must be met
1. Temporal ordering- the IV must come before the DV
2. Association- variables must be associated with each other; we can
determine this by running a correlations or regression test in spss
3. No spurious relationship - when the IV seems to be affecting the DV but in
reality there is a third variable (confounding variable) which is affecting
the dependent variable
Ex. ice cream sales increase shark deaths- the confounding variable
is weather which is causing people to go in the ocean more
- True experiment: conduct a post test to see if there is a difference in
control and experimental group
1. At least one experimental and one control group
2. Random assignment to both groups
a. Matching
3. Manipulation of an independent/ predictor variable
 Quasi- experiment: If asked to evaluate a program, it is a quasi-experiment.
- Can't randomly assign individuals to control/ exp groups (ex:
pregnant women to alcohol study)
- Research design that includes most, but not all, elements of
an experiment.
- Design to maximize internal validity (threat: confounding
variables); to reduce confounding variables, we match
individuals in the sample
- When random assignment is not possible, researchers can
use different types of quasi-experimental designs.
- Designed to use confounding variables
- Paired samples t test; independent samples t
Secondary analysis:
Quantitative:
Qualitative:
Mixed Methods:
Ethnography:
Grounded theory:
Survey:
Narrative study:

Sampling Methods:
*Random assignment is a way to minimize confounding variables (variables that
influence both the independent and dependent variables)
is a central feature of the classical experiment. The most important charac-
teristic of random assignment (sometimes referred to as randomization) is
that it produces experi- mental and control groups that are statistically
equivalent.
Sampling error: to minimize sampling error, get a larger sample
Sampling size: large= more representative, but smaller sample error because
individuals are more likely to be represented; bc its so large, there is more room
for error in the study
Within sampling frame (list of entire population), we have sampling elements;

Probability Sampling​: Process of selecting a subset of elements from a

comprehensive list of the population. This approach requires that every population
element have a known, nonzero chance of being selected.
● ​Simple random sampling​: Any sampling element can appear in a
sample only once. Should, during the selection of a sample, an element be
selected a second (or more) time, a researcher should simply move on with
the sample selection.
 ○ Random number generator
○ Putting all names in a hat and drawing out names

● S
​ ystematic Sampling​: Probability sampling approach in which the
sample is constructed by selecting elements from a sampling frame using a
sampling interval(number or distance between each sampling element).
K = N (population in the sampling frame)/n (total sample size desired)

● Stratified Sampling: the sampling frame is first divided into mutually

exclusive sub groups in strata, from each subgroup you select your sample
● Group of 100, divide into 4 groups, choose 3 people per group
○ Stratified sampling is a method for obtaining a greater degree of
representativeness—decreasing the probable sampling error.
○ Recall that sampling error is reduced by two factors in the sample
design: (1) a large sample produces a smaller sampling error than a
small sample does, and (2) a homogeneous population produces
samples with smaller sampling errors than a heterogeneous
population does.
● Cluster Sampling:
● Multistage Sampling:involves the initial sampling of groups of
elements—clusters—followed by the selection of elements within each of
the selected clusters.
●
- Comprehensive list that includes all elements from the population;
from sampling frame, the researcher selects his samples
Nonprobability sampling: Process of selecting a subset of elements in the
population without the presence of a comprehensive list of population members.
Some members of the population have a zero chance of being selected into the sample,
and the probability of being selected into a nonprobability sample is frequently
unknown.
● Quota Sampling​: Nonprobability sampling approach in which a
researcher gathers cases for the sample that have specific characteristics
such as urban, suburban, and rural households. This is roughly the
nonprobability sampling equivalent to stratified sampling.
○ addresses the issue of representativeness, although the two
methods approach the issue quite differently. Quota sampling
begins with a matrix or table describing the characteristics of the
target population we wish to represent. To do this, we need to
know, for example, what pro- portion of the population is male or
female and what proportions fall into various age categories,
education levels, ethnic groups, and so forth.
○ Needs to be proportioned with the population.
● Purposive Sampling​: Process of selecting a subset of elements from a
comprehensive list of the population. This approach requires that every
population element have a known, nonzero chance of being selected.
○ may be appropriate to select a sample based on our own knowledge
of the popu- lation, its elements, and the nature of our research
aims—in short, based on our judgment and the purpose of the study
○ Characteristics such as employment, history
○
● Convenience sampling:​sample is gathered simply based on
convenience and ease of finding.
● Snowball Sampling: current respondents of study seek out others with
same characteristic; mainly used if the characteristic is rare- ex: albino
individuals
○ Commonly used in field observation studies or qualitative inter-
viewing, snowball sampling begins by identifying a single subject or
small number of subjects and then asking the subject(s) to identify
others like him or her who might be willing to participate in a study.

Research Questions:
Data collection:
Primary data:
Secondary data: data collected by someone other that the primary user; such as
arrest/ prison records, census, data collected by government departments
 - advantages and Disadvantages of Secondary Data
- The advantages of secondary analysis are obvious and enormous: it
is cheaper and faster than collecting original data, and, depending on who
did the original study, you may benefit from the work of topflight
professionals and esteemed academics.
- Potential disadvantages must be kept in mind, however. The key
problem involves the recurrent question of validity. When one researcher
collects data for one particular purpose, you have no assurance that those
data will be appropriate to your research interests. Typically, you’ll find
that the original researcher collected data that “come close” to measuring
what you are interested in, but you may wish key variables had been
operationalized just a little differently. The question, then, is whether
secondary data provide valid measures of the variables you want to
analyze.
Measurement/ Conceptual issues:
What are you measuring? Construct/ operational definition
Measure: tools used to gather data (ex: scale to measure weight, questionnaire)

MEASUREMENT: consistency of measurement, did the individuals interpret the

question (measurement) the same way?
- Self reporting is not reliable because its based off their own memory, and some
may not want to include all details
- Reliability- follow ups
- More than 1 observer
- Pre test post test
CONCEPTUAL: validity
- Internal validity:
- Are there confounding variables?
- Testing- Ppl who have already been pretested, the post test can be affected
- maturation (ppl getting tired)
- reactive effect to study (when they are sensitized to the treatment)
- Reactivity- ppl altering their responses bc they know they're being
recorded
- Attrition: Mortality- people dropping out of study
- History: occurrence of an event that affects the study and results are
influenced
- External validity:
- Is the data generalizable?
- Because people know they are being studied, by reacting a certain way,
generalizability of data may not be generalizable
Ethical issues: chap3
IRB concerns:
Anonymity: A research subject is considered anonymous when the researcher
cannot associ- ate a given piece of information with the person.
- Anonymity addresses many potential ethical difficulties. Studies that use
field observation techniques are often able to ensure that research subjects
cannot be identified.
Confidentiality: Confidentiality means that a researcher is able to link
information with a given person’s identity but essentially promises not to do so
publicly.

Anonymity and confidentiality are two ways to protect the privacy of research
subjects.

Data issues:
Validity: is the measurement accurately measuring what it is supposed to
measure?
- Face validity: appears to be valid on its face
- Content validity:
- Internal validity:
- Construct Validity
- Internal Validity: can conclude a causal relationship between variables^
threats: experimental mortality; when participants drop out of the
research for multiple reasons******
- External Validity: generalizability of the findings
Threats: testing, individuals already take the test
-
- Case Study Approach
- Construct:Multiple sources of evidence, Establish chain of
causation, Member checks
- Internal: Pattern-matching, Time-series analysis
- External: Replicate through multiple case studies
Reliability: whether the measure provides consistent results
- Test retest reliability: test the person, retesting and getting the same
results
- Interrater reliability: More than one observer
Generalizability: Desirable characteristic of findings from research indicating
that the results from a sample can be applied to the larger population from which
the sample is taken.
Data richness:
Implications/ Significance:

Questions and Answers from Professor

1. Quasi-experiment is used when the researcher does not manipulate the independent
variable. So, say I want to conduct research on the effect or the impact of an
intervention. Quasi would be more suitable. However, how then would I include a
control group to compare it against? Possible?

Answer: Yes, Quasi is more suitable. You don’t have to include a control group because
you are comparing the before intervention and after intervention. But if you do, you will
draw a sample from persons who did not receive treatment/undergone the intervention
and compare this group to those who received treatment/undergone intervention.

2. The effect or impact of an intervention can be a one-group posttest only quasi

experiment? What if I wanted to use an interrupted time series quasi experiment?

Answer: One-group posttest only is not the best to use because you cannot compare
the before treatment/intervention. Interrupted time series or REPEATED MEASURES
(suggested) are more suitable.

3. Using a quasi-experiment doesn’t include randomly assigning participants to the

control and experimental group (if possible and ethical, to use control groups).

Answer: Random assignment is NOT the same as random sampling. It just means that
you cannot randomly assign someone who has received the treatment/intervention to
those who received and those who don’t receive. They have already received it. You just
need to add a control group of those who did not receive treatment/intervention to
compare against the experimental group.

4. How would I control for confounding/control/extraneous variables? Matching?

Answer: Yes, you can control for these variables through matching.

5. What sampling technique would be most suitable for quasi-experiments?

Answer: I am using systematic random sampling.

6. Can I combine this sampling technique with secondary data? For example, say my
sampling frame is from a prison database, from this, can I then use this sampling
technique to select my sample?
Answer: Yes, you can combine both.

7. Is ICPSR related to conducting an experiment?

Answer: Yes. You can get your sample from ICPSR. You won’t have to worry about
ethical concerns because these studies have already been conducted. So, you can get
your sampling frame of ICPSR and use a sampling technique to select your sample and
then conduct the experiment.

8. What is the difference between intervening variable and confounding variable? How
can we control for intervening variables?

Answer: Confounding variable is just a third variable that is affecting the dependent
variable. Intervening variable is a variable that makes it appear like there is a
relationship when there is not. There is no direct relationship between the
independent variable and the dependent variable. I am just sticking to confounding
variables. You can control for confounding variables statistically by using linear
regression- by adding a third variable (confounding variable) in SPSS. To add, before
using experiment, you must determine that there is an association. You can do this
statistically by running a correlation or linear regression.

716: Using Computers in Computer Research

Examining the distribution of variables

SPSS Steps: analyze →

descriptives→ explore→ select
variable & move into dependent →
none under boxplots → ok →
display both → ok

Skewness:

Kurtosis: pointiness of the distribution; refers to

the degree to which scores cluster at the ends of
the distribution (known as the tails); A
distribution with positive kurtosis has many
scores in the tails (a so-called heavy-tailed
distribution) and is pointy. This is known as a
 leptokurtic distribution. In contrast, a distribution with negative kurtosis is
relatively thin in the tails (has light tails) and tends to be flatter than normal
(platykurtic distribution).

- If skewness & kurtosis is less than 1, it is non significant.

-
- If a random variable’s kurtosis is greater than 3, it is said to be Leptokurtic.
If its kurtosis is less than 3, it is said to be Platykurtic.

Overall normality: If the Skewness or Kurtosis statistic is greater than +2 or

less than -2, the variable is not normally distributed. If it falls within +2 and -2, it
is normally distributed.
Descriptive Statistics: analyze→ descriptive statistics → frequencies→ move
variable → display histogram → select all stats
Mean
Median
Mode
Standard deviation: average distance from the mean
Skewness

“What percentage of the variable is x?” analyze → descriptive statistics → frequencies→

move variable → display histogram → select all stats → look @ frequency table,
cumulative percentage

If the dependent variable is quantitative, we are comparing averages

Independent variable: variable being manipulated (control group in experiment)

Dependent/ outcome variable: can NOT be manipulated (experimental group in

experiment)

Control variable: used to reduce internal validity

Experimental variable:

Tests:
Bivariate & Univariate
Independent Samples T Test: Compare 2 independent groups on an outcome
variable (EX: do males and females differ on a test value or qualitative answer);
used to compare 2 populations when nothing is known about either population;
Uses one variable that’s dichotomous and one quantitative
- Analyze → compare means → independent samples t test → (grouping
variable splits people into 2 groups you want to compare; fill in the codes
in ? → define groups, add codes → continue → select variable for test
variable → ok [look at levens test and determine if we accept or reject the
null] → use appropriate t test
○ APA format: A Levene’s test was run and the assumption of homogeneity
of variance was not met, p < .001. Married individuals were statistically
older (M = 45.02, SD 14.78) than unmarried individuals (M= 42.68, SD=
19.35), t (19.000.99) = -10.25, p < .001

Paired Samples T Test:

- Analyze → compare means → paired samples t test → move selected
variables → ok
○ Output: look @ sig value to determine if difference in sample means is
meaningful or not (tells us the probability of null hypothesis being true)
○ APA format: People spend statistically significantly more time on their
work computers at work (M=15.37, SD= 15.35) than at home (M=3.07, SD=
6.74), t( degree of freedom (N-1)) = [value under t], p < .001

The Chi- square test of independence: looks at the relationship between 2

QUALITATIVE variables
- Analyze → descriptive statistics → crosstabs → take 2 qualitative
variables and put 1 in columns and 1 in rows (will be specified; variable
that comes first/ columns [fewest levels], variable that comes
second/rows) → statistics → ✓ chi square → ✓ Phi & cramers V→
continue → cells → ✓column under percentages ✓ observed counts →
continue → ok
○ APA format: There is a statistically significant relationship between how
interested a person is in new inventions and technologies, 𝞆2(df) = (value
#), p < .001 or p= xxx
 ○ Statistically significant means there is a pattern (one leads to the other),
but the test will not tell you why it exists; can't assume that one variable is
causing the other

One Sample T Test: Compare one sample to known or hypothetical value or

proportion
- Analyze → compare means → one sample t test → select variable you're
interested in → test value (population mean you want to compare the
sample mean to) → ok
○ APA format: This sample (M= (sample mean), SD= [sample s.d]) is
statistically significantly older than the overall U.S population (μ=
[population mean]), t( population df) = (number under t in 2 decimals), p<
.001

The Chi- square goodness of fit: looks at the proportions found in a particular
sample and determines whether those proportions are statistically significantly
different from the proportions found in another group. So for example, I know
that the U. S. population is generally evenly distributed in terms of gender –
there are approximately equal numbers of men and women in the population. So
I can test my sample of Florida college students to see if my sample is also evenly
distributed in terms of gender.; qualitative & nominal variable
- Analyze → non parametric tests → legacy → dialogs → chi square OR
analyze → non parametric tests → chi square → THEN select variable and
move under test variable list → ✓ all categories equal & ✓get from data
→ ok
○ APA format: There is no statistically significant difference between the
gender distribution in our sample and that of the U.S population, χ2(degree
of freedom), p=.796
○ Or; There is a significant difference between the marriage status in the
ICPSR sample and that of the U.S population, χ2 (1)= 12782.914, p < .001.
A majority of the ICPSR sample is not married.

Correlations: looks for relationships between quantitative variables. You can

however use dichotomous qualitative variables (2 levels)
- Analyze → correlate → bivariate → move variables over → ✓ pearson
under correlation coefficient → two tailed → ✓ flag significant
correlations → ok
○ 2 tailed = looking for either positive or negative relationship
○ 1 tailed = looking for a relationship in only one direction
 ○ 0.10 - WEAK, 0.30- MODERATE, .050- STRONG
○ APA format: There is a weak negative statistically significant relationship
between age and hours spent on a computer per week, r( df which is n-2)=
[pearson correlation decimal], p<. 001. Older individuals tend to spend less
time on a computer than younger individuals (prediction).
○ Or - There is no statistically significant relationship between age and hours
spent on a computer per week, r(n-2) = [pearson correlation decimal], p=
0.143.

One way ANOVA: Compare 3 or more independent groups on an outcome

variable (EX: do individuals of different races differ on a test value or qualitative
answer)
Factor variable - qualitative variable
Dependent variable- quantitative variable

Analyze → compare means → one way anova → move over dependent variable → move
over factor variable (independent) → options, click descriptive statistics & homogeneity
of variance, welch → ok → post hoc, click bonferroni & games- howell → continue→ ok

Output: look at homogeneity of variance (based on mean) → has homogeneity of

variance been met? If less than .05, we reject null = assumption of homogeneity of
variance has been violated (not met), → use welch's f test → if sig is less than .05, reject
null and use games howell → compare

Equal variances not assumed →games howell

equal variances assumed → bonferroni
Only mention ones that are statistically significant

A homogeneity of variance test was run and the assumption was not met, F (4, 6581) =
6.23, p < .001. There is a statistically significant relationship between cocaine
availability and age when an individual first used cocaine, Welch’s F(4, 2151.528) =
21.99, p< .001. Individuals who stated getting cocaine is impossible (M= 20.94, SD=
5.64) are statistically significantly older than those who stated getting cocaine is fairly
difficult (M= 20.01, SD= 4.79 p= .011), fairly easy (M= 19.63, SD= 4.62, p< .001), and
very easy (M= 19.03, SD= 4.77, p < . 001). Individuals who stated getting cocaine is very
difficult (M= 20.54, SD= 5.10) are statistically significantly older than individuals who
stated getting cocaine is fairly easy (M= 19.63 S.D= 4.62, p < .001) and very easy (M=
19.03, SD= 4.77, p < .001). Individuals who stated getting cocaine is fairly difficult (M=
20.01, SD= 4.79) are statistically significantly older than those who stated it is very easy
(M= 19.03, SD= 4.77, p < .001). Overall, the harder it is for an individual to get cocaine,
the older the individual is when they first use cocaine.

TEST OF NORMALITY***
- Mann- Whitney U:
- Wilcoxon signed-rank test:
- Kruskal-Wallis:

Simple Linear Regression:

Type of analysis that looks at relationships between variables that can be
described as forming a straight line on a scatter plot.
Regression equation= y=mx+b, y= a+bx

Outcome variable→ quantitative

Predictor: quantitative or qual/dichotomous

analyze→ regression→ linear→ move over outcome/ dependent variable → move

predictor/ independent variable→ ok

Rsquare : .01- .09 = weak model

.09-.25 = moderate model
.25+ = strong model
A model of respondent age predicting time spent on a home computer per week was
found to be statistically significant, [look at anova results] F(1, 11905)= 100.17, p<.001.
This is a weak model, R^2= .008; .08% of the variance in hours spent on a home
computer per week is predicted by this model. Respondent age is a statistically
significant predictor of hours spent on a home computer per week, b= .-031, p <.001, for
every 1 year increase in age, the model predicts a decrease of .031 hours spent on a home
computer per week.

Multiple Regression:
 Outcome variable→ quantitative
Predictor variable → qualitative or dichotomous
Dummy coded- must be coded as 0 & 1, deals with interpretation of regression
coefficient of a dichotomous predictor

analyze→ regression→ linear→ move over outcome/ dependent variable → move

predictor/ independent variable→ add second predictor under independent→ ok

When you run a second model to compare, use adjusted r squared

B value for the qualitative dichotomous variable will the the value for the difference of
the means in the 2 groups

A model of respondent age and whether a person majored in math or engineering

predicting hours spent on a home computer per week was statistically significant, F(2,
11904)= 119.84, p <.001, R^2=.02. This is a weak model; 2% of the variance in hours
spent on a home computer per week is predicted by this model. Respondent age is still a
statistically significant predictor of hours spent on a home computer per week, b= -.03,
p<.001; for every 1 year increase in age, the model predicts a decrease of .03 hours
spend on a home computer per week. Majoring in math or engineering is also a
statistically significant predictor of hours spent on a home computer per week, b= 2.77,
p<.001; the model predicts that individuals who majored in math or engineering in
college spend an average of 2.77 hours more time on a home computer per week than
those that didn’t.

Correlation:

Questions for review:

- What can be some conceptual concerns with quant/ qual research?

- For the test of normality, do we use descriptive statistics or we do we use mann
whitney u, wilcoxon signed rank test, & kruskal wallis
- For quasi, say our population is persons with mental illness, we could divide this
group into two groups where one receives the treatment and the other does not
receive the treatment? Also, for a repeated measures design, we do NOT divide
the group, and then we test the same group before the treatment and then after
the treatment. Both of these acceptable for a quasi experiment?
- What can be a measure/instrumentation for this?
- What's an ideal answer to explain kurtosis and overall normality for the comp?
If the dependent variable is skewed, run a non parametric test. (mann whitney u)

Notes from shanelle:

- The independent variable is the program.
- The measure is the question asked to participants
- If you manipulate the IV and place one set into the exp group and the other in the control group, that’s true
experiment
- An Ethical concern that must be address then is if the treatment is beneficial to the exp group, we MUST
state that the treatment will be given to the control group as well
- In linear regression, the dependent variable should be quantitative and the independent/predictor variable
should be either quantitative or dichotomous
- for linear, the dependent variable can be quantitative or dichotomous.
-