NATIONAL UNIVERSITY OF SCIENCE

Labor & Workplace Studies&188-3

TECHNOLOGY
and Chicano Studies 191-5
STATISTICS &Applied Research
OPERATIONS Methods
RESEARCH in the LA
DEPARTMENT
Labor Community
SORS2206: SURVEY METHODS
Winter Quarter 2014

Mupondo Constantine Ndava

Principles of Survey
Methodology
W I N2021,
NUST T E RSORS2 0 1 4 , UCLA
 Outline

• What is a survey?
• What is survey methodology?
• Survey design
• Survey quality
• Ethical issues in using survey methodology

2
 What is a survey?

• A systematic method for gathering

information from (a sample of) individuals for
the purposes of describing the attributes of
the larger population of which the
individuals are members.
• The attributes attempt to describe basic
characteristics or experiences of large and
small populations in our world.

(Enanoria, 2005)

3
 Features of a survey

• Information is gathered by asking people

questions
• Information is collected either by having
interviewers ask questions and record
answers or by having people read or hear
questions and record their answers
• Information is collected from only a subset of
the population to be described (a sample)
rather than from all members
(Groves, et al., 2009)
4
 First questions to ask about a new
survey
• What is the target population (whom is it
studying?)
• What is the sampling frame (how do they
identify the people who have a chance to be
included in the survey)?
• What is the sample design (how do they select
the respondents)?
• What is the mode of data collection (how do
they collect data)?
• Is it an ongoing survey or a one-time survey?
5
 The survey process

what we usually ask

ourselves when we have a A study plan is a summary
problem, when we are of the subject requirements
stressed, or when we are not of your course and indicates
getting the results we desire. when planned subjects
These types of situations should be studied.
give us an opportunity to
gain awareness.

the extent to which a study

establishes a trustworthy
the extent to which you can generalize cause-and-effect relationship
your findings to different groups of between a treatment and an outcome.
(Hulley et al., 2007)
people, situations, and measures.
6
 Elements of survey research
Element Purpose

Research questions What questions will the study address?

Background and significance Why are these questions important?

Design
Time frame
Approach
Subjects Who are the subjects and how will they
Selection criteria be selected?
Sampling design
Variables What measurements will be made?
Predictor variables
Confounding variables
Outcome variables
Statistical issues How large is the study and how will it be
Hypothesis analyzed?
Sample size (Hulley et al., 2007)
Analytic approach
7
 Research question
Research question = the objective of the study, the
uncertainty the researcher wants to resolve. It often begins with
a genera question and then is narrowed down to become a
specific, concrete, researchable issue.

Example research question:

Does publication pressure have an impact on the practice of
science?

Breaking down the question:

How do scholars perceive the publish-or-perish culture?
What are the perceived intended and unintended consequences?
How do scholars perceive academic success?
How do publication pressure affect the intrinsic rewards of the job?

8
 Background and significance
• Sets the proposed study in context and gives its
rationale:
• What is known about the topic at hand?
• Why is the research question important?
• What kind of answers will the study provide?
• This section should:
• Cite previous research that is relevant
• Indicate the problems with the prior research and
what uncertainties remain
• Specify how the findings of the proposed study will
help resolve these uncertainties and lead to new
scientific knowledge

9
 Design

(Groves et al., 2009)

10
 Design: constructs

• Constructs: the elements of information that

are sought by the researcher.
• In some sense, constructs are ideas
• Can be abstract

• Example:
Perceived consequences of publication pressure
Perception of determinants of academic success

11
 Design: measurement

• Measurements are more concrete than

constructs
• Ways to gather information about constructs
• Often the questions posed to a respondent, using
words
• Critical task for measurement:
• Design questions that produce answers reflecting
perfectly the constructs you are trying to measure

12
 Example of constructs and measurements
Construct:
Perceived Authoritative Advantage (The extent to which a biomedical scientist
believes that publishing in an open access journal would enhance his or her
authority)

Publishing in an open access journal makes me more prestigious.

Publishing in an open access journal makes me more well-known.
Publishing in an open access journal makes me more concept-marker.
Publishing in an open access journal makes me nobler.
Publishing in an open access journal improves my priority.
Publishing in an open access journal makes me more trustworthy.
Publishing in an open access journal makes me more credible.
Publishing in an open access journal makes me more reliable.
Publishing in an open access journal makes me more scholarly.
Publishing in an open access journal makes me more authoritative by copyright

13
 Example of constructs and
measurements
• Measurements:

Publishing in an open access journal makes me

more prestigious.
Strongly agree neutral disagree Strongly No opinion
agree disagree
5 4 3 2 1 N/A

Survey methodology, Wuhan

6/22/12 14
University
 Design: response

• The data produced in a survey come from

information provided through the survey
measurements

Survey methodology, Wuhan

6/22/12 15
University
 Design: representation dimension
• The target population
• E.g., children and teenagers aged 6-18, households in
China, persons with disabilities
• The frame population:
• The set of target population members that has a chance to
be selected into the survey sample
• E.g., using 户籍文档 to sample children and teenagers, using
telephone directory to select households
• The sample
• A sample is selected from a sampling frame.
• The respondents

Survey methodology, Wuhan

6/22/12 16
University
 Design: sampling
• Sampling plan • Procedures for
• How an adequate obtaining population
sample size will be estimates from the
determined sample data and for
• The choice of media estimating the
through which the reliability of those
survey will be
administered
population estimates
• Telephone interviews
• F-to-F interviews
• Mailed questionnaire
• Online questionnaire

17
 Design: sample selection

• Sample size is determined by:

• Desired degree of precision the quality, condition, or fact of being exact and accurate.
• Statistical power required
• Ability of the researcher to gain access to the
study subjects
• Degree to which the population can be stratified
• Selection of the relevant units of analysis

18
Study design sets up inferences

(Hulley et al., 2007)

19
 Design: survey instrument

• Standards for good survey questions

• Question wording
• Feasible and ethical
• Biased wording
• Biased context

20
 Types of survey questions

• Open-ended questions
• Closed-ended questions
• Questions that describe and evaluate people,
places, and events
• Evaluative scales: usually 5-7 response options
• Agreement scales
• Other rating scales: importance (1st, 2nd, 3rd,…)
• Questions that measure responses to ideas,
analyses or proposals
• Questions that measure knowledge: familiarity
with something

21
 Survey
process

(Groves et al., 2009)

22
Example: ARL Annual Salary Survey
http://www.arl.org/stats/annualsurveys/salary/salform11.shtml

• http://www.arl.org/stats/annualsurveys/
salary/salform11.shtml
• Cover letter to directors
• Instructions about data input
• Survey form
• Data template
• ARL library codes
• Standard for race and ethnicity

23
Survey Methodology
QUALITY CONTROL

24
 A few important terms

• Interviews: verbal surveys

• Questionnaire: written surveys
• Reliability: the consistency of survey
responses over time
• Item consistency: whether the responses for
each question are consistent across
constructs
• Test administration and scoring consistency:
the probability of errors caused by
carelessness in administration or scoring
25
 Important terms (cont’d)
• Validity: the extent to which the measurements of the
survey provide the information needed to meet the
study’s purpose

• Content validity considers whether the questions measure the

content they were intended to measure.

• Predictive validity examines whether the responses are able to

predict a criterion measure.

• Concurrent validity addresses the correlation of survey results

with results from other sources.

• Construct validity asks whether the survey questions capably

measure hypothetical constructs.

26
 What is survey methodology

• Survey methodology is the study of survey

methods and the sources of error in surveys.
• “Error” describes deviations from the desired
outcome
• Survey methodology studies ways to minimize
such errors.

27
 Survey quality
perspective (1)

Issues of
measurement:
• Formulating
questions
• Response order
effects
• Bias in
questionnaires
• Formatting the
questionnaire
• Evaluating survey
questions
A response order effect occurs when the distribution of responses to a
closed-ended survey question is influenced by the order in which the
response options are offered to respondents. Primacy and recency effects
are two common types of response order effects.
The primacy effect, in psychology, is a cognitive bias that results from
The recency effect is a cognitive bias in which those items, ideas, or disproportionate salience of initial stimuli or observations. If, for example, a
arguments that came last are remembered more clearly than those that came subject reads a sufficiently-long list of words, he or she is more likely to
first. remember words read toward the beginning than words read in the28 middle.
Survey quality
perspective (2)

29
 Implementing the study

• Modes of administration
• Pilot studies
• Quality control
• Procedures
• Data
• Post-collection of response data

30
Ethical issues

31
 Ethical issues in research involving
human subjects
• The Belmont Report Basic Ethical Principles:

1. Respect for persons

2. Beneficence - (1) do not harm and (2)
maximize possible benefits and minimize
possible harms
3. Justice – fair procedures and outcomes in
the selection of research subjects

32
U.S. federal policy for the protection of
human subjects
• Regulations for:
• The definition of research
• IRB membership
• IRB functions and operations
• IRB review of research
• Criteria for IRB approval of research
• IRB’s authority to suspend or terminate approval of
research
• IRB records
• General requirements for informed consent
• Documentation of informed consent

33
Definition of human subjects research
as defined by federal policy
• Research means a systematic investigation,
including research development, testing and
evaluation, designed to develop or contribute
to generalizable knowledge.

• Human subject means a living individual about

whom an investigator (whether professional or
student) conducting research obtains
• (1) Data through intervention or interaction with the
individual, or
• (2) Identifiable private information.

34
 IRB application content
• Purpose of Research (Rationale or research question)
• Approach/Method –What are you going to do and what is going to
happen to the subject?
• Qualifications of the Researchers
• Characteristics of the Subject population to be Recruited
• Special Groups Involved in Research
• Type of Informed Consent to Be Obtained
• Precautions to Ensure Privacy and Confidentiality
• Risks to Subjects (physical, psychological, financial, etc.) and how the
risks will be managed.
• Benefits to Subjects and Society at Large - How Do the Benefits Outweigh the
Risks?
• All applications should be completed under the guidance of the
student’s faculty advisor and reviewed by the faculty advisor prior to
submission.
• The application must be signed by the faculty advisor and the
student .

35
 Informed consent
• Required elements include (are not limited to):
• Statement – that the study involves “research”
• Purpose of Research
• Procedures involved in the Research
• Duration of Involvement in the Research
• Participation is Voluntary-Right to withdraw at anytime
without Penalty
• Right to confidentiality
• Risks and Benefits
• Who to Contact for More Information or Questions
• Who to Contact if a Participant is Harmed or Has
Concerns
36
 Informed consent (cont’d)

• Use everyday language appropriate to

potential participant (readable and clear)
• On institution’s Letterhead
• Title of Research
• Signature lines

37
 References
• Enanoria, W. (2005). Introduction to survey methodology.
http://www.idready.org/courses/2005/spring/
survey_IntroSurveyMethods.pdf
• Glasow, P.A. (2005). Fundamentals of survey research methodology.
http://www.mitre.org/work/tech_papers/
tech_papers_05/05_0638/05_0638.pdf
• Groves, R. et al. (2009). Survey Methodology. Hoboken, NJ: John
Wiley & Sons.
• Hulley, S.B. et al. (2007). Designing Clinical Research. 2nd ed.
Philadelphia, PA: Lippincott Williams & Wilkins.

38