ADVANTAGES AND DISADVANTAGES OF PROBABILITY AND NON-PROBABILITY

BASED SURVEYS OF THE ELDERLY AND DISABLED.

Virginia M. Lesser, Director, OSU Survey Research Center,

National Center for Accessible Transportation
Oregon State University

SUMMARY Q2: Today development policies encourage separation of residential and commercial land uses. In 2020, should
Although it may be desired to extrapolate conclusions from a sample survey to a large population, often Corvallis have residences in one area, commercial uses in another, or should Corvallis have mixed use
times this may be cost prohibitive. For example, conducting a probability survey of all households in the neighborhoods with residences within walking distance of shops and other services?
United States can be expensive. Many telephone numbers need to be dialed to obtain just one participating
Comparison of Response From a Probability
household due to an increasing number of answering machines and refusals to participate. On the other and Non-Probability Sample (Q2)
hand, researchers may use a less expensive approach such as obtaining membership lists that often include
name, address and telephone number. A drawback of using such lists is that now the researcher is limited 80
to drawing conclusions only to members on the list. For example, results of a survey drawn from individuals 60
40
with a disability that are National Multiple Sclerosis Society (NMSS) members can draw conclusions to
20
members of the NMSS, not all individuals with a disability. An alternative approach is to use a probability- 0
based method of sampling individuals. If the sample design and survey operations are designed and Separate Mixed Use
implemented well, probability sampling can use results based on a small sample to make inferences about a
Non-Prob Sample Probability Sample
large population. Examples comparing these two approaches are discussed in this presentation.
Clearly, the comparison of these two questions indicates very different responses from a probability sample of
households and one obtained from an undefined sample of households.
PURPOSE
Surveys are used to obtain up to date information on public opinion to various topics. This paper will discuss
two sampling methodologies to obtaining survey data. We wish to determine if results of a survey based on ACCESSIBLE TRANSPORTATION STUDY:
individuals obtained from disability membership lists provide similar results compared with a survey based The National Center for Accessible Transportation is determining the perception of individuals traveling with a
on individuals drawn from a random sample of individuals across the U.S. disability stakeholder on the state of accessibility in air travel why individuals with a disability do not travel by airplane
1. Approach using sampling membership lists. or other forms of public transportation.
Simple.
Approach: Compare two sampling methods:
Less Expensive.
A. Non-probability sampling: Obtain a sample of individuals with a specific disability from a membership list of a
Results applicable to members only.
major national disability organization. Used probability sampling when sampling from the list.
2. Approach using probability sampling of households.
B. Probability sampling: Obtain a random sample of individuals with a disability, regardless of whether they are
Expensive.
members of a disability organization.
Results extrapolated to the population which was used to draw the sample.
For both: Identify individuals from the samples with a disability and who traveled by air. Send these individuals a
questionnaire on transportation experiences.
Study Design:
METHODS A. Sampling Membership Lists:
In Probability sampling, every element in the population has a known nonzero probability of being included Contacted 63 major U.S. disability groups (list provided by the U.S. Department of Health and Human
in the sample. In survey sampling, the sample is selected from a countable population of individuals. Services Office on Disability).
In Non-probability sampling, every element in the population does not have a nonzero probability of being Contacted Executive Directors or Presidents of the organizations multiple times using email and/or
included in the sample. Examples: telephone from March through December of 2006.
1. Quota sampling obtains interviews from a specified number of individuals from certain demographic Explained the objectives of NCAT, the procedures of study, confidentiality of data, and reimbursement of
subgroups or membership lists. expenses.
2. Judgment sampling, respondents are selected that are considered to be most representative of the A random sample of individuals from participating organizations will be selected and mailed a screener
population being surveyed. questionnaire to determine eligibility of disability and travel.
This approach is an example of a probability sample of individuals who belong to a disability organization,
but not a probability sample of individuals with a disability living in the U.S.
EXAMPLE OF A PROBABILITY AND NON-PROBABILITY STUDY: B. Random Probability Sample:
The OSU Survey Research Center conducted a survey for the City of Corvallis in 1997 to obtain opinions on All residents in the U.S. that have a landline telephone are eligible to be called in this study. Phone numbers
the development of the city over the next 20 years. Two approaches were used to obtain opinions for the are selected by random sampling, using random digit dialing techniques.
residents. The interviewer explained the objectives of NCAT, procedures of study, and assured confidentiality of
1. Probability survey: all households within the urban growth boundary had an opportunity to be responses.
selected and participate in a telephone survey. The probability of a household participating was This approach is an example of a probability sample of individuals with a disability living in the U.S.
known and accounted for in the data analysis.
2. Non-probability survey: A questionnaire was distributed in newsletter sent to all residents within the
urban growth boundary that were identified on a mailing list. Additional copies of the questionnaire Results:
were available at public buildings for residents to complete and submit. In this approach, there was
no limit on how many times an individual could complete and submit a questionnaire. Response Disposition for Disability
Organizations (n=63)

No response

Results of two key questions: Maybe, send info.

Based on this list, 15,000 members of the National
Yes Multiple Sclerosis Society have been surveyed at this
Q1: In 1997, the population of Corvallis is nearing 50,000. At the current rate of growth the population in No members stage.
2020 will be 66,600. If you were living in Corvallis in 2020, would you feel 66,600 is too large a population Undeliverable
for Corvallis, about right or too small? Cannot share list

Comparison of Response from a Probability and No members

Non-Probability Sample (Q1)
B. Random probability sample. As of June 1, 9000 telephone numbers called during May. Of this, 23 individuals
60 were identified that have a disability and have traveled by air over the last five years and agreed to answer
additional questions concerning their recent experiences.
40
20
0 DISCUSSION
Too Large About Right Too Small Not Sure Probability samples are expensive but provide results that can be extrapolated to a wide population.
Non-probability samples are less expensive but are limited in extrapolating results. Interpretation of results should
Non-Prob Sample Probability Sample
be viewed with caution.