184 8 The nature of quantitative research

● Quantitative research can be characterized as exhibiting certain preoccupations, the most central of
which are: measurement; causality; generalization; and replication.
● Quantitative research has been subjected to many criticisms by qualitative researchers. These
criticisms tend to revolve around the view that a natural science model is inappropriate for studying
the social world.

? QUESTIONS FOR REVIEW

The main steps in quantitative research

● What are the main steps in quantitative research?
● To what extent do the main steps follow a strict sequence?
● Do the steps suggest a deductive or inductive approach to the relationship between theory and
research?

Concepts and their measurement

● Why is measurement important for the quantitative researcher?
● What is the difference between a measure and an indicator?
● Why might multiple-indicator approaches to the measurement of concepts be preferable to those that
rely on a single indicator?

Reliability and validity

● What are the main ways of thinking about the reliability of the measurement process? Is one form of
reliability the most important?
● ‘Whereas validity presupposes reliability, reliability does not presuppose validity.’ Discuss.
● What are the main criteria for evaluating measurement validity?

The main preoccupations of quantitative researchers

● Outline the main preoccupations of quantitative researchers. What reasons can you give for their
prominence?
● Why might replication be an important preoccupation among quantitative researchers, in spite of
replications in business research being fairly rare?

The critique of quantitative research

● ‘The crucial problem with quantitative research is the failure of its practitioners to address adequately
the issue of meaning.’ Discuss.
● In the critique by qualitative researchers of quantitative research, how central is the adoption by
quantitative researchers of a natural science model of conducting research?

ONLINE RESOURCES
www.oup.com/uk/brm5e/

Visit the Interactive Research Guide that accompanies this book to complete an exercise focused on
the nature of quantitative research.
 CHAPTER

9
SAMPLING IN
QUANTITATIVE RESEARCH
Introduction 186 Time and cost 196
Non-response 196
Introduction to sampling 187
Heterogeneity of the population 197
Sampling error 189
Types of non-probability sampling 197
Types of probability sample 191 Convenience sampling 197
Simple random sample 191 Quota sampling 198
Systematic sample 191 Limits to generalization 201
Stratified random sampling 192
Error in survey research 202
Multi-stage cluster sampling 192

The qualities of a probability sample 193 Sampling issues for online surveys 202

Sample size 195 Key points 204

Absolute and relative sample size 195 Questions for review 205

CHAPTER OUTLINE

This chapter and the three that follow it are very much concerned with principles and practices associated
with social survey research. Sampling principles are not exclusively a concern for survey research; for
example, they are relevant to the selection of documents for content analysis (see Chapter 13). However,
in this chapter the emphasis will be on sampling in connection with the selection of people who would be
asked questions by interview or questionnaire. The chapter explores:

• the related ideas of generalization (also known as external validity) and of a representative sample (the
latter allows the researcher to generalize findings from a sample to a population);

• the idea of a probability sample—that is, one in which a random selection process has been employed;

• the main types of probability sample: the simple random sample; the systematic sample; the stratified
random sample; and the multi-stage cluster sample;

• the main issues involved in deciding on sample size;

• different types of non-probability sample, including quota sampling, which is widely used in market
research and opinion polls;

• potential sources of error in survey research;

• some sampling issues in online surveys.

186 9 Sampling in quantitative research

Introduction FIGURE 9.1

Steps in planning a social survey

This chapter is concerned with some important aspects Decide on topic/area to be researched
of conducting a survey, but it presents only a partial pic-
ture, because there are many other steps. In this chapter
Review literature/theories relating to topic/area
we are concerned with the issues involved in selecting
respondents for survey research, although the principles
involved apply equally to other approaches to quantitative Formulate research questions
research, such as content analysis. Chapters 9, 10, and 11
deal with the data collection aspects of conducting a sur- Consider whether a social survey is appropriate
vey, while Chapters 15 and 16 deal with the analysis of (if not, consider an alternative research design)
data.
Figure 9.1 is intended to outline the main steps
Consider what kind of population will be appropriate
involved in planning and conducting survey research.
The planning will begin with an examination of gen-
eral research issues that are to be investigated. These Consider what kind of sample design will be employed
are gradually narrowed down so that they become
research questions, which may take the form of hypoth- Explore whether there is a sampling frame that can be
eses, though this is not necessarily the case. The move- employed
ment from research issues to research questions is
likely to be the result of reading the literature relating Decide on mode of administration
to the issues. (face-to-face; telephone; postal; email; online)
Once the research questions have been formulated,
the planning of the fieldwork can begin. In practice,
Develop questions (and devise answer
decisions relating to sampling and to the research alternatives for closed questions)
instrument will overlap, but they are presented in Fig-
ure 9.1 as part of a sequence. The survey researcher
Review questions and assess face validity
needs to decide what kind of population is suited to
the investigation of the topic and also needs to formu-
late a research instrument and decide how it should Pilot questions
be administered. The ‘research instrument’ may be a
structured interview schedule (discussed in Chapter 10) Revise questions
or a self-completion questionnaire (Chapter 11). There
are several different ways of administering such instru-
Finalize questionnaire/schedule
ments. Figure 9.2 outlines the main types that are likely
to be encountered.
Quantitative research almost invariably involves sam- Sample from population
pling. In this chapter, we will be mostly concerned with
sampling for social survey research involving data collec- Administer questionnaire/schedule to sample
tion by structured interview or questionnaire. In social
survey research, sampling constitutes a key step in the
research process, as illustrated in Figure 9.1. However, Transform completed questionnaires/schedules
into computer-readable data (coding)
other methods of quantitative research also involve
sampling considerations, as will be seen in Chapter 13,
where we will examine structured observation and ­content Analyse data
analysis. The principles of sampling involved are more or
less identical in connection with these other methods, Interpret findings
though frequently other considerations come to the fore
as well.
Consider implications of findings for research questions
 Introduction to sampling 187
FIGURE 9.2
Main modes of administration of a survey

Survey

Structured Self-completion
interview questionnaire

Supervised Postal
Face-to-face Telephone Internet
5 6

Paper + pencil CAPI Paper + pencil CATI Online

Email
1 2 3 4 9

Embedded Attached
7 8

Introduction to sampling
Many of the readers of this book will be university or them, since conducting survey research by interview is
college students. At some point in your stay at your uni- considerably more expensive and time-consuming, all
versity (we will use this term from now on to include things being equal, than by questionnaire (see Chapter
colleges) you may have wondered about the attitudes of 10). It is almost certain that you will need to sample
your fellow students to various matters, or about their students from the total population of students in your
behaviour in certain contexts, or something about their university to collect data from a manageable number of
backgrounds. If you were to decide to examine any or respondents.
all of these three areas, you might consider conducting But will any old sample suffice? Would it be sufficient
structured interviews or sending out questionnaires in to locate yourself in a central position on your campus
order to find out about their behaviour, attitudes, and (if your university has one) and then interview the stu-
backgrounds. You would, of course, have to consider dents who come past you and whom you are in a posi-
how best to design your interviews or questionnaires, tion to interview? Alternatively, would it be sufficient to
and the issues that are involved in the decisions that go around your student union asking people to be inter-
need to be made about designing these research instru- viewed? Or to send questionnaires to everyone on your
ments and administering them will be the focus of course?
Chapters 9–11. However, before getting to that point, The answer depends on whether or not you want to be
you are likely to be confronted with a problem. Let us able to generalize your findings to the entire student body
say that your university has around 10,000 students. in your university. If you do—and as discussed in Chap-
It is extremely unlikely that you will have the time and ter 7, one of the preoccupations of quantitative research-
resources to collect data from all these students. It is ers is generalization—it is unlikely that any of the three
unlikely that you would be able to send questionnaires sampling strategies proposed in the previous paragraph
to all 10,000 and then process the results, and even would provide you with a representative sample of all stu-
more unlikely that you would be able to interview all of dents in your university. To be able to generalize your
188 9 Sampling in quantitative research

9.1 KEY CONCEPT

Basic terms and concepts in sampling
• Population: basically, the universe of units from which the sample is to be selected. The term ‘units’ is
employed because it is not necessarily people who are being sampled—the researcher may want to sample
from a universe of nations, cities, regions, firms, etc. Thus, ‘population’ has a much broader meaning in
sampling than it does in the everyday use of the term (meaning the total number of people in a nation or town).
• Sample: the segment of the population that is selected for investigation. It is a subset of the population. The
method of selection may be based on a probability or a non-probability approach (see below).
• Sampling frame: the listing of all units in the population from which the sample will be selected.
• Representative sample: a sample that reflects the population accurately so that it is a microcosm of the
population.
• Sampling bias: a distortion in the representativeness of the sample that arises when members of the population
(or more precisely the sampling frame) vary in terms of how likely they are to be included in the sample, such
that some groups are over-represented and others under-represented by comparison to their proportions in the
population.
• Probability sample: a sample that has been selected using random selection so that each unit in the population
has a known chance of being selected. It is generally assumed that a representative sample is more likely to be
the outcome when this method of selection from the population is employed. The aim of probability sampling is
to keep sampling error (see below) to a minimum.
• Non-probability sample: a sample that has not been selected using a random selection method. Essentially, this
implies that some units in the population are more likely to be selected than others.
• Sampling error: the difference between a sample and the population from which it is selected, even though a
probability sample has been selected.
• Non-sampling error: differences between the population and the sample that arise either from deficiencies in
the sampling approach, such as an inadequate sampling frame or non-response (see below), or from such
problems as poor question wording, poor interviewing, or flawed processing of data.
• Non-response: a source of non-sampling error that is particularly likely to happen when individuals are being
sampled. It occurs whenever some members of the sample refuse to cooperate, cannot be contacted, or for some
reason cannot supply the required data (for example, because they are no longer working in the organziation).
• Census: the collection of data from an entire population. Thus, if data are collected in relation to all units in a
population, rather than in relation to a sample of units of that population, the data are treated as census data.
The phrase ‘the census’ typically refers to collection of data from all members of the population of a nation
state—that is, a national census. Many countries around the world undertake a census every few years as a
way of generating a snapshot of a range of social and economic features of their populations. However, in a
statistical context, like the term population, the idea of a census has a broader meaning than this.

findings from your sample to the population from which sample? There are various reasons, of which the follow-
it was selected, the sample must be representative. This ing stand out.
is a fundamentally important point, because the aim of
analyzing data from a sample is to make inferences about • The first two approaches depend heavily upon the
availability of students during the time or times that
the larger population from which the sample was drawn,
you search them out. Not all students are likely to be
without having to try to collect data from the whole pop-
equally available at that time, so the sample will not
ulation. See Key concept 9.1 for an explanation of terms
reflect those students who are not available.
concerning sampling.
Why might the strategies for sampling students previ- • The first two approaches also depend on the students
ously outlined be unlikely to produce a representative going to the locations. Not all students will necessarily
 Sampling error 189
pass the point where you locate yourself or go to the Three sources of bias can be identified (see Key con-
student union, or they may vary significantly in the cept 9.1 for an explanation of terms):
frequency with which they do so. Their movements
are likely to reflect such things as where their halls of • If a non-probability or non-random sampling method is
used. If the method used to select the sample is not ran-
residence or accommodation are situated, or where
dom, there is a possibility that human judgement will
their departments are located, or their social habits.
affect the selection process, making some members of
Again, to rely on these locations would mean missing
the population more likely to be selected than others.
out on students who do not frequent them.
This source of bias can be eliminated through the use
• It is possible, not to say likely, that your decisions about of probability or random sampling, the procedure for
which people to approach will be influenced by your which is described later in this chapter.
judgements about how friendly or cooperative the
people concerned are likely to be or by how comforta- • If the sampling frame is inadequate. If the sampling
frame is not comprehensive or is inaccurate, or suffers
ble you feel about interviewing students of the same
from some other kind of similar deficiency, the sample
(or opposite) gender to yourself, as well as by many
that is derived cannot represent the population, even if
other factors.
a random or probability sampling method is employed.
• The problem with the third strategy is that students on
your course by definition take the same subject as each • If some sample members refuse to participate or cannot be
contacted—in other words, if there is non-response. The
other and therefore will not be representative of all
problem with non-response is that those who agree to
students in the university.
participate may differ in various ways from those who do
In other words, in the case of all three sampling not agree to participate. Some of the differences may be
approaches, your decisions about whom to sample are significant to the research question or questions. If the
influenced too much by personal judgements, by prospec- data are available, it may be possible to check how far,
tive respondents’ availability, or by your implicit criteria when there is non-response, the resulting sample differs
for inclusion. Such limitations mean that, in the language from the population. It is often possible to do this in
of survey sampling, your sample will be biased. A biased terms of characteristics such as gender or age, or, in the
sample is one that does not represent the population from case of something like a sample of university students,
which the sample was selected. As far as possible, bias whether the sample’s characteristics reflect the entire
should be removed from the selection of your sample. In population in terms of the students’ areas of study. How-
fact, it is incredibly difficult to remove bias altogether and ever, after non-response it is usually extremely difficult
to derive a truly representative sample. What needs to be to determine whether differences exist between the
done is to ensure that steps are taken to keep bias to an population and the sample in terms of ‘deeper’ factors,
absolute minimum. such as attitudes or patterns of behaviour.

Sampling error
In order to appreciate the significance of sampling we would expect our sample of 50 to be equally split
error for achieving a representative sample, consider in terms of this variable, as in Figure 9.4. If there is a
Figures 9.3–9.7. Imagine we have a population of small amount of sampling error, so that we have one
200 employees and we want a sample of 50. Imagine employee too many who is not appraised and one too
as well that one of the variables of concern to us is few who is, it will look like Figure 9.5. In Figure 9.6 we
whether or not employees receive regular performance see a rather more serious degree of over-representa-
appraisals from their immediate supervisor, and that tion of employees who do not receive appraisals. This
the population is equally divided between those who time there are three too many who are not appraised
do and those who do not. This split is represented by and three too few who are. In Figure 9.7 we have a
the vertical line that divides the population into two very serious over-representation of employees who do
halves (see Figure 9.3). If the sample is representative, not receive performance appraisals, because there are
190 9 Sampling in quantitative research

FIGURE 9.3 FIGURE 9.6

Having performance appraisals in a population of 200 A sample with some sampling error

Have performance Do not have performance Have performance Do not have performance
appraisal appraisal appraisal appraisal

FIGURE 9.4 FIGURE 9.7

A sample with no sampling error A sample with a lot of sampling error

Have performance Do not have performance Have performance Do not have performance
appraisal appraisal appraisal appraisal

FIGURE 9.5 35 employees in the sample who are not appraised,

A sample with very little sampling error which is much larger than the 25 who should be in the
sample.
Have performance Do not have performance It is important to appreciate that, as suggested above,
appraisal appraisal probability sampling does not and cannot eliminate sam-
pling error. Even with a well-crafted probability sample,
a degree of sampling error is likely to creep in. How-
ever, probability sampling stands a better chance than
non-probability sampling of keeping sampling error in
check so that it does not end up looking like the outcome
featured in Figure 9.7. Moreover, probability sampling
allows the researcher to employ tests of statistical signifi-
cance that permit inferences to be made about the popu-
lation from which the sample was selected. These will be
addressed in Chapter 15.
 Types of probability sample 191

Types of probability sample

Imagine that we are interested in levels of training, skill 4. List all the employees in the population and assign
development, and learning among employees, and that them consecutive numbers from 1 to N. In our case,
we want to examine the variables that relate to varia- this will be 1 to 9000.
tion in levels of training they have undertaken. We might 5. Using a table of random numbers, or a computer pro-
decide to conduct our research in a single nearby com- gram that can generate random numbers, select n
pany. This means that our population will all be employ- (450) different random numbers that lie between 1
ees in that company, which in turn will mean that we will and N (9000). Select the 450 employees whose num-
be able to generalize our findings only to employees of ber corresponds to each of the random numbers.
that company. We simply cannot assume that levels of
6. The employees to which the n (450) random numbers
training and their correlates will be the same in other
refer constitute the sample.
companies. We might decide that we want our research
to be conducted only on full-time employees, so that Two points are striking about this process. First, there is
part-time and subcontracted workers are omitted. Imag- almost no opportunity for human bias to manifest itself.
ine too that there are 9000 full-time employees in the Employees would not be selected on such subjective cri-
company. teria as whether they looked friendly and approachable.
The selection of whom to interview is entirely mechani-
cal. Secondly, the process is not dependent on the
Simple random sample employees’ availability. They do not have to be working
The simple random sample is the most basic form of in the interviewer’s proximity to be included in the sam-
probability sample. With random sampling, each unit of ple. The process of selection is done without their knowl-
the population has an equal probability of inclusion in edge. It is not until they are contacted by an interviewer
the sample. Imagine that we decide that we have enough that they know that they are part of a social survey.
money to interview 450 employees at the company. This It used to be common for researchers to use a table of
means that the probability of inclusion in the sample is random numbers to select a sample, and some textbooks
still include such tables as an appendix. The easiest way
450 to generate a table, however, is to use one of the many
i.e. 1 in 20
9000 random number generators which can be found online.
Alternatively, it is possible to generate a list of random
This is known as the sampling fraction and is expressed as
numbers using Microsoft Excel, SPSS, or other statistical
n packages.
N

where n is the sample size and N is the population size.

Systematic sample
The key steps in devising our simple random sample A variation on the simple random sample is the system-
can be represented as follows: atic sample. With this kind of sample, you select units
directly from the sampling frame—that is, without
1. Define the population. We have decided that this will
resorting to random numbers.
be all full-time employees at the company. This is our
We know that we are to select 1 employee in 20. With
N and in this case is 9000.
a systematic sample, we would make a random start
2. Select or devise a comprehensive sampling frame. between 1 and 20 inclusive, possibly by using a table of
It is likely that the company’s HR department will random numbers. Let’s assume that we start with the
keep records of all employees. This will enable us number 16. Thereafter, we would take every twentieth
to exclude those who do not meet our criteria for employee on the list. So the sequence will go:
inclusion—i.e. part-time employees and those who
work on the premises but are not employees of the 16, 36, 56, 76, 96, 116, etc.
company. This approach obviates the need to assign numbers
3. Decide your sample size (n). We have decided that to employees’ names and then to look up names of the
this will be 450. employees whose numbers have been drawn by the
192 9 Sampling in quantitative research

random selection process. It is important to ensure, how- than one stratifying criterion. Thus, it may be that you
ever, that there is no inherent ordering of the sampling would want to stratify by department, by gender, and by
frame, since this may bias the resulting sample. If there whether or not employees are above or below a certain
is some ordering to the list, the best solution is to rear- salary level or occupational grade. If it is feasible to iden-
range it. tify employees in terms of these stratifying criteria, it is
possible to use pairs of criteria or several criteria (such
as departmental membership plus gender plus occupa-
Stratified random sampling tional grade).
In our imaginary study of company employees, one of Stratified sampling is really feasible only when the
the features that we might want our sample to exhibit relevant information is available. In other words, when
is a proportional representation of the different depart- data are available that allow the ready identification of
ments in which employees work. It might be that the members of the population in terms of the stratifying cri-
kind of department an employee works in is viewed as terion (or criteria), it is sensible to employ this sampling
relevant to a wide range of attitudinal features that are method. But it is unlikely to be practical or economical
in turn relevant to the study of skill development and if the identification of population members for stratifica-
training. Generating a simple random sample or a sys- tion purposes entails a great deal of work because there
tematic sample might yield such a representation, so that is no available listing in terms of strata.
the proportion of employees from the sales and market-
ing department in the sample would be the same as that
Multi-stage cluster sampling
in the employee population and so on. Thus, if there are
1800 employees in the sales and marketing department, In the example we have been dealing with, employees
using our sampling fraction of 1 in 20, we would expect to be interviewed are located in a single company. Inter-
to have 90 employees in our sample from this department viewers will have to arrange their interviews with the
of the company. However, because of sampling error, it sampled employees, but, because they are all working
is unlikely that this will occur and more likely that there on the same premises (for the purposes of this hypotheti-
will be a difference, so that there may be, say, 85 or 93 cal example we are assuming that the company has only
from this department. one location), they will not be involved in a lot of travel.
Because it is very likely that the company will include However, imagine that we wanted a national sample of
in its records the department in which employees are employees. It is likely that interviewers would have to
based, or indeed may have separate lists of employees travel the length and breadth of the country to interview
for each department, it will be possible to ensure that the sampled individuals. This would add a great deal
employees are accurately represented in terms of their to the time and cost of doing the research. This kind of
departmental membership. In the language of sampling, problem occurs whenever the aim is to interview a sam-
this means stratifying the population by a criterion (in ple that is to be drawn from a widely dispersed popula-
this case, departmental membership) and selecting tion, such as a national population, or a large region, or
either a simple random sample or a systematic sample even a large city.
from each of the resulting strata. In the present example, One way in which it is possible to deal with this poten-
if there are five departments we would have five strata, tial problem is to employ cluster sampling. With cluster
with the numbers in each stratum being one-twentieth of sampling, the primary sampling unit (the first stage of
the total for each department. the sampling procedure) is not the units of the popula-
The advantage of stratified sampling in a case like tion to be sampled but groupings of those units. It is the
this is clear: it ensures that the resulting sample will be latter groupings or aggregations of population units that
distributed in the same way as the population in terms are known as clusters. Imagine that we want a nationally
of the stratifying criterion. If you use a simple random representative sample of 5000 employees who are work-
or systematic sampling approach, you may end up with ing for the 100 largest companies in the UK (this infor-
a distribution like that of the stratified sample, but it is mation is publicly available and generated through the
unlikely. Two points are relevant here. First, you can con- FTSE—Financial Times Stock Exchange—index; com-
duct stratified sampling sensibly only when it is relatively pany size is measured in terms of market capitalization).
easy to identify and allocate units to strata. If it is not Using simple random or systematic sampling would yield
possible or it would be very difficult to do so, stratified a widely dispersed sample, which would result in a great
sampling will not be feasible. Secondly, you can use more deal of travel for interviewers. One solution might be
 The qualities of a probability sample 193

9.2 RESEARCH IN FOCUS

A cluster sample survey of Australian workplaces
and employees
The 2015 Study of Australian Leadership (SAL) (see Research in focus 3.13) utilized a cluster-sampling approach.
The aim of the study was to collect data from workplaces and their employees. Because no population frame of
Australian workplaces exists, a sample of organizations was drawn from a commercial list of businesses. The
sample of organizations was stratified by industry and size. Workplaces were then selected from within
organizations, except in the case of single-site organizations, where the organization and the workplace were one
and the same. In multi-site organizations, different numbers of workplaces were selected, depending on the total
number of sites: one workplace was selected if the organization had 5 or fewer workplaces; two were selected if
the organization had 6 to 20 workplaces; and three were selected if the organization had 21 or more workplaces.
Employees were then sampled from within the sample of workplaces. In workplaces with fewer than 200
employees, all employees were surveyed. In larger workplaces a random sample of employees was surveyed. This
design produced a representative sample of Australian workplaces and employees within them, which allowed
workplace and employee data to be linked for the purposes of analysis.

to sample companies and then employees from each of be sampled from each of the industries, and then approxi-
the sampled companies. A probability sampling method mately 400 employees from each of the companies would
would need to be employed at each stage. Thus, we be interviewed. Thus, there are three separate stages:
might randomly sample ten companies from the entire
population of the 100 largest companies in the UK, thus
• group 100 largest UK companies by market capitaliza-
tion;
yielding ten clusters, and we would then interview 500
randomly selected employees at each of the ten compa- • sample one company from each of the major industries;
nies. Research in focus 9.2 gives an example of a study • sample 400 employees from each of the companies.
that used cluster sampling. In a sense, cluster sampling is always a multi-stage
A potential problem, however, is that there is no guar- approach, because clusters are always sampled first and
antee that these ten companies reflect the diverse range then something else—either further clusters or popula-
of industrial activities that are engaged in by the popula- tion units—is sampled.
tion as a whole. One solution to this problem would be Many examples of multi-stage cluster sampling entail
to group the 100 largest UK companies by industry. This stratification. We might, for example, want further to strat-
could be done using the codes from the Standard Indus- ify the companies according to whether their headquarters
trial Classification system or SIC, which is used widely are located in the UK or abroad. To do this we would group
to classify companies for the purposes of collecting data. companies according to whether their headquarters were
The next step would be to randomly sample companies based in the UK or elsewhere and then select one or two
from each of the major groups. One company might then companies per industry from each of the two strata.

The qualities of a probability sample

The reason why probability sampling is such an impor- to say that we treat the population data and the sample
tant procedure in social survey research is that based data as the same. Let us say we wish to examine the level
on information about a random sample, it is possible to of skill development in our sample of 450 employees.
make inferences about the entire population from which We will treat skill development as the number of train-
it was selected. In other words, we can generalize find- ing days completed in the previous 12 months. We can
ings derived from a sample to the population. This is not use the mean number of training days undertaken by the
194 9 Sampling in quantitative research

sample (X) to estimate the population mean (m). The the population mean will have known margins of error.
mean, or more properly the arithmetic mean, is the simple For more detail on this, see Tips and skills ‘Generalizing
average. This generalization from the sample mean to from a random sample to the population’.

TIPS AND SKILLS

Generalizing from a random sample to the population
Using our imaginary study of training and skill development in a single nearby company, let us say that the
sample mean is 6.7 days of training per employee (the average amount of training received in the previous
12 months in the sample). A crucial consideration here is: even if we used probability sampling, how confident
can we be that the mean number of 6.7 training days is likely to be found in the population? If we take an
infinite number of samples from a population, the sample estimates of the mean of the variable under
consideration will vary in relation to the population mean. This variation will take the form of a bell-shaped
curve known as a normal distribution (see Figure 9.8). The shape of the distribution implies that there is a
clustering of sample means at or around the population mean. Half the sample means will be at or below the
population mean; the other half will be at or above the population mean. As we move to the left (at or below the
population mean) or the right (at or above the population mean), the curve tails off, implying that there would
be fewer and fewer samples generating means that depart considerably from the population mean. The
variation of sample means around the population mean is the sampling error and is measured using a statistic
known as the standard error of the mean. This is an estimate of the amount that a sample mean is likely to
differ from the population mean.

FIGURE 9.8
The distribution of sample means
Number of samples

–1.96 Population +1.96

SE mean SE
Value of the mean
Note: 95 per cent of sample means will lie within the shaded area. SE =
standard error of the mean.

This consideration is important, because sampling theory tells us that 68 per cent of all sample means will lie
between plus or minus 1.00 standard error from the population mean and that 95 per cent of all sample means
will lie between plus or minus 1.96 standard errors from the population mean. It is this second calculation that is
crucial, because it is at least implicitly employed by survey researchers when they report their statistical findings.
They typically employ 1.96 standard errors as the crucial criterion in how confident they can be in their findings.
Essentially, the criterion implies that you can be 95 per cent certain that the population mean lies within plus or
minus 1.96 sampling errors from the sample mean.

If a sample has been selected according to probability sampling principles, we know that we can be 95 per cent
certain that the population mean will lie between the sample mean plus or minus 1.96 multiplied by the standard
error of the mean. This is known as the confidence interval. If the mean number of training days in the previous
 Sample size 195

twelve months in our sample of 450 employees is 6.7 and the standard error of the mean is 1.3, we can be 95 per
cent certain that the population mean will lie between

6.7 + (1.96 × 1.3)

and

6.7 − (1.96 × 1.3)

i.e. between 9.248 and 4.152.

If the standard error was smaller, the range of possible values of the population mean would be narrower; if the
standard error was larger, the range of possible values of the population mean would be wider.

If a stratified sample is selected, the standard error of the mean will be smaller; this is because the variation
between strata is essentially eliminated, because the population will be accurately represented in the sample in
terms of the stratification criterion or criteria employed. This consideration demonstrates the way in which
stratification injects an extra increment of precision into the probability sampling process, since a possible source
of sampling error is eliminated.

By contrast, a cluster sample without stratification exhibits a larger standard error of the mean than a comparable
simple random sample. This occurs because a possible source of variability between employees (that is,
membership of one department rather than another, which may affect levels of training undertaken) is
disregarded. If, for example, some departments have a culture of learning in which a large number of employees
were involved, and if these departments were not selected because of the procedure for selecting clusters, an
important source of variability would have been omitted. It also implies that the sample mean would be on the low
side, but that is another matter.

Sample size
One question about research methods that we are asked national probability sample of 1000 individuals in the UK
by students almost more than any other relates to the size is as likely to be representative as a national probability
of the sample: ‘How large should my sample be?’ or ‘Is my sample of 1000 individuals in the USA, even though the
sample large enough?’ The decision about sample size is latter has a much larger population. It also means that
not a straightforward one: it depends on a number of con- increasing the size of a sample increases the precision of a
siderations, and there is no one definitive answer. This is sample. This means that the 95 per cent confidence inter-
frequently a source of great disappointment to those who val referred to in Tips and skills ‘Generalizing from a ran-
pose such questions. Moreover, most of the time decisions dom sample to the population’ narrows. However, a large
about sample size are affected by considerations of time sample cannot guarantee precision, so that it is probably
and cost (see Tips and skills ‘Sample size and probability better to say that increasing the size of a sample increases
sampling’). Therefore, invariably decisions about sample the likely precision of a sample. This means that as sam-
size represent a compromise between the constraints of ple size increases, sampling error decreases. Therefore,
time and cost, the need for precision, and a variety of fur- an important component of any decision about sample
ther considerations that will now be addressed. size could be how much sampling error one is prepared
to tolerate. The less sampling error one is prepared to tol-
erate, the larger a sample will need to be. Fowler (1993)
Absolute and relative sample size however warns against a simple acceptance of this crite-
One of the most basic considerations, and one that is pos- rion. He argues that in practice researchers do not base
sibly the most surprising, is that, contrary to what you their decisions about sample size on a single estimate
might have expected, it is the absolute size of a sample of a variable. Most survey research is intended to gen-
that is important, not its relative size. This means that a erate a host of estimates—that is, of the variables that
196 9 Sampling in quantitative research

TIPS AND SKILLS

Sample size and probability sampling
As we have said, the issue of sample size is the matter that most often concerns students and researchers. Basically,
the bigger the sample, the more representative it is likely to be (provided the sample is randomly selected), regardless
of the size of the population from which it is drawn. However, when doing research projects, students typically need
to do their research with very limited resources. You should try to find out from your department or school if there are
any guidelines about whether or not samples of a minimum size are expected. If there are no such guidelines, you
will need to conduct your mini-survey in such a way as to maximize the number of interviews you can manage or the
number of questionnaires you can send out and process, given the amount of time and resources available to you.
Also, in many if not most cases, a truly random approach to sample selection may not be open to you. The crucial
point is to be clear about and to justify what you have done, as well as to demonstrate the limitations of your sample.
Explain the difficulties that you would have encountered in generating a random sample. Explain why you could not
include any more in your sample of respondents. But, above all, do not make claims about your sample that are not
sustainable. Do not claim that it is representative or that you have a random sample when it is clearly not the case
that either of these is true. In other words, be frank about what you have done. People will be much more inclined to
accept an awareness of the limits of your sample design than claims about a sample that are patently false. Also, it
may be that there are lots of good features about your sample—the range of people included, the good response rate,
the high level of cooperation you received from the firm. Make sure you play up these positive features at the same
time as being honest about your sample’s limitations.

make up the research instrument that is administered. cost at such a juncture, since striving for smaller and
He also observes that it is not normal for survey research- smaller increments of precision becomes an increasingly
ers to be in a position to specify in advance ‘a desired level uneconomic proposition.
of precision’ (1993: 34). Moreover, since sampling error
will be only one component of any error entailed in an
Non-response
estimate, the notion of using a desired level of precision
as a factor in a decision about sample size is not realis- Considerations about sampling error do not end here.
tic. Instead, to the extent that this notion does enter into The problem of non-response should also be borne in
decisions about sample size, it often does so in a general mind. Most sample surveys include a certain amount
rather than a calculated way. of non-response. Thus, it is likely that only some of our
sample will agree to participate in the research. If it is
our aim to ensure as far as possible that 450 employ-
Time and cost ees are interviewed and if we think that there may be
Time and cost considerations become very relevant in a 20 per cent rate of non-response, it may be advis-
this context. In the previous paragraph it is clearly being able to sample 540–550 individuals, on the grounds
suggested that the larger the sample size, the greater that approximately 90 will be non-respondents. For
the precision (because the amount of sampling error example, of the 143 survey questionnaires sent to com-
will be less). By and large, up to a sample size of around panies in T. C. Powell’s (1995) study of total quality
1000, the gains in precision are noticeable as the sample management, only 40 were returned and of these only
size climbs from low figures of 50, 100, 150, and so on 36 were usable, making a response rate of 25 per cent.
upwards. After a certain point, however, often in the This raises the question of whether or not this sample
region of 1000, the sharp increases in precision become is big enough to represent companies in the geographi-
less pronounced, and, although it does not plateau, there cal area of the north-eastern USA that the study claims
is a slow down in the extent to which precision increases to represent (see Chapter 11 for a further discussion of
(and hence the extent to which the sample error of the acceptable response rates). The issue of non-response,
mean declines). Considerations of sampling size are and in particular of refusal to participate, is of par-
likely to be profoundly affected by matters of time and ticular significance, because it has been suggested by
 Types of non-probability sampling 197

9.3 KEY CONCEPT

What is a response rate?
When a social survey is conducted, whether by structured interview or by self-completion questionnaire, it is
invariably the case that some people who are in the sample do not participate. The response rate is, therefore, the
percentage of a sample that does, in fact, agree to participate. However, the calculation of a response rate is a little
more complicated than this. First, not everyone who replies will be included: if a large number of questions are not
answered by a respondent, or if there are clear indications that he or she has not taken the interview or
questionnaire seriously, it is better to employ only the number of usable interviews or questionnaires as the
numerator. Similarly, it also tends to occur that not everyone in a sample turns out to be a suitable or appropriate
respondent or can be contacted. Thus the response rate is calculated as follows:

number of usable questionnaires

× 100
total sample − unsuitable or uncontactable members of the sample

some researchers that response rates (see Key con- organizational research. Strategies that can improve
cept 9.3) to surveys are declining in many countries. responses to survey instruments such as structured
This implies that there is a growing tendency towards interviews and questionnaires will be examined in
people refusing to participate in survey research. As Chapters 10 and 11.
long ago as 1973, an article in the American maga-
zine Business Week carried an article ominously
Heterogeneity of the population
entitled ‘The Public Clams up on Survey Takers’. The
magazine asked survey companies about their experi- Yet another consideration is the homogeneity and het-
ences and found considerable concern about declin- erogeneity of the population from which the sample
ing response rates (Business Week 1973). Similarly, is to be taken. When a sample is very heterogeneous,
in Britain, a report from a working party on the Mar- such as a sample of a whole country or city, the popu-
ket Research Society’s Research and Development lation which it represents is likely to be highly varied.
Committee in 1975 pointed to concerns among mar- When it is relatively homogeneous, such as members of
ket research companies. However, analyses of this a company or of an occupation, the amount of variation
issue by by Baruch and Holtom (2008) and Anseel is less. The implication of this is that, the greater the
et al. (2010) suggested that there is little evidence of heterogeneity of a population, the larger a sample will
a generalized decline in response rates in surveys in need to be.

Types of non-probability sampling

The term non-probability sampling is essentially an Convenience sampling
umbrella term to capture all forms of sampling that are
A convenience sample is one that is simply available to
not conducted according to the norms of probability
the researcher by virtue of its accessibility. Imagine that
sampling outlined above. It is not surprising, therefore,
a researcher who teaches at a university business school
that the term covers a wide range of types of sampling
is interested in the way that managers deal with ethical
strategy, at least one of which—the quota sample—is
issues when making business decisions. The researcher
claimed by some practitioners to be almost as good as a
might administer a questionnaire to several classes of stu-
probability sample. In this section we will cover two main
dents, all of whom are managers taking a part-time MBA
types of non-probability sample: the convenience sample
degree. The chances are that the researcher will receive
and the quota sample.
all or almost all the questionnaires back, so that there
198 9 Sampling in quantitative research

will be a good response rate. The findings may prove common and in some areas, such as consumer behav-
quite interesting, but the problem with such a sampling iour research, they have become the norm. Nonetheless,
strategy is that it is impossible to generalize the findings, there is evidence that we should be cautious in general-
because we do not know of what population this sample izing from convenience samples, particularly when they
is representative. They are simply a group of managers are samples of undergraduate students (Peterson and
who are available to the researcher. They are almost cer- Merunka 2014).
tainly not representative of managers as a whole—the It is common in some kinds of business research to
very fact they are taking this degree programme marks use university undergraduate students as a convenience
them out as different from managers in general. sample, particularly in experimental research. This has
This is not to suggest that convenience samples should been criticized by Hooghe et al. (2010), who argue that it
never be used. Let us say that our lecturer/researcher is can endanger the validity and generalizability of findings
developing a battery of questions that are designed to because undergraduate students are likely to think and
measure the ethical decision-making processes used by act differently from the general population. For instance,
managers. It is highly desirable to pilot such a research people from lower socio-economic groups are under-
instrument before using it in an investigation, and admin- represented in undergraduate student populations; stu-
istering it to a group who are not a part of the main study dents are likely to have more socio-economic resources
may be a legitimate way of carrying out some preliminary than is typical of the general population; and students
analysis of such issues as whether or not respondents may be inclined to exert more-than-typical cognitive
tend to answer a particular question in identical ways, or effort in order to give the ‘right’ answer. Moreover, it is
whether or not one question is often omitted when manag- highly problematic to make claims about managerial or
ers respond to it. In other words, for this kind of purpose, workplace phenomena based on such samples, as many
a convenience sample may be acceptable, though not undergraduates have little or no full-time workplace or
ideal. A second kind of context in which it may be at least managerial experience. This has led some management
fairly acceptable to use a convenience sample is when the journals to refuse to publish papers based on student
chance presents itself to gather data from a convenience samples, unless the sample is merely a convenient way
sample and it represents too good an opportunity to miss. to sample managers, as in the case of part-time MBAs
The data will not allow definitive findings to be generated, discussed above. The study described in Research in
because of the problem of generalization, but they could focus 9.4 used a convenience sample taken from the ‘real
provide a springboard for further research or allow links world’ rather than from among students.
to be forged with existing findings in an area.
It also perhaps ought to be recognized that conve-
Quota sampling
nience sampling probably plays a more prominent role
than is sometimes supposed. In the field of business and Quota sampling is used intensively in commercial
management, convenience samples are certainly very research, such as market research and political opinion

STUDENT EXPERIENCE
Using convenience sampling in a dissertation
Jordan conducted research on self-expression in the workplace. Her approach required a fairly large sample of
employees. The one-year duration of her honours program, and the need to fit her research around a demanding
coursework load, meant that she had to find an efficient way to generate a sample which would allow her to
conduct her analysis. She chose to survey graduate employees in a private-sector organization, who were
undergoing a training program run by an NGO she had previously worked for. All 515 graduate employees who
underwent the training program were given a paper survey to complete during their training program. She
explained: ‘[the NGO] provides training to this particular organisation and to all of the graduate employees … 515
of them. They … provide this training in two places in Melbourne and Sydney … the reason why I got 515 surveys
was because time was set aside during the workshop.’ Jordan’s use of convenience sampling provided her with an
effective means to collect data which allowed her to conduct her analysis in a situation in which she did not have
the time or resources to generate a random sample of employees.
 Types of non-probability sampling 199

9.4 RESEARCH IN FOCUS

Convenience sampling in a study of
discrimination in hiring
Derous et al. (2017) conducted a study as a means of studying how ethnic cues influenced the outcomes of
curriculum vitae (CV) screening during recruitment processes. Most studies of CV screening have used samples of
university students (Derous et al. 2017: 862), but this study utilized a sample of human resource (HR)
professionals as a means to increase the likelihood that the conclusions drawn were representative of processes
and outcomes involving respondents who undertake recruitment in their daily working lives.

The study was conducted in Belgium. The authors needed a large sample for their study, but there was no
sampling frame available from which to draw a random sample of respondents, so the sample was drawn from
membership lists of HR professional associations, business publications, and the researchers’ own networks. This
generated a sample of 1463 respondents, of whom 424 agreed to participate. Participants were all Caucasian.
Clearly this strategy did not generate a random sample and the sample cannot be regarded as representative in a
statistical sense. Nonetheless, the sampling strategy represented a viable way to draw a large sample of actual HR
professionals which could not otherwise have been generated.

The researchers provided each respondent with job advertisements for two different jobs in two different kinds of
businesses. They were each also provided with four fictional CVs, each including a photo of a fictional applicant. The
CVs showed a variety of combinations of skin tone (light versus dark) and name (Flemish versus Arab/Maghreb).
Respondents were asked to rate each applicant using a three-item measure with Likert scale responses. For
example, ‘Given all the information you read about this applicant, how likely is it that you would invite this applicant
for a job interview?’ (1 = not likely at all; 7 = very likely). The findings showed the equally-qualified candidates with
a dark skin tone were systematically rated less suitable for jobs than those with light skin tone, with the effect
varying depending on the nature of the job. Name did not appear to matter. The findings suggested that there was
systematic ethnically-based discrimination in CV screening and that there were subtle effects which arose from the
kinds of jobs involved.

polling. The aim of quota sampling is to produce a sample Accordingly, an interviewer may know that among the
that reflects a population in terms of the relative propor- various subgroups of people, he or she must find and
tions of people in different categories, such as gender, interview five Asian, 25-to-34-year-old, lower-middle-
ethnicity, age groups, socio-economic groups, and region class females in the area in which the interviewer has
of residence, and in combinations of these categories. been asked to work. The interviewer usually asks people
However, unlike a stratified sample, the sampling of indi- who are available to him or her about their characteris-
viduals is not carried out randomly, since the final selec- tics in order to determine their suitability for a particu-
tion of people is left up to the interviewer. lar subgroup. Once a subgroup quota (or a combination
Once the categories and the number of people to be of subgroup quotas) has been achieved, the interviewer
interviewed within each category (known as quotas) will no longer be concerned to locate individuals for that
have been decided upon, it is then the job of interview- subgroup.
ers to select people who fit these categories. The quo- The choice of respondents is left to the interviewer,
tas will typically be interrelated. In a manner similar to subject to the requirement of all quotas being filled, usu-
stratified sampling, the population may be divided into ally within a certain time period. Those of you who have
strata in terms of, for example, gender, social class, age, ever been approached on the street by a person toting
and ethnicity. Census data might be used to identify the a clipboard and interview schedule and have been asked
number of people who should be in each subgroup. about your age, occupation, and so on, before being
The numbers to be interviewed in each subgroup will asked a series of questions about a product or whatever,
reflect the population. Each interviewer will probably have almost certainly encountered an interviewer with
seek out individuals who fit several subgroup quotas. a quota sample to fill. Sometimes, he or she will decide
200 9 Sampling in quantitative research

not to interview you because you do not meet the crite-

• It is undoubtedly cheaper and quicker than an inter-
ria required to fill a quota. This may be due to a quota view survey on a comparable probability sample. For
already having been filled or to the criteria for exclusion example, interviewers do not have to spend a lot of
meaning that a person with a certain characteristic that time travelling between interviews.
you possess is not required.
A number of criticisms are frequently levelled at quota
• Interviewers do not have to keep calling back on
people who were not available at the time they were
samples.
first approached.
• Because the choice of respondent is left to the inter-
• Because calling back is not required, a quota sample is
viewer, the proponents of probability sampling argue easier to manage. It is not necessary to keep track of
that a quota sample cannot be representative. It may people who need to be recontacted or to keep track of
accurately reflect the population in terms of superficial refusals. Refusals occur, of course, but it is not neces-
characteristics, as defined by the quotas. However, in sary (and indeed it is not possible) to keep a record of
their choice of people to approach, interviewers may which respondents declined to participate.
be unduly influenced by their perceptions of how
friendly people are or by whether the people make eye
• When speed is of the essence, a quota sample is invalu-
able when compared to the more cumbersome proba-
contact with the interviewer (unlike most of us, who
bility sample. Newspapers frequently need to know,
look at the ground and shuffle past as quickly as pos-
quickly, how a national sample of voters feel about a
sible because we do not want to be bothered in our
certain topic or how they intend to vote at that time.
leisure time).
Similarly, if there is a sudden major news event, such
• People who are in an interviewer’s vicinity at the times as a major terrorist attack or a natural disaster, the
he or she conducts interviews, and are therefore avail- news media may seek a more or less instant picture of
able to be approached, may not be typical; in that case, the nation’s views or responses. Again, a quota sample
those who are included in the sample will not be typi- will be much faster.
cal. There is a risk, for example, that people in full-time
paid work may be under-represented.
• As with convenience sampling, quota sampling is use-
ful for conducting development work on new meas-
• The interviewer is likely to make judgements about ures or on research instruments. It can also be usefully
certain characteristics in deciding whether or not to employed in relation to exploratory work from which
approach a person: in particular, judgements about new theoretical ideas might be generated.
age. Those judgements will sometimes be incorrect—
for example, when someone who is eligible to be inter-
• Although the standard error of the mean should not
be computed for a quota sample, it frequently is. As
viewed, because a quota that he or she fits is unfilled,
Moser and Kalton (1971) observe, some writers
is not approached because the interviewer makes an
argue that the use of a non-random method in quota
incorrect judgement (perhaps that the person is older
sampling should not act as a barrier to such a compu-
than he or she looks). In such a case, a possible element
tation because its significance as a source of error is
of bias is being introduced.
small when compared to other errors that may arise
• It has also been argued that the widespread use of in surveys (see Figure 9.9). However, they go on to
social class as a quota control can introduce difficul- argue that at least with random sampling the
ties, because of the problem of ensuring that inter- researcher can calculate the amount of sampling
viewees are properly assigned to class groupings error and does not have to be concerned about its
(Moser and Kalton 1971). potential impact.
• It is not permissible to calculate a standard error of the
There is some evidence to suggest that, when compared
mean from a quota sample, because the non-random
to random samples, quota samples often result in biases
method of selection makes it impossible to calculate
(Yang and Banamah 2014). They under-represent people
the range of possible values of a population.
in lower social strata, people who work in the private sec-
All of this makes the quota sample look a poor bet, and tor and manufacturing, and people at the extremes of
there is no doubt that it is not favoured by academic income, and they over-represent women in households
researchers. It does have some arguments in its favour, with children and people from larger households (Marsh
however. and Scarbrough 1990; Butcher 1994).
 Limits to generalization 201
FIGURE 9.9
Four sources of error in social survey research

Error

Sampling Sampling-related Data collection Data processing

error error error error

Limits to generalization
One point that is often not fully appreciated is that even were involved in heavy industry, such as steel-making or
when a sample has been selected using probability sam- shipbuilding. The population from which the sample was
pling, any findings can be generalized only to the pop- selected consisted of all accountants and engineers who
ulation from which that sample was taken. This is an worked for these companies. Respondents were chosen
obvious point, but it is easy to think that findings from a randomly according to certain criteria for stratification,
study have some kind of broader applicability. If we take including age, job title, level in the company, and length
our imaginary study of training and skill development of service. It is interesting that there is no mention of
among employees of a company, any findings could be gender in the study, although we can fairly safely assume
generalized only to that company. In other words, you that, given that this was a study of accountants and engi-
should be very cautious about generalizing to employ- neers in the late 1950s, there is likely to be a male bias to
ees at other companies. There are many factors that may the study. The maximum number of individuals selected
imply that the level of training and skill development is for interview in each company was approximately 50. As
higher (or lower) in that company than among company the authors acknowledge, ‘the fact that this work was
employees as a whole. There may be a higher (or lower) done within a thirty-mile radius around Pittsburgh will
level of skill required in order to do the jobs that the com- inevitably raise questions about the degree to which the
pany requires its employees to do; there may be more (or findings are applicable in other areas of the country’
less) money in the company’s training budget; there may (1959: 31). The findings may also reflect the values of
be more (or less) of a culture of learning at this company; high individualism, self-interest, and high masculinity
or the company may recruit a higher (or lower) propor- that have been identified as characteristic of American
tion of employees who are already skilled. There may be culture (Hofstede 1984). This is part of the reason that
many other factors too. there have been so many attempts to replicate the study
Similarly, we should be cautious of overgeneralizing in on other occupational groups and in other localities,
terms of locality. A frequent criticism made of research including different cultures and nationalities.
on employee motivation relates to the limited extent to However, there could even be a further limit to gen-
which it can be assumed to be generalizable beyond the eralization that is implied by the Herzberg et al. sample.
confines of the national or regional culture on which The main study was conducted in the late 1950s. One
the study is based. For example, Herzberg et al. (1959) issue that is rarely discussed in this context, and that
conducted semi-structured interviews with 203 accoun- is almost impossible to assess, is whether or not there
tants and engineers in the Pittsburgh area in the USA. is a time limit on the findings that are generated. Quite
Most of the companies that constituted sites for the study aside from the fact that we need to appreciate that
202 9 Sampling in quantitative research

the findings cannot (or at least should not) be gener- twenty-first century. Quite aside from changes that
alized beyond the Pittsburgh area, is there a point at might have occurred naturally, the erosion and virtual
which we have to say, ‘Well, those findings applied to dismantling of the student grant system in the UK and
the Pittsburgh area then, but things have changed and the introduction of tuition fees have changed the ways
we can no longer assume that they apply to that or any students finance their education, including perhaps
other locality’? We are, after all, used to assuming that a greater reliance on part-time work (Lucas 1997), a
changed circumstances will prompt changed behaviour greater reliance on parents, and use of loans. But, even
and attitudes. To take a simple example: no one would when there is no definable or recognizable source of rel-
be prepared to assume that the findings of a study in evant change of this kind, there is nonetheless the pos-
1980 of UK university students’ budgeting and personal sibility (or even likelihood) that findings are temporally
finance habits would apply to students in the early specific.

Error in survey research

We can think of ‘error’, a term that has been employed data collection error. This source of error includes such
on a number of occasions, as being made up of four main factors as poor question wording in self-completion
factors (see Figure 9.9): questionnaires or structured interviews; poor inter-
viewing techniques; and flaws in the administration of
• Sampling error: see Key concept 9.1 for a definition.
research instruments.
This kind of error arises because it is extremely unlikely
that one will end up with a perfectly representative • Finally, there is data processing error. This arises from
sample, even when probability sampling is employed. faulty management of data, in particular errors in the
coding of answers.
• We can distinguish what might be thought of as sam-
pling-related error. This is error that is subsumed under The third and fourth sources of error relate to fac-
the category non-sampling error (see Key concept 9.1) tors that are not associated with sampling and instead
but that arises from activities or events that are related relate much more closely to concerns about the valid-
to the sampling process and are connected with the ity of measurement, which was addressed in Chapter 8.
issue of generalizability or external validity of find- The kinds of steps that need to be taken to keep these
ings. Examples are an inaccurate sampling frame and sources of error to a minimum in the context of social
non-response. survey research will be addressed in the next three
chapters.
• There is also error that is connected with the imple-
mentation of the research process. We might call this

Sampling issues for online surveys

Online surveys have become increasingly popular in • Many people have more than one email address.
recent decades (Chapter 11 discusses them further).
• Email addresses tend to be much more fleeting than
An obvious question to ask is how the material on sam- postal addresses.
pling presented above might apply to them. Certainly,
sampling in online surveys poses some specific chal- • Many people use more than one internet service
provider (ISP).
lenges. A major issue and limitation is that not everyone
in any nation is online, and of those who are, some may • A household may have one computer but several users.
not have the technical ability to handle online question- • Internet users are a biased sample of the population, in
naires. Certain other features of online communications that they tend to be better educated, wealthier,
make the issue more problematic:
 Sampling issues for online surveys 203
younger, and more often male than female (Blasius newsgroup message boards, to suitable mailing lists,
and Brandt 2010). or on web pages and social media. The result will be a
sample of entirely unknown representativeness, and
• Few sampling frames exist of the general online popu-
lation, and most of these are likely to be expensive to it is impossible to know what the response rate to the
acquire, since they are controlled by ISPs or may be questionnaire is, since the size of the population is also
confidential. unknown. On the other hand, given that we have so little
knowledge and understanding of online behaviour and
Such issues make the possibilities of conducting online attitudes relating to online issues, it could reasonably be
surveys using probability sampling principles difficult argued that some information about these areas is a lot
to envisage. This is not to say that online surveys should better than none at all, provided that the limitations of the
not be considered. Indeed, for researchers in the field findings in terms of their generalizability are appreciated.
of business, there may be more opportunities than for More recently, there has been increasing interest in
researchers in other areas. For example, in many orga- the use of online mechanisms for recruiting participants.
nizations, most if not all non-manual workers are likely These include systems for recruiting paid participants, for
to be online and familiar with the details of using email example Amazon Mechanical Turk and Google AdWords,
and the internet, so that a suitable sampling frame of as well as for seeking volunteers, for example on Face-
email addressees is available or can relatively easily be book. It is important to note that such mechanisms gen-
compiled. In such circumstances, surveys can be con- erate convenience samples, rather than random samples.
ducted using essentially the same probability sampling There is little research on their effectiveness in generat-
procedures as those outlined above. For certain kinds of ing representative samples, but one recent study reports
business research, such as investigations involving sur- that volunteer-seeking methods appear to deliver more
veys of organizational members, email-based surveys heterogeneous and thus perhaps more representative
may present sampling problems that differ little from samples than systems which recruit paid participants
offline modes of administration, other than the possibil- (Antoun et al. 2016).
ity of higher levels of non-response. Similarly, surveys of A further issue in relation to online sampling and
members of commercially relevant online groups can be sampling-related error is the matter of non-response
conducted using probability sampling principles. If we (see Key concept 9.3). There is growing evidence that
are interested in collecting data from members of online online surveys typically generate lower response rates
communities, for example those using social networking than postal questionnaire surveys (Tse 1998; Sheehan
technology for business purposes, then virtually by defi- 2001; Pedersen and Nielsen 2016). In the early years,
nition they would need to be contacted online to gener- in the late 1980s, response rates for email surveys were
ate a sample. C. B. Smith (1997) conducted a survey of quite encouraging (Sheehan and Hoy 1999), but since
Web presence providers (people or organizations that the mid-1990s they have been declining to lower lev-
are involved in creating and maintaining Web content). els than those for most postal questionnaires (Sheehan
She acquired her sample from a directory of providers, 2001). Two factors may account for this decline: the
which acted as her sampling frame. novelty of email surveys in the early years and a grow-
As Couper (2000) notes of surveys of populations ing antipathy towards unsolicited emails among online
using probability sampling procedures: communities. A meta-analysis that compared Web sur-
Intra-organizational surveys and those directed at users veys with other modes of survey administration found
of the Internet were among the first to adopt this new that on average, Web surveys produce a response rate
survey technology. These restricted populations typi- that is 11 per cent lower (Lozar Manfreda et al. 2008).
cally have no coverage problems . . . or very high rates However, response rates can be boosted by following
of coverage. Student surveys are a particular example of two simple strategies:
this approach that are growing in popularity.
1. Contact prospective respondents before sending
(2000: 485) them a questionnaire. This is regarded as basic ‘ne-
Hewson and Laurent (2008) suggest that when there is tiquette’. Bosnjak et al. (2008) found that response
no sampling frame, which is normally the case with sam- rates to a web-based panel survey could be enhanced
ples to be drawn from the general population, the main by pre-notifying prospective participants. They found
approach taken to generating an appropriate sample is to that pre-notifications sent by text (SMS) message
post an invitation to answer a questionnaire on relevant were more effective than when sent by email but that
204 9 Sampling in quantitative research

a combination of both was more effective than text of the way through, resulting in unusable partially com-
messages alone. pleted questionnaires in most cases. Interestingly, they
2. As with postal questionnaire surveys, follow up non- also found that respondents were most likely to abandon
respondents at least once. their questionnaires part of the way through when in
the middle of completing a series of open questions. The
3. There is evidence that incentives can increase re-
implications of this finding reflect the advice you will
sponse rates in online surveys (Pedersen and Nielsen
encounter in Chapter 11: that it is probably best to ask
2016).
as few open questions in self-completion questionnaires
The case for the first of these two strategies in boost- as possible.
ing response rates is not entirely clear (Sheehan 2001), Further evidence suggests that having a progress indi-
but seems to be generally advisable. However, as previ- cator in an online survey can reduce the number of people
ously noted, with many online surveys it is impossible to who abandon the questionnaire part of the way through
calculate a response rate, since, when participants are completion (Couper et al. 2001). A progress indicator
recruited through invitations and postings on discussion is usually a diagrammatic representation of how far the
boards, etc., the size of the population of which they are respondent has progressed through the questionnaire at
a sample is almost impossible to determine. any particular point. Couper et al. also found that it took
Crawford et al. (2001) report the results of a survey less time for respondents to complete related items (for
of students at the University of Michigan that experi- example, a series of Likert items) when they appeared
mented with a number of possible influences on the on a screen together than when they appeared singly.
response rate. Students in the sample were initially Respondents also seemed less inclined to omit related
sent an email inviting them to visit the website, which questions when they appeared together on a screen
allowed access, via a password, to the questionnaire. rather than singly.
Some of those emailed were led to expect that the ques- However, it is important not be too sanguine about
tionnaire would take 8–10 minutes to complete (in some of these findings. One difficulty with them is that
fact, it would take considerably longer); others were the samples derive from populations whose members are
led to expect that it would take 20 minutes. As might not as different from one another as would almost cer-
be expected, those led to believe it would take longer tainly be found in samples deriving from general popu-
were less likely to accept the invitation, resulting in a lations. Another is that it must not be forgotten that, as
lower response rate for this group. However, Crawford previously noted, access to the internet is still not univer-
et al. also found that those respondents who were led to sal, and there is evidence that those with internet access
believe that the questionnaire would take only 8–10 min- differ from those without, both in terms of personal char-
utes were more likely to give up on the questionnaire part acteristics and attitudinally.

!
KEY POINTS
● Probability sampling is a mechanism for reducing bias in the selection of samples.
● Ensure you become familiar with key technical terms in the literature on sampling, including
representative sample; random sample; non-response; population; sampling error; etc.
● Randomly selected samples are important because they permit generalizations to the population and
because they have certain known qualities.
● Sampling error decreases as sample size increases.
● Quota samples can provide reasonable alternatives to random samples, but they suffer from some
deficiencies.
● Convenience samples may provide interesting data, but it is crucial to be aware of their limitations in
terms of generalizability.
● Sampling and sampling-related error are just two sources of error in social survey research.
● Online surveys pose specific challenges in terms of sampling.
 Questions for review 205

? QUESTIONS FOR REVIEW

● What does each of the following terms mean: population; probability sampling; non-probability
sampling; sampling frame; representative sample; and sampling and non-sampling error?
● What are the goals of sampling?
● What are the main areas of potential bias in sampling?

Sampling error
● What is the significance of sampling error for achieving a representative sample?

Types of probability sample

● What is probability sampling and why is it important?
● What are the main types of probability sample?
● How far does a stratified random sample offer greater precision than a simple random or systematic
sample?
● If you were conducting an interview survey of around 500 people in a large city, what type of
probability sample would you choose and why?
● A researcher positions herself on a street corner and asks one person in five who walks by to be
interviewed: she continues doing this until she has a sample of 250. How likely is she to achieve a
representative sample?

The qualities of a probability sample

● A researcher is interested in levels of job satisfaction among manual workers in a firm that is
undergoing change. The firm has 1200 manual workers. The researcher selects a simple random
sample of 10 per cent of the population. He measures job satisfaction on a Likert scale comprising
ten items. A high level of satisfaction is scored 5 and a low level is scored 1. The mean job
satisfaction score is 34.3. The standard error of the mean is 8.57. What is the 95 per cent confidence
interval?

Sample size
● What factors would you consider in deciding how large your sample should be when devising a
probability sample?
● What is non-response and why is it important to the question of whether or not you will end up with a
representative sample?

Types of non-probability sampling

● Are non-probability samples useless?
● ‘Quota samples are not true random samples, but in terms of generating a representative sample there
is little difference between them, and this accounts for their widespread use in market research and
opinion polling.’ Discuss.

Limits to generalization
● ‘The problem of generalization to a population is not just to do with the matter of getting a
representative sample.’ Discuss.