Research Design and quality

Prof Dave Root and Dr Ria Vosloo

BUQS4041A/4031A
27 May 2022 1
 Content
• Empirical questions (from assignment 2)
• Purpose of Research design
• Methodological coherence
• Possible designs
• Mixed and multi - designs
• Basic data collection techniques

2
 The U Model- Overview
The Problem Area Contribute
!
The
Research
Problem

Research Answer The

Question research Question
Structured
Research Analyze and
Question synthesize the
results

Reality
Detailed Literature HOW TO
Review EXTRACT
EMPIRICAL
EVIDENCE
3
 Supporting the argument
Contribute
The Problem Area !

The
Research
Problem

Research
Question
Structured
Research !
Question

Reality
Detailed Literature HOW TO
Review EXTRACT
EMPIRICAL
EVIDENCE
4
 Types of Questions - Empirical
Question Type Questions
Exploratory Questions What is the case?
What are the key factors?
Descriptive Questions How many?
What is the incidence of X?
Are x and y related?
Causal Questions Why?
What are the causes of X?
Evaluative Questions What was the outcome of?
Has P been successful?
Predictive Questions What will the effect of x be on y?
Historical Questions What led to y happening?

Babbie and Mouton, 20015

 Types of Questions – Non Empirical

Question Type Questions

Meta analytical What is the state of the art regarding x?
questions? What are the key debates regarding y?
Conceptual What is the meaning of concept w?
questions?
Theoretical What are the most plausible theories/ models
questions? of z?
What is the most convincing explanations of
y?
Philosophical/ What is the ideal profile of z?
normative What is meant by m?
questions? Do q have d?
Babbie and Mouton, 20016
 Questions that you asked in analysis
• Is it qualitative or quantitative , what paradigm?
• Are there ethical concerns and what are they?
• What is the theoretical grounding? (specific
Literature)? (summarize to Lit Review topics)
• What sources of data/ evidence exist?
• Do I have ethical access to that data/evidence?
• Are there validated and reliable instruments?
• Do you have or can you acquire the skills set to
answer this question)
7
 Where is the evidence
Primary Data
Surveys/
Laboratory
comparative
Experiments Field studies
experiments Ethnographic
studies
PAR
Programme
evaluation Content
High analysis/ Low
Control historical Control

Discourse
Secondary data analysis
analysis / statistical
modelling

Secondary Data Babbie and Mouton, 20018

 Research Design Choice and
Coherence
• What is a ‘Research Design’?
• A general plan of how you will go about answering
your research questions
• Recipe or road map?
• Clear objectives
• Sources of data
• How you will collect and analyse the data
• Ethical issues around the above
• Constraints you will encounter
• Demonstration that you have thought through your
research design
9
 The Research ‘Onion’
Strategic choice

Qualitative Mono method

Survey Qualitative
Experiment Archival Research Mixed method
Qualitative
Cross sectional Case Study
Mixed
Methods
research
Longitudinal
Action Mixed method
Strategies
(Designs) Narrative Ethnography Research Quantitative
Grounded
enquiry Mono method
Research
Quantitative
Quantitative
Methodological Saunders et al
choice 10
 The Research ‘Onion’ – Outer Layers
Philosophy

Positivism
Induction Realism

Abduction
Interpretivism

Deduction Pragmatism
Approach
Saunders et al
11
 • The ‘outer layers’ of the Onion
Strategic Choice
• Concerned about the research philosophy
• Types of data or philosophical beliefs about the
nature of knowledge? (epistemological position?)
• The high level decisions in how to approach the
research (framework) that determines the
detailed work to be undertaken.
• Often thought to be an expression of the
researcher’s basic philosophical beliefs
(epistemology) and understanding of the research
to be undertaken

Saunders et al
• 12
 Strategic Choices
• Deduction
• Top down
• Conclusion follows from premises (available facts)
• conclusion guaranteed
• Induction
• Bottom up
• Likely based on premises
• Involves uncertainty
• conclusion merely likely
• Abduction
• taking your best shot
• Most likely explanation
 The Research ‘Onion’
The next layers
Qualitative Mono method
Survey Qualitative
Experiment Archival Research Mixed method
Qualitative
Case Study
Mixed
Methods
Strategies research
(Designs) Action Mixed method
Narrative Ethnography Research Quantitative
Grounded
enquiry Mono method
Research
Quantitative
Quantitative
Methodological
choice Saunders et al
14
 Tactical Choices

Decisions on the approach(s) adopted to

collect evidence (data) and analysing that
evidence, which can be used for
understanding a phenomenon thus
contributing to the body of knowledge

Saunders et al 15
 Tactical Decisions

• Decisions around the selection (and defence of

that choice) of the primary set of tools used by a
researcher to explore the issues of interest and to
attempt to answer the research question
• Methodological Choices

Saunders et al 2011

16
Research Methodology and Methods

• Methodology refers to the procedural framework

within which the research is conducted (strategies
of ‘the onion’)
• Method refers to the various ways in which data
can be collected and analysed (techniques and
procedures of ‘the onion’)

Saunders et al 2011

17
 Quantitative or Qualitative

• Each type of data implies certain beliefs and

assumptions about how we make sense of the
world
• Coherence comes by making a choice that aligns
with how we feel comfortable about ‘reality’
• Quantitative – implies a positivist epistemological
position (positivism)
• Qualitative – implies an interpretivist
epistemological position (interpretivism)

Saunders et al
18
 Five main purposes Mixed Method
(Greene, 2007)
• Expansion: different methods assess different
phenomena to expand scope and range of study
(address insufficient information – Creswell, 2008)
• Triangulation: measure same phenomena using
different methods to increase confidence in the
conclusions reached (more evidence the better and
multiple angles – Creswell, 2008)

Urmilla Bob 19
 Five main purposes Mixed Method
(Greene, 2007)
• Complementarity and initiation: methods used to
investigate different aspects or dimensions of same
phenomena. Initiation entails using the information to
ask “I wonder why?” questions
• Development: results from one method informs the
development of another methods

Key principle: A mixture or combination of methods that

has complementary strengths and non-overlapping
weaknesses.

Urmilla Bob 20
 Choosing the Research Tactics

• Consider carefully the determining issues and

understand the alternatives available

Saunders et al 2011

21
 Laboratory Experiments

• Used in physical, life sciences and management

• Limited application to management type research
• Allows for controlled environment
• Possible to replicate

22
 Design of Experiments
• A statistics-based approach to design
experiments
• A methodology to achieve a predictive
knowledge of a complex, multi-variable
process with the fewest acceptable trials.
• An optimization of the experimental process
itself

23
 Field Experiments

• Example of Hawthorne Exp. (Elton Mayo)

• Limits to type of questions that can be asked
• Less positivistic because of lack of controlled
conditions and ability to replicate conditions
• Results often interpreted in a more qualitative
fashion

24
 The Large-Scale Survey

• Administering questionnaires
• Opportunities for large quantities of data
• Often used for opinions, attitudes and hard facts
• Strictly positivistic logic

25
 Forecasting

• Associated with mathematical and statistical

techniques of time-series and regression analysis
• Allow projections on the basis of historical
evidence
• Highly quantitative approach
• Results may be interpreted qualitatively

26
Simulation and Stochastic Modelling

• Requires a strong mathematical backgrounds

• Used for investigating situations which do not
lend themselves to strictly deterministic or
analytical treatment

27
 Game or Role-Playing

• Popular in Human relations and organisational

behaviour
• A high level simulation of inter-personal reactions
or group decision making
• Individuals are asked to participate in a business
game by playing out a specific role

28
 In-depth Surveys

• Relatively small sample size

• Structure/unstructured interviews
• Note taking or recorded interviews
• Generation of transcripts
• Analysis of transcripts can be positivist or
Interpretivist

29
 Action Research
• Increasingly popular approach (education, medical
practice)
• Allows for co-operative inquiry
• Particularly suited to managing change
• Incorporates feedback from observations into the
system under study
• Based around the notion of intervention in a
system or situation
• Very demanding but very rewarding

30
 Participatory Action Research
• PAR is a cyclical, reflective process where participation occurs:
– Problem formulation
– Planning initial design of project
– Implementation and monitoring the project
– Reaching final conclusions, creating new meaning and
generating new knowledge
– Communication of results
– Review and reflection
– Generating of problem solutions and applying research
outcomes
– Assessment of results
– Validation of findings
31
 Evaluation study
Purpose of evaluation:
• Judgement
• Improvement orientation
• Knowledge orientation

What to evaluate:
• Need
• Process
• Outcome
• Efficiency
• Theoretical assumptions

Babbie and Mouton, 2001, Rossi and Freeman 2004 32

 Futures Research

• A way of considering and developing predictions

e.g. Clem Sunter’s speciality
• Looks ahead, not backwards
• Uses techniques such as scenario projections and
Delphi studies
• Involves the summation of opinions form experts
as well as attempting to draw expert opinions
towards a group consensus

33
 Focus Groups
• Homogeneous group of people
• Explore issues through group discussion
• Role of moderator
• Size and composition of group
• Uses
• Costs
• Advantages/Disadvantages

34
 Case Study
• Arose from the need to understand and explain complex
phenomena
• Can be used for theory generation
• Deal with unique situations
• Selecting the case – compelling case or theoretical choice
• Defining the unit of study is key
• Single cases Multiple cases or imbedded cases
• Suited to mixed methodology
• Allows for several methods of data collection – multiple data
sources:
– Documents
– Interviews
– Archives
– Observation 35
 Ethnography

• Requires researcher to become part of the ‘tribe’

and fully participate in its society
• Used to look at socially constructed meaning and
culture
• Words study and learning used
• Also used to describe study of a particular society
(But disputed – Spradley 1979)
• Interviewing and data gathering techniques are
used

36
 Scenario Research
• Similar to game or role playing
• Involves collecting evidence forma group of
suitably qualified experts who are asked to discuss
the implications of a particular hypothetical setting
• Evidence collected ad processed in much the same
way as in-depth surveys

37
 Types of Mixed Methods Designs
Concurrent Mixed Methods Designs
Convergent Parallel Design
QUAN QUAL
Data & Data &
Interpretation
Results Results

Embedded Design

QUAN Intervention QUAN

Pre-test Post-test
Interpretation
Data & Data &
qual
Results Results
Process

38
John W. Creswell 2010
 Sequential Designs
Explanatory Design
QUAN qual
Data & Data &
Interpretation
Results Results
Following up

Exploratory Design
QUAL quan
Data & Data &
Results Building to
Results Interpretation

Sequential Embedded Design

Before-
QUAN After-
intervention Intervention
Trial intervention
qual qual Interpretation

John W. Creswell 2010 39

 Emancipatory Theory Design
A Framework
(Participatoryaction research, community health research, critical theory, case
study, narrative study, experimental study, Design science etc. )

QUAL quan
Data & Data &
Results Results
Building to Interpreta
tion
OR
QUAN QUAL
Data & Data &
Results Interpretation Results

40

John W. Creswell 2010

 Multiphase (or Multi-project) Design

QUAN QUAL
Data & Results Data & Results

Interpretation

QUAL quan
Data & Data &
Results Results Interpretation
Building to

John W. Creswell 2010 41

 Choosing Tactics

• Choose carefully
• Pick one that suits you
• Must answer your question (change the
question?)
• Ensure strategy and tactics align
• Consider triangulation
• Avoid contradictions
• Ensure it is ‘doable’
• Make sure your supervisor agrees!

42
 Time Horizon

• Cross-sectional
• Longitudinal

43
 The Research ‘Onion’
The Inside

Operational choices

Cross sectional

Longitudinal
Time Horizon

Saunders et al
44
 Cross-sectional Design

• Evidence collected on more than one case at a a

single point in time on two or more characteristics
(variables) of the phenomenon under study.
• Data can be qualitative or quantitative
• Objective is to detect and describe “correlations”

45
 Longitudinal Design

• Used to describe change

• Time series of data
• Sample is surveyed on more than one occasion
• Problems do arise due to loss of cases

46
 Operational Choices
• Techniques and procedure
• Data collection and Data Analysis

Saunders et al 2011

47
 Collection types
Type Options Advantages Disadvantages
Observations • Complete • First hand • Researcher
participant experience experienced as
(concealed role) • Can record on intrusive
• Observer as going • Private
participant • Unusual information
(known role) aspects noted observed that
• Participant as • Exploring topics can not be
observer that may be reported
(participant role uncomfortable • Some
stronger) to discuss participants
• Complete (e.g. children)
observer problems with
rapport
John W. Creswell 2014 48
 Collection types
Type Options Advantages Disadvantages
Interviews • Face to face, one • Useful when • Info filtered
on one observation not through views
• Telephone possible of interviews
• Group • Can provide • Designated
• Focus group historical place and not
• Email/ electronic information in natural
• Allows some (field) situation
control to • Researcher
researcher presence
(varies) might bias
response
• people not
equally
articulate and
John W. Creswell 2014 perceptive 49
 Collection types
Type Options Advantages Disadvantages
Documents • Public • Obtain words and • people not
documents language of equally
• Private participants articulate and
documents • Accessed at perceptive
convenient time • Protected/
• Unobtrusive access
• Participants gave • May be
attention when difficult to get
recorded to place
• Save time/cost of • Incomplete
ranscribing • May not be
authentic or
accurate
John W. Creswell 2014 50
 Collection types
Type Options Advantages Disadvantages
Audio- • Photographs • May be • May be difficult to
visual • Videos unobtrusive interpret
• Art objects • Provide • May not be accessible
• Artefacts opportunity to publically or privately
• Computer share reality • Presence of recorder
messages directly disruptive or influence
• Sounds • Attention visually
• film

51
John W. Creswell 2014
 Some Thoughts

• Don’t try to be fancy

• Pick something that you are intuitively
comfortable with
• Pick something that minimises the practical
constraints on collecting data:
– Look at the data you’ve got
– Look at the data you can get
– Use that to inform you choices

52
 What must be specified in a research
design
Techniques and procedures:
•Research protocol (including ethical aspects)
•Population identification
•Sampling procedure
•Data collection procedure
•Data collection instruments
•Data capturing
•Data analysis

Based on Saunders 2007

53
 Quality in Research Design

• This is a critical aspect of your research:

• The “three gods “of research:
• Some thoughts:
– Validity
– Reliability
– Generalize-ability/ transferability
OR
– Trustworthiness (Gupa)

54
 Reliability -What do we mean

Reliable – Not Valid– Not Valid and

Valid Reliable Reliable

Babbie and Mouton, 2001 55

 Reliability - Quantitative
The extent to which
something produces
consistent results

Research Measurements
studies

Inter-observer/ Questionnaires
Inter rater

Test-retest Internal
consistency

Split Halves Cronbachs

56 α
 How to we achieve reliable results

There are two issues-

1. Is the study reliable?
a. Your research protocol and the consistency in applying this
affect the reliability
2. Is your instrument reliable
a. Has it been used and what has been proven
b. If two people apply it (interview, observations etc.) to they
get different results
c. Questionnaires- if a person repeats the questionnaire- is
the result the same
d. In the instrument (only if it measures one
concept – Cronbach's alpha etc! 57
 Validity

• Validity is an over-arching concern of research

– Measurement – are the observations directly and truly
linked to the dimension or concept claimed?
– Research design – how well does the experiment or study
control the situation so that we are confident that the
relationships or results observed were due to the impact
of the independent variable?

• Operationalization – The entire procedure used to

produce the measurement. This includes any
instrument, but also instructions for its use.
58
 Validity - Quantitative
The extent to which
something measures what
it purports to measure

Research Measurements
studies

Face Predictive Concurrent Construct

Internal External

Ecological Sampling
59
 Sampling and selection
• In quantitative work and in the case where we
can use random sampling – predictive

60
 Major Threats to Validity

• Type I error: believing a principle to be

true when it is not (i.e., mistakenly
rejecting the null hypothesis)
• Type II error: rejecting a principle when,
in fact, it is true
• Type III error: asking the wrong
question

61
 Sources of invalidity
• Incorrect theory

• Measurement doesn’t match intention.

• Many interpretations of meaning of the

measurement.

62
 Qualitative Research
• In qualitative research, prior elimination of
threats to validity (by design e.g. sampling
and controls) is less possible because:
– qualitative research is more inductive, and
– it focuses primarily on understanding
particulars rather than generalizing to
universals.
– Random sampling is not really possible

63
Quality in Qualitative Research Design

• Validity
• Reliability
• generalize-ability/ transferability
• Trustworthiness

RIGOR IS IN THE PROCESS

There are serious conflicts in academic opinion
regarding what concepts to use!

64
 In Qualitative Research
(using traditional quantitative terms)
Tests Tactics Phase
Construct Multiple sources of evidence Data collection
Validity Establish chain of evidence Data collection
Review draft CS report Composition
Internal Pattern matching Data analysis
Validity Explanation building Data analysis
Adress rival explanations Data analysis
Use logic models Data analysis
External Use theory in single-case studies Research design
Validity Use replication logic in multiple case Research design

Reliability Use case study protocols Data collection

Develop case study database Data collection
R.Yin, 2009 65
Other Methods of Evaluating Qualitative
Research Quality
• Lincoln and Guba’s classic work shed light
on how to assess truth in a qualitative
report
• Offered four alternate tests of quality that
reflect the assumptions of the qualitative
paradigm:
– Credibility
– Dependability
– Transferability
– Confirmability
66
 Credibility
• Refers to accuracy
– Description must be plausible and recognized by
participants
• Enhanced by:
– Prolonged time in the field repeatedly observing
and interacting with participants
– Using different data sources, methods, data type
• Conducting member checks
– Involving other investigators in the study

67
 Some Strategies
• Credibility
– Prolonged and varied field experience
– Time sampling
– Reflexivity (field journal)
– Triangulation
– Member checking
– Peer examination
– interview technique
– Establishing authority of researcher
– Structural coherence
– Referential adequacy
– Rival explanations
68
 Dependability
• Refers to the stability and trackability of
the changes in data over time and
conditions
– Want to determine the extent to which
another researcher with similar training and
rapport with participants would make the
same observations
– This is determined by an audit trail
– Involves auditing research process,
documenting all the raw data generated, and
assessing method of data analysis
69
 Some Strategies

• Dependability
– Dependability audit
– Dense description of research methods
– Stepwise replication
– Triangulation
– Peer examination
– Code-recode procedure

70
 Transferability

• Refers to the generalizability of the study

findings to other settings, populations, and
contexts
– Report must provide sufficient detail so that
readers can assess this
– Lack of transferability is viewed as a weakness
of qualitative methods
– Selection (not sampling) is a key here

71
 Some Strategies

• Transferability
– Nominated sample
– Comparison of sample to demographic data
– Time sample
– Dense description

72
 Confirmability

• Refers to the objectivity of the data

– Would another researcher agree about the
meanings emerging from the data
– An audit trail is used in which the researcher
explicates how personal biases may have
come into play

73
 Some Strategies

• Confirmability
– Conformability audit
– Triangulation
– Reflexivity

74
Key Qualitative Research References
• Babbie, E. and Mouton, J. (2001) The Practice of Social
Research. Oxford University Press: South Africa.
• Bryman. A. (2006) Integrating quantitative and qualitative
research: how is it done?, Qualitative Research 6(1). Sage:
Thousand Oaks, CA
• Creswell, J.W. (2014) Research Design. 4th edition. Sage: Los
Angeles
• Eisenhardt, K.M. (1989) Building Theories from Case Study
Research, Academy of Management Review 14(4): 532-550.
• Eisenhardt, K.M. (1991) Better Stories and Better Constructs:
The Case for Rigor and Comparative Logic, Academy of
Management Review 16(3): 620-627.
• Lee, T.W. (1999) Using Qualitative Methods in Organizational
Research, Sage: London
• Marsland, N. et al. Methodological Framework for
combining quantitative and qualitative Survey
Methods. Collaborative report. 75
Key Qualitative Research References
• Rossi, P.H., Lipsey, M.W. and Freeman. H.E. (2004)
Evaluation- A Systematic Approach. 7th edition. Sage: London
• Yin, R. (2013) Case Study Research. Design and Methods. 5th
edition. Sage: London.
• Yin, R. (2011) Qualitative Research from Start to Finish. The
Guilford Press: London.
• Yin, R. (2006) Mixed Methods Research -Are the methods
Genuinely integrated or merely parallel, Research in the
Science 13(1) Mid South Educational Research Association
• Whitehead, T. L. (2005) Basic Classical Ethnographic
Research Methods. EICCARS Working Paper Series
• Corbin, J. and Strauss, A. (2008) Basics of qualitative
research. 3rd edition. Sage: London. (Grounded theory)

General Research text book: Research Methods for

Business Students, Saunders et al (from edition 5 will
work) 76
Questions?