Agenda

 2 Main QA approaches
 Coding exercise 1
 Coding exercise 2
 Slides on Qualitative Analysis
 Brainstorming Exercise (if time)
 Affinity Diagramming Exercise (if time)

2
Qualitative Research: Common Features
of Analytic Methods (Miles &
Huberman,1994)
1 Affixing codes to a set of field notes drawn
from data collection
2 Noting reflections or other remarks in margin
3 Sorting or shifting through the materials to
identify similar phrases, relationships
between themes, distinct differences
between subgroups and common sequences
Qualitative Research: Common Features
of Analytic Methods (Miles &
Huberman,1994)
4 Isolating patterns and processes, commonalties
and differences, and taking them out to the
field in the next wave of data collection
5 Gradually elaborating a small set of
generalisations that cover the consistencies
discerned in the data base
6 Confronting those generalisations with a
formalised body of knowledge in the form of
constructs or theories
2 general research approaches

deductive approach vs inductive approach

5
deductive research approach

THEORY Top-down approach

Theory testing

A priori codes
HYPOTHESIS

OBSERVATION

CONFIRMATION
6
inductive research approach

THEORY

TENTATIVE HYPOTHESIS

PATTERN bottom-up approach

Theory building

OBSERVATION Emergent codes

7
deductive or inductive

8
Often use a hybrid approach

 A set of a priori codes reflecting your

understanding of the topic and your research
questions
 Emergent codes added as you code the data
and find other factors/topics/codes that you
had not considered

9
Exercise 1

 Open coding
 Inductive analysis
 Exploratory research
 Theory building research

 http://b.socrative.com/login/student/
 Room: 7f156b7b

10
Exercise 2

 Coding with pre-defined categories

 Deductive analysis
 Theory Testing

 http://b.socrative.com/login/student/
 Room: 7f156b7b

11
 Qualitative Inquiry - Purpose

The purpose of qualitative inquiry is to produce

findings. The Data Collection process is not an
end in itself. The culminating activities of
qualitative inquiry are analysis, interpretation,
and presentation of findings.
 Qualitative Inquiry -
Challenge
To make sense of massive amounts of data,
reduce the volume of information, identify
significant patterns and construct a framework
for communicating the essence of what the
data reveal
 Qualitative Inquiry - Problem

‘…have few agreed-on canons for qualitative

data analysis, in the sense of shared ground
rules for drawing conclusions and verifying
sturdiness’ (Miles and Huberman, 1984)
The Creativity of Qualitative
Inquiry
 ‘..the human element of qualitative inquiry is both
its strength and weakness - its strength is fully
using human insight and experience, its weakness
is being so heavily dependent on the researcher’s
skill, training, intellect, discipline, and creativity.
The researcher is the instrument of qualitative
inquiry, so the quality of the research depends
heavily on the qualities of that human being’
(Patton, 1988)
The Science and Art of
Qualitative Inquiry (Patton,
1988)  The Science
The scientific part is systematic, analytical,
rigorous, disciplined, and critical in perspective
 The Art
The artistic part is exploring, playful,
metaphorical, insightful, and creative
1. Analysis Considerations
1 Words
2 Context (tone and inflection)
3 Internal consistency (opinion shifts during groups)
4 Frequency and intensity of comments (counting,
content analysis)
5 Specificity
6 Trends/themes
7 Iteration (data collection and analysis is an iterative
process moving back and forth)
2. The Procedures
1 Coding/indexing
2 Categorisation
3 Abstraction
4 Comparison
5 Dimensionalisation (relationships)
6 Integration
7 Iteration
8 Refutation (subjecting inferences to scrutiny)
9 Interpretation (grasp of meaning - difficult to describe
procedurally)
 Critical Thinking
‘Critical Thinking calls for a persistent effort to
examine any belief or supposed form of
knowledge in the light of the evidence that
supports it and the further conclusions to
which it tends’ (Glaser, 1941)
or more simply!
Critical Thinking means weighting up the
arguments and evidence for and against.
 Critical Thinking
• Key points (Glaser, 1941):
– Persistence: Considering an issue carefully
and more than once
– Evidence: Evaluating the evidence put
forward in support of the belief or viewpoint
– Implications: Considering where the belief
or viewpoint leads; what conclusions would
follow; are these suitable and rational; and
if not, should the belief or viewpoint be
reconsidered
 Guidance for Creative Thinking
1 Be open
2 Generate options
3 Divergence before convergence
4 Use multiple stimuli - triangulate
5 Side track, zig-zag, and circumnavigate
6 Change patterns of thinking
7 Make linkages
8 Trust yourself
9 Work and play at it
 The Credibility of Qualitative
Analysis
1 Rigorous techniques and methods for gathering high-
quality data that is carefully analysed, with attention
to issues of validity, reliability, and triangulation
2 The credibility of the researcher, which is dependent
on training, experience, track record, status, and
presentation of self
3 Philosophical belief in the phenomenological
paradigm, that is, a fundamental appreciation of
naturalistic inquiry, qualitative methods, inductive
analysis and holistic thinking
 A Credible Qualitative Study
A credible qualitative study needs to address
the following issues:
1 What techniques and methods were used to
ensure the integrity, validity, and accuracy
of the findings
2 What does the researcher bring to study in
terms of qualifications, experience, and
perspective
3 What paradigm orientation and
assumptions ground the study
Principles of Analysing
Qualitative Data
1 Proceed systematically and rigorously (minimise human
error)
2 Record process, memos, journals, etc.
3 Focus on responding to research questions
4 Appropriate level of interpretation appropriate for
situation
5 Time (process of inquiry and analysis are often
simultaneous)
6 Seek to explain or enlighten
7 Evolutionary/emerging
 Inter-rater reliability

 What if you have more than one person

coding?
 How much agreement do they have?
 At what point should you test their agreement?
 Other than comparing counts, how can you
validate the coding/analysis?
 https://www.academia.edu/458025/The_place_of_inter-rater
_reliability_in_qualitative_research_an_empirical_study

26
 Interface Design and Usability Engineering
Articulate: Brainstorm Refined Completed
•who users are designs designs designs
Goals: •their key tasks

Task
Psychology of Graphical
centered Participatory
everyday screen
system interaction
Evaluate things design Usability Field
design
User Task / Interface testing testing
Methods: Participatory Cognitive guidelines
design involvement
scenario
Representation walk-through Style Heuristic
User-
& metaphors guides evaluation
centered
design

low fidelity high fidelity

prototyping prototyping
methods methods

Products: User and Throw-away Testable Alpha/beta

task paper prototypes systems or
descriptions prototypes complete
specification
27
 brainstorming
 the point is:
 to generate MANY, WIDE-RANGING ideas
 nutty and absurd are GOOD. go for the extremes
(to get out of the rut)
 riff off other’s ideas.

 the point is NOT:

 to generate excellent, complete, feasible ideas
… pressure stifles
 to develop or critique ideas
… go wide. deep is for later.
process

1. prepare a list of topics / questions

ahead of time; or in a preliminary brainstorm
2. facilitator takes team through list of topics
switch topic when energy ramps down

3. Note taker takes notes (very important)

4. switch roles so everyone can play
5. ground rules
6. Follow up
brainstorming is like
popcorn
ground rules
 Postpone and withhold your judgment of ideas:
never criticize
 Encourage wild and exaggerated ideas
 Quantity counts at this stage, not quality
 Switch topics when the popcorn slows down
 Build on the ideas put forward by others
 Every person and every idea has equal worth
 Elect a facilitator (calls switches) and a note-
taker (one thought per post it!)
Post brain-storm
 collect the notes

 go through carefully, with judgment turned on

 look for
 interesting, surprising ideas that might work
 ideas that will combine well
 promising directions on which you should brainstorm
more

 keep your notes. at a later design stage, come back to

them and see if anything else has become useful in the
meantime.
Sometimes you have a lot of
ideas to make sense of!

33
 work consolidation:
abstracting specific insights
 one tool: the affinity diagram

 can use to “consolidate” insights from collected or

generated data. for example:
 brainstorming about design problems
 categories of problems
 brainstorming about design ideas
 categories of ideas
 comments from users
 categories of desirable / successful features
 how do you make an affinity
diagram?
1. team writes down all data & insights on post-it notes;
be sure you can link the post-it detail back to its source!
2. stick one post-it on the wall
a whiteboard or big sheet of butcher paper is best
3. arrange the other post-its around it, grouping by affinity
to each other. iteration will be required.
4. look at each group and see what it has in common;
name and describe each group.
5. “snapshot” the result for documentation
1. digital photo  your design website or notebook
2. transfer post-its onto paper, 1 sheet / notes-cluster
 scan  website
 why does an affinity diagram work?
• use physical arrangement/proximity to understand
connections
• openness to serendipity
• low cost to rearrange ideas
• many variants:
 arrange along axes rather than by affinity
 tie causes to effects
 group evidence under assertions
Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and
Konstantin Beznosov. 2008. Guidelines for designing IT security
management tools. In Proceedings of the 2nd ACM Symposium on
Computer Human Interaction for Management of Information
Technology (CHiMiT '08). ACM, New York, NY, USA, , Article 7 , 10 pages.
DOI=10.1145/1477973.1477983
http://doi.acm.org/10.1145/1477973.1477983

38
 Methodology (Phase I)

• Field study: • Field studies

• Interviews • Interviews
• Participatory observation • Questionnaires
• Prototyping
• Cognitive walkthroughs
• Surveying other literature

39
Methodology (Phase I)

Categorized
List of
Guidelines

40
High level Category

Low level Category

Guideline

Guideline ID
number

41
Methodology (Phase I)

Guidelines
Framework42
Methodology (Phase II)

43
Framework for classification of guidelines
Task Specific
Configuration and
Intensive Analysis
Deployment

Organizational Complexity
Diverse

Specificity
Distributed ITSM
Stakeholders
Communication

Technological Complexity

General Usability Guidelines

44
 Framework for classification of guidelines
Task Specific Guidelines
Configuration and Deployment Guidelines Intensive Analysis Guidelines
Make configuration manageable [3,20] Provide customizable alerting [20]
Support rehearsal and planning [3,6,7,20,44] Provide automatic detection [26,41]
Make configuration easy to change [20,46] Provide data correlation and filtering [1,26]
Provide meaningful errors [20, 34,46]

Organizational Complexity Guidelines

More Specific
Diverse Stakeholders Guidelines Communication Guidelines Distributed ITSM Guidelines
Provide flexible reporting [9,18,33,35] Provide communication Support collaboration [6,7,20]
Provide an appropriate UI for integration [6,7,28,45] Work in a large workflow [8,9,20]
stakeholders [9,35] Facilitate archiving [17,21]

Technological Complexity Guidelines

Make tools combinable [8,9,20,26] Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45]

Help task prioritization [15,44] Use different presentation / interaction methods [1,4,5,29,41,48,49]

Provide customizability [9,33] Support knowledge sharing [9,12,14,27,32,37,47]

General Usability Guidelines

45
Class will be 1 big group
3 volunteer note takers

 Problem:
How to design the user interface for a
car proximity detection system

Brainstorm 3 aspects of the problem:

(e.g., physical form factor, safety issues,
input techniques, etc.)
 go: 5 minutes
affinity diagram exercise

Now take your notes from the earlier

brainstorming and create an affinity diagram

go: 8 minutes
debrief