DATA GATHERING

Usman Ahmad.
 Five key issues
1. Setting goals
 Decide how to analyze data once collected

2. Identifying participants
 Decide from whom to gather data
 How many participants are needed

3. Relationship with participants

 Clear and professional
 Informed consent when appropriate

4. Triangulation
 Look at data from more than one perspective
 Collect more than one type of data, for instance, qualitative data from experiments and
qualitative data from interviews

5. Pilot studies
 Small trial of main study
 A pilot study is a research study conducted before the intended study.
2
Data Collection Techniques

 Interviews
 Questionnaires
 Observation
 Choosing and combining techniques
 Interviews
Unstructured interviews

 This is an interview which does not use any set of

questions.
 Instead the interviewer has a topic area and talks
to the interviewee in a conversational way.
 Open ended questions
 Produces qualitative data

4
 Interviews
Structured interviews

 A structured interview is where the interviewer

has a pre prepared set of questions (in the form
of a questionnaire) which he/she reads out to the
interviewee.
 They are not allowed to move away from the set
questions except to clarify the meaning of the
question.

5
 Interviews
Semi-Structured interviews

 Structured interviews are lists of questions which

are focused on strictly.
 Whereas semi structured interviews have lists of
questions or areas of interest that the
interviewer uses to direct the interview but the
interviewer can ask other questions.

6
 Interviews
Focus groups or Group Interview

 The interviewer usually meets 8-10 people at one

go.
 A type of group interview where the group
discusses various pre-set issues.

7
 Interview questions
 Two types:
 ‘Closed questions’ have a predetermined answer format, for
example, ‘yes’ or ‘no’
 ‘Open questions’ do not have a predetermined format
 Closed questions are easier to analyze
 Avoid:
 Long questions
 Compound sentences — split them into two
 Jargon and language that the interviewee may not understand
 Leading questions that make assumptions, for example, why
do you like …?
 Unconscious biases, for instance, gender stereotypes

8
 Running the interview
Introduction: Introduce yourself, explain the goals of
the interview, reassure about the ethical issues, ask to
record, and present the informed agreement form.
Warm-up: Make first questions easy and non-
threatening.
Main body: Present questions in a logical order
A cool-off period: Include a few easy questions to
defuse tension at the end
Closure: Thank interviewee, signal the end, for
example, switch recorder off.
9
 Other forms of interviews

Digital conferencing systems such as Skype, Zoom,

email, and smartphones can be used to conduct
interviews. Some advantages are:
 Participants are in their own environment so are
more relaxed
 Participants don’t need to travel
 Participants don’t need to worry about what to wear
 For interviews involving sensitive issues, it is easier
for interviewees to be anonymous

New 10
 Enriching the interview process
Props: Devices for prompting interviewee, for example, use
a prototype, scenario

11
 Contextual Inquiry
 An approach to ethnographic study. Often conducted as an
internship where user is expert, designer is beginner
 A form of interview, but
 at users’ workplace (workstation)
 2 to 3 hours long
 Four main principles:
 Context: see workplace & what happens
 Partnership: user and developer collaborate
 Interpretation: observations interpreted by user and
developer together
 Focus: project focus to understand what to look for

12
 Questionnaires
 Questions can be closed or open
 Closed questions are easier to analyze, and
may be distributed and analyzed by computer
 They can be managed to large populations
 Circulated by paper, email and the web
 Sampling can be a problem when the size of
a population is unknown as is common online
evaluation
13
 Questionnaire design
 The impact of a question can be influenced by
question order.
 You may need different versions of the
questionnaire for different populations.
 Provide clear instructions on how to complete the
questionnaire.
 Strike a balance between using white space and
keeping the questionnaire compact.
 Avoid very long questions and questionnaires
 Decide on whether phrases will all be positive, all
negative, or mixed.
14
 Question and response format
 ‘Yes’ and ‘No’ checkboxes
 Checkboxes that offer many options
 Rating scales
 Likert scales
 Semantic scales
 3, 5, 7 or more points

 Open-ended responses 15
 Encouraging a good response
 Make sure that the purpose of study is clear
 Promise anonymity
 Ensure that questionnaire is well designed
 Offer a short version for those who do not have
time to complete a long questionnaire
 If mailed, include a stamped, addressed envelope
 Follow-up with emails, phone calls, or letters
 Provide an incentive
 40 percent response rate is good, 20 percent is
often acceptable
16
Advantages of online questionnaires

 Relatively easy and quick to distribute

 Responses are usually received quickly
 No copying and postage costs
 Data can be collected in database for
analysis
 Time required for data analysis is reduced
 Errors can be corrected easily
17
Example of an online questionnaire

Questionnaire shows check boxes, radio boxes, and pull-down menus

18
Problems with online questionnaires
 Sampling is problematic if population size
is unknown
 Preventing individuals from responding
more than once can be a problem
 Individuals have also been known to
change questions in email questionnaires

19
Deploying online questionnaires
 Plan the timeline
 Design offline
 Program/complete online template
 Test the survey to make sure that it behaves
as you would expect
 Test it with a group that will not be part of the
survey to check that the questions are clear
 Recruit participants
20
 Observation
 Direct observation
 Structuring frameworks
 Degree of participation (insider or outsider)
 Ethnography
 Used in controlled environments

 Indirect observation: tracking users’ activities

 Diaries
 Interaction logging
 Video and photographs collected remotely by drones or
other equipment
21
Structuring frameworks to guide observation
 Three easy-to-remember parts:
 The person: Who?
 The place: Where?
 The thing: What?
 A more detailed framework (Robson, 2014):
 Space: What is the physical space like and how is it laid out?
 Actors: What are the names and relevant details of the people
involved?
 Activities: What are the actors doing and why?
 Objects: What physical objects are present, such as furniture
 Acts: What are specific individual actions?
 Events: What do you observe in a special event?
 Time: What is the sequence of events?
 Goals: What are the actors trying to accomplish?
 Feelings: What is the mood of the group and of individuals?
22
 Planning and conducting observation in
the field
 Decide on how involved you will be: from
passive observer to active participant
 How to handle sensitive topics, for
example, culture, private spaces, and so on
 How to collect the data:
 What data to collect
 What equipment to use
 When to stop observing
23
 Ethnography
 Ethnography is a philosophy with a set of techniques
that include participant observation and interviews
 Ethnographers engage themselves in the culture that
they study
 A researcher’s degree of participation can vary
 Analyzing video and data logs can be time-consuming
 Collections of comments, incidents, and
artifacts(samples) are made
 Types of observation
 Participant Observation
 Non-participant Observation
 Direct Observation
 Indirect Observation
 Controlled Observation
 Uncontrolled Observation
 Participant Observation
 In this observation, the observer is a part of the
phenomenon or group which observed and he
acts as both an observer and a participant.
 Example, a study of tribal customs by an
anthropologist by taking part in tribal activities
like folk dance.
 The person who are observed should not be
aware of the researcher’s purpose. Then only
their behavior will be ‘natural.’
 Non - Participant Observation
 In this method, the observer stands apart and
does not participate in the phenomenon
observed.
 Naturally, there is no emotional involvement on
the part of the observer.
 This method calls for skill in recording
observations in an unnoticed manner.
 Example : use of recording devices to examine
the details of how people talk and behave
together.
 Direct Observation
 This means observation of an event personally
by the observer when it takes place.
 This method is flexible and allows the
observer to see and record delicate aspects of
events and behavior as they occur.
 He is also free to shift places, change the
focus of the observation.
 Example: Observer is physically present to
monitor
 Indirect Observation
 This does not involve the physical
presence of the observer, and the
recording is done by mechanical,
photographic or electronic devices.
 Example : Recording customer and
employee movements by a special motion
picture camera mounted in a department
of large store.
 Controlled Observation
 Controlled observation is carried out either
in the laboratory or in the field.
 It is typified by clear and explicit decisions
on what, how, and when to observe.
 Uncontrolled Observation
 This does not involve over extrinsic and
intrinsic variables (temp, weather, mood).
 Also called natural observation.
 It is primarily used for descriptive
research.
 Participant observation is a typical
uncontrolled one.
Other Data Gathering Techniques
 Notes & still camera
 Audio & still camera
 Video
 Tracking users:
- diaries
- interaction logging

32
 Notes plus still camera
 Taking notes
 Least technical way
 Most feasible way
 Could prove a challenge as one has to observe
and listen while taking notes.
 Writing skills such as writing speed, use of short
hand could prove useful
 Other alternative are include another person
in the data collection process; use a laptop
 Use a camera to record images

33
Audio recording plus still camera
 Audio recording
 Light and easy to carry
 Less disturbing than a video camera
 Recording long interview hours could
be a challenge
 Lack of a visual record
 Cannot be used in a noisy environment

34
 Videos
 Captures both visual and audio data
 Extremely intrusive
 Instruments could be expensive but
digicams/handheld camera are a cheaper
option
 One may tend to focus only on what is
seen through the lens and ignore what
has been missed out.
 Sound may get muffled when recording
under noisy conditions
 Analysis of video data could be very time
consuming.
35
Diaries (Indirect Observation)

 Classical paper based or using online

forms linked to database etc.
 Participant records what he/she did and
experience she had while doing the task
 Diaries collected from all the participants
are then analyzed.

36
 Advantages
 Inexpensive
 Subjects could be tested location
independent
 Experimenter not required to be
present
 Large amount of data could be
gathered

37
 Disadvantages
 Relies on the reliability of the
participant
 Problem of participant not
remembering actual experience of
event (better or worse)
 Subject is the experimenter and
hence variability in data – not much
control other than structures
templates to collect information.

38
Interaction logging (Indirect Observation)

 Record all the interactions of the

subject with the system (key presses,
mouse movements, eye tracking,
etc.) combined with audio and video
logs.
 Log is time stamped.
 Data can be analyzed to understand
cognitive aspects, user behaviors
and reactions to variations in task
etc.

39
Interaction logging (Indirect Observation)

 Useful for web studies.

 Testing Web layouts, displays, GUI
interfaces etc.
 Understanding the users’ reactions to
variations in tasks and layout
combinations.
 Testing the efficiency of the interface
and its ease of use.

40
Choosing and combining techniques

Depends on the:

 Focus of the study

 Participants involved
 Nature of the technique(s)
 Resources available
 Time available

41
 Summary
 Data gathering sessions should have clear goals
 An informed consent may be needed
 Five key issues of data gathering are: goals, choosing
participants, triangulation, participant relationship, pilot
 Data may be recorded using handwritten notes, audio or
video recording, a camera, or any combination of these
 Interviews may be structured, semi-structured, or
unstructured
 Focus groups are group interviews
 Questionnaires may be on paper, online, or telephone
 Observation may be direct or indirect, in the field, or in
controlled settings
 Techniques can be combined depending on the study
focus, participants, nature of technique, and available
resources and time
42