Program Evaluation: The Structured Interview

swords-to-plowshares.org
VETERAN EMPLOYMENT COLLABORATION
Program Evaluation:
The Structured
Interview

War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’
mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all
veterans in need, and to prevent and end homelessness and poverty among veterans.
Introduction
GOALS: PROVIDE
• Introductory to intermediate level content
• Easy-to-use guide
• Lots of practical tips
• Stretch your thinking
• Opportunity for your input
• Time for Q & A
TOPICS
• Structured interviews in context of survey tools
• Best practices
• Protocol design
• Question types
• Tips for interviewers
• Dealing with data

Introduction
SURVEYTOOLS
• Open-ended interviews (rare)
• Semi-structured interviews
• Structured interviews
• Self-completed questionnaire forms
(on line or on paper)

Interviews vs. Questionnaires
INTERVIEWS QUESTIONNAIRES
 More labor intensive  Less labor intensive
 Higher skill level  Lower skill level
 Can follow-up for clarity  Not possible to follow up
 Richer data  Thinner data
 Questions & activities  Questions only
 Less anonymous (courtesy bias)  Anonymous

Using Structured Interviews
Take advantage of opportunity to discuss/exchange
Employment placement
Exit interviews for staff and clients
Exploring potential partnerships
Consider addressing issues related to outputs (activities) as well as outcomes (resulting
improvements to quality of life)
Using Structured Interviews
→ Take advantage of opportunity to
discuss/exchange
→ Employment placement
→ Exit interviews for staff and clients
→ Exploring potential partnerships
→ Consider addressing issues related to outputs
(activities) as well as outcomes (resulting
improvements to quality of life)

Best Practices – Research Planning
GATHER INPUT FROM STAKEHOLDERS
o Funder requirements
o In-house development department
o Those responsible for data collection and management
CONSENSUS ON PROJECT GOALS
o Reason for research
o Research questions
o Application of findings

Best Practices - Research
METHODOLOGY
o Match the tool to the research goals and context
o Recognize strengths and limits of tool
o Pilot test all phases of protocol (recruitment, data
collection, data management, analysis)
o Sample size
WORKING BILINGUALLY
o Appropriate for region
o Back translation to test instrument
o Allow for translation of data
o Native speakers preferred

Best Practices - Respondents
HSS human subject research guidelines
(well-being, respect, protect)
Culturally responsible approach (more than
just linguistics)
Accessible content
Don’t assume literacy

Structured Interview
PROTOCOL DESIGN - GENERIC
 Similar rules of thumb for questionnaires.
 Brief explanation of the purpose.
 Start easy…
• Break the ice
• Move toward more complex or personal content
• Explore the positives of X before turning to
examine the negatives
 “Areas of Inquiry”
• Begin open-ended… then hone in with pre-
meditated prompts.
 Wrap-up – it can be useful to get direct re: research
goal.

PROTOCOL DESIGN - INTAKE
 The respondent is likely to be seeking you out,
rather than having been recruited for the interview.
 The nature of this interaction, the intake is to
collect/share lots of personal information. You
might start off with demographics and service
history, etc.
 Caseworkers are comfortable working from
unscripted forms.

Poll Questions
HOW WELL DOYOUTHINKTHE WEBINAR IS GOING?
1. Extremely well.
2. Very well.
3. Somewhat well.
4. It's okay.
5. Wait, there's something wrong with this question.

QUESTION DESIGN
 Similar rules of thumb for questionnaires.
 NO nice-to-know questions.
 Clear, concise wording (don’t make respondent struggle to understand).
 NO double-barreled questions.
 Examine draft for bias, leading or judgmental verbiage.
 Give permission to respondent to disagree or be candid.
 Prepare appropriate prompts and follow-up questions in advance.
 Keep fixed response choices simple, consider using visual aid.
 Use fixed response questions as springboards for conversation.

QuestionType:
OPEN-ENDED WITH PRE-MEDITATED PROMPTS
Example: What can you tell me about today’s workshop?Was it useful?
Useful prompts when the respondent needs encouragement:
 Did you get any new ideas? Can you describe that/one for me?
 Did you pick up any information that you think will be helpful?
What information?
 Did you get your questions answered? What questions did/do you
have?
 Were there any highlights? Did anything jump out at you?
 Were you able to _________________________________?

QuestionType:
OPEN-ENDED WITH PRE-MEDITATED PROMPTS
Example: What can you tell me about today’s workshop?Was it useful?
Possible follow-ups or probes:
 Was that new information for you?
 How do you think you might be able to use/follow up…
 Is there anything that could have made it more useful…

QuestionType:
FIXED RESPONSE
Multiple Choice & Ratings Scales
 Keep simple
 Consider providing a printout of the scale graphic
 Use as a springboard for discussion: Why did you
score X the way you did?
What are 3 words that describe the program you
attended today?
 You can let this go (great on questionnaire forms
too) or follow up on one of the words they offer.

QuestionType:
USING IMAGES
Reflect on images to encourage discussion
 Be careful of getting too concrete, respondents can
fixate on the example.
Card Sorts
 Print a set of images or phrases on cards which
respondents can sort into categories.
 Preferences, likelihood of behaviors or choices, level
of interest or familiarity, etc.

Tips for the Interviewer
USEFUL ALL-PURPOSE FOLLOW-UPS
Remember, context and tone of voice make all the difference.
 Can you tell me more about that… about X?
 What do you mean when you say X?
 Can you think of an example?
 Was that what you had expected?
 Why do you think it was memorable?
 What made you think that?
 Why do you think that was/happened?
 How do you feel about that?

TECHNIQUE
 Consider yourself as a neutral but friendly extension of the
survey instrument. Your professional, friendly manner
helps the respondent feel confident.
 After delivering your concise question, be willing to wait
for a response, but be ready with prompts if needed.
 Remember what you’ve learned about active listening and
put it to use.
 When probing for clarity, use respondents’ words back to
them.
 Once you get started, avoid narrating the process.

TECHNIQUE
 Avoid “leading” the respondent by suggesting ideas or vocabulary.
They can feel influenced to agree with your suggestion and/or
struggle to make it fit their experience.
 Natural, conversational prompts are helpful, e.g., “Mhm.” Avoid
comments which include suggestions of judgement, e.g., “That’s a
great observation!” You can say, “I’m interested to hear more
about that.”
 Sometimes starting a sentence and purposely leaving it open is a
good way to prompt a respondent. E.g.: So, you would say that
it….? So, does that mean you would…?

Recording
Audio recording
Handwritten notes
Entering into database at keyboard
(intake)

Data Management & Analysis
Custom Databases
SurveyMonkey
Microsoft Excel
Coding open-ended data = categorizing

Poll Questions
HOW WOULDYOU RATETHE USEFULNESS OF
THIS WEBINAR?
1. Very useful.
2. Somewhat useful.
3. Not very useful.
4. About as useful as this question is without opportunity
to follow-up about the rating I give.
5. More useful than answer No. 4 suggests.

ThankYour ForYour Attendance
Copyright © 2017 by Swords to Plowshares
All rights reserved, including the right of reproduction in
whole or in part in any form.
Swords to Plowshares
The California Veterans Employment
andTraining Collaborative
1060 Howard Street
San Francisco, CA 94103
Amy Fairweather
Director
afairweather@stp-sf.org
Wendy Meluch
Principal & Founder
VisitorStudies.com
wendy@visitorstudies.com
(415) 897.4051

Program Evaluation: The Structured Interview

More Related Content

Viewers also liked (16)

More from Swords to Plowshares (20)

Recently uploaded (20)

Program Evaluation: The Structured Interview

Editor's Notes