Scribd
Upload
96 views

Matt Calvert, UW-Extension Youth Development Specialist PDAC Wisline Web February 5, 2009

This document provides guidance on analyzing and interpreting data from program evaluations. It discusses the importance of having an analysis plan early in the process to ensure the right data is collected to answer evaluation questions. Both quantitative and qualitative data analysis techniques are covered, including descriptive statistics, pre/post surveys, coding themes from interviews and focus groups. The document emphasizes interpreting results by discussing what was learned, drawing conclusions and recommendations, and acknowledging limitations.

Uploaded by

Nagarjuna Reddy
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
96 views

Matt Calvert, UW-Extension Youth Development Specialist PDAC Wisline Web February 5, 2009

This document provides guidance on analyzing and interpreting data from program evaluations. It discusses the importance of having an analysis plan early in the process to ensure the right data is collected to answer evaluation questions. Both quantitative and qualitative data analysis techniques are covered, including descriptive statistics, pre/post surveys, coding themes from interviews and focus groups. The document emphasizes interpreting results by discussing what was learned, drawing conclusions and recommendations, and acknowledging limitations.

Uploaded by

Nagarjuna Reddy
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 28

Analyzing and interpreting data

Matt Calvert, UW-Extension Youth


Development Specialist
PDAC Wisline Web
February 5, 2009
Theres a world of difference between truth and facts.
Facts can obscure the truth.
- Maya Angelou

True or False?
Complex analysis impresses people.

I am generally able to analyze and


interpret the program data I gather

Data analysis and interpretation

Think about analysis EARLY


Start with a plan
Code, enter, clean
Analyze
Interpret
Reflect

What did we learn?


What conclusions can we draw?
What are our recommendations?
What are the limitations of our analysis?
3

Why do I need an analysis plan?


To make sure the questions and
your data collection instrument will
get the information you want.
To align your desired report with
the results of analysis and
interpretation.
To improve reliability--consistent
measures over time.
4

Key components of a data analysis plan


Purpose of the evaluation
Questions
What you hope to learn from the
question
Analysis technique
How data will be presented

Analyzing and Interpreting Quantitative


Data
Quantitative Data is
Presented in a numerical format
Collected in a standardized manner
e.g. surveys, closed-ended interviews,
tests
Analyzed using statistical techniques

True or False?

Quantitative data we gather in


Extension are more generalizable
than qualitative data.
Stating limitations weakens the
evaluation

Analyzing Survey Data


Do you want to report
how many people answered a, b, c,
d?
the average number or score?
a change in score between two
points in time?
how people compared?
how many people reached a certain
level?
8

Common descriptive statistics

Count (frequencies)
Percentage
Mean
Mode
Median
Range
Standard deviation
Variance
Ranking
9

Other Statistics
Statistical Significance
Factor Analysis
Etc.
Not often used in Extension program
evaluationgenerally require
randomization, large samples, and/or
control groups
10

Getting your data ready


Assign a unique identifier
Organize and keep all forms
(questionnaires, interviews,
testimonials)
Check for completeness and
accuracy
Remove those that are incomplete
or do not make sense
11

Data entry
You can enter your data
By hand
By computer
http://learningstore.uwex.edu/Using-Excelfor-Analyzing-Survey-QuestionnairesP1030C0.aspx

12

Data entry by computer


By Computer
Excel (spreadsheet)
Microsoft Access (database mngt)
Quantitative analysis: SPSS (statistical
software)
Qualitative analysis: Epi info (CDC data
management and analysis program:
www.cdc.gov/epiinfo); In ViVo, etc.
13

Data entry computer screen


Smoking: 1 (YES) 2 (NO)

Survey
ID
001
002
003
004
005

Q1 Do you
smoke?
1
1
2
2
1

Q2 Age
24
18
36
48
26

Q3 Support
ordinance?
2
2
1
1
1

14

Dig deeper
Did different groups show different
results?
Were there findings that surprised
you?
Are there things you dont
understand very well further study
needed?

15

Supports
restaurant
ordinance

Opposes
restaurant
ordinance

Undecided/
declined to
comment

8
(15% of
smokers)

33
(60% of
smokers)

14
(25% of
smokers)

Non-smokers
(n=200)

170
(86% of nonsmokers)

16
(8% of nonsmokers)

12
(6% of nonsmokers)

Total
(N=255)

178
(70% of all
respondents)

49
(19% of all
respondents)

26
(11% of all
respondents)

Current
smokers
(n=55)

16

Pre-post or post-then-pre Data


Check dataany individual not responding to
both pre and post should be discarded
Decide:
Report individual change or combined change?
Compare to a standard?

See PD&E Evaluation Quick Tip 30


17

Interpretation of Pre/Post Data


Which statement is the most significant
to you?
The number of club officers reporting
strong or very strong knowledge of
parliamentary procedure increased
from 4 (50%) to 6 (75%).
The number of club officers reporting at
least some knowledge of
parliamentary procedure increased
from 5 (63%) to 8 (100%).
50% of the 8 participants reported an
increase in their knowledge of
parliamentary procedure.
Self-reports of knowledge of
parliamentary procedure on a scale
from 1=minimal to 4=very strong
averaged 2.375 before the training
and 3.0 after.

Rate your knowledge of parliamentary


procedure : 1=minimal 2=some
3=strong 4=very strong

Respondent

Pre-

Post-

(missing)

(missing)

18

Discussing limitations
Written reports:
Be explicit about your limitations
Oral reports:
Be prepared to discuss limitations
Be honest about limitations
Know the claims you cannot make
Do not claim causation without a true
experimental design
Do not generalize to the population without
random sample and quality administration
(e.g., <60% response rate on a survey)
19

Analyzing and Interpreting Qualitative


Data
Qualitative data is thick in detail and
description.
Data often in a narrative format
Data often collected by observation, openended interviewing, document review
Analysis often emphasizes understanding
phenomena as they exist, not following predetermined hypotheses
20

Quiz

Data have their own meaning


Qualitative analysis is easier than
quantitative analysis

21

Analyzing qualitative data


Content analysis steps:
1. Transcribe data (if audio taped)
2. Read transcripts
3. Highlight quotes and note why important
4. Code quotes according to margin notes
5. Sort quotes into coded groups (themes)
6. Interpret patterns in quotes
7. Describe these patterns
22

Hand coding
qualitative data

23

Emergent or pre-conceived categories?


Consider this sectionwhat themes
emerge related to arts and
communications programs?

24

Focus group sample

Q. How are the arts/comm. Activities different from other projects?


When I was little, fair was never that exciting to me. I didnt realize how much else there was to do. Its shown me how many
different things you could do and get money for it, so you cant go wrong.
Art projects are more personal. Its more of an expression of yourself. Where with an animal, you just show the animal off. With
art, you show you can do something good.
(lots of agreement with this).
Q. What about A&C helps with communication?
Youre forced into the situations. A lot of clubs require demonstrations. The first year, everyone encouraged me just to talk.
To have someone push you along the way helps.
In a 4-H Club, you get to be friends. Its more like family.
If youre ticked off at your family, you can go up to your room and draw. Feel better.
It feels more comfortable because youre not the only one. People at school say, shes an arts nerd. You can relate to them. And
even theyre so diverse. You cram people in and say youve got to get along.
With school, you only go part of the year. With 4-H, its all year a few times a month. At school, its every single day. Its too much.
Q. How have you been changed?
Entering art in fair has made me not worry so much about what other people think about my work. Ill take into consideration what
they say, but most of its opinion. They can have their own style through photography and I can have mine through sketching.
I think it depends on the suggestion. I usually consider it a lot and it might help me or it might not.
Its always there in the back of your mind at least.
Its your own style. If people say, you could change this, I can say thank you for your input, but Im not worried.
Q. How is 4-H different from other school and community experiences?
Working with people from all over the state and bring their ideas back. Also, how theyre doing the Madison cow parade. I
wouldnt have even heard about that. This year I have one of the micro-cows that was selected.
School is an assignment. You have to do that project. Where 4-H you can do any kind of art, any way you want to do it.
But at school, you can also exhibit. You can get opinions from an art teacher like from a judge. Its not that different except that
you can do what you want.
Although you do learn a lot of techniques from school that you might not learn from 4-H.

25

Using theories and categories


Consider this section againhow are
arts and communication programs
providing the 4 Essential Elements?
Generosity Opportunity to value and practice service for
others

Belonging An inclusive environment and safe environment


Independence Opportunity to see oneself as an active
participant in the future; Opportunity for self-determination

Mastery Engagement in Learning and Opportunity for Mastery


26

Ensuring Validity in Qualitative Analysis


Be systematic
Use multiple raters
Attend to context (e.g. keep track of
who said what)
Account for outlying and surprising
statements
Triangulate
27

Resources
Building Capacity in Evaluating
Outcomes Curriculum and Training
(October 2009)
http://www.uwex.edu/ces/pdande/evaluati
on/bceo/index.html

28

You might also like