Presenting Survey Data 


and Results"
Professor Ron Fricker"
Naval Postgraduate School"
Monterey, California"

Reading Assignment:"
None"
2/15/13
1

Goals for this Lecture"

•  Discuss a bit about how to display survey

data and results in a briefing"
–  How to structure slides"
–  What to include"
–  Displaying margins of error"
•  Learn about a couple of useful new plots"
•  Discuss how to calculate response rates and
possible issues"

2/15/13
2

A Bit About Briefing Survey Results"

•  When briefing survey results:"

–  DON’T just present the data question-by-question
in the order asked in the survey – boring!"
–  DO tell a story"
•  Focus on the questions that"
–  Answer the survey objective"
–  Give results interesting to the “client”"
»  Sometimes it’s the outliers or tails…"
•  Order the presentation of results so that it’s
logical and interesting to the listener"

2/15/13
3

A Good Briefing Outline"

•  Survey objective(s)"
•  Outline of the survey instrument"
–  Perhaps a brief discussion of design development "
•  Fielding methods and details"
–  Response rate(s)"
•  Comparison of sample to population ""
–  Demonstrate how representative (or not) sample is"
•  Results (see next slide)"
•  Conclusions & discussion"

2/15/13
4

Displaying Survey Results"

•  For results slides, use a small number of

standardized formats"
–  Put the “take away” summary in slide header"
–  Give actual survey question verbiage and number
who answered the question"
•  When giving percentages, show the n as well,
and vice versa"
•  As appropriate, display uncertainty due to
sampling (i.e., the margin of error)"
–  Use actual quotes (e.g., from open-ended
questions) to reinforce graphs and plots"

2/15/13
5

Almost 50% of DL Students (313 of 633)
Agree Library Critical to Their Studies"

45" •  “I believe that the NPS library is an essential

tool that should always be available to DL
students.”"
40" •  “The NPS Library is a valuable resource. Not
every class requires need for use, however,
35" some do.”"
•  “I have used the NPS Library for a good deal
of my day job work as well. It has provided
30" me an easy avenue to periodicals and
journals that are often difficult to get access
to through local research streams.”"
Percent!

25"

20"

15"

10"

5"

n=95" n=218" n=196" n=86" n=27"

0"
Strongly Agree" Agree" Neutral" Disagree" Strongly Disagree"

“How much do you agree or disagree with the following statement: 
‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

2/15/13
6

Those Who Disagreed Were Largely
Engineering and OR Disciplines"
Program! Percent!
SE (Nuclear) – Electrical (n=3)" 100"
45"
SE (Nuclear) – Mechanical (n=1)" 100"
40" Systems Analysis (n=27)" 42"
Cost Estimating and Analysis (n=12)" 27"
35"
Electronic Sys. Engineering (n=47)" 23"
MSES (n=47)" 23"
30"
EMBA (mil & civ) (n=148)" 21"
Percent!

25" Systems Engineering (n=230)" 16"

Program Management (n=68)" 10"
20"
Systems Eng. Management (n=31)" 3"

15"

10"

5"

n=95" n=218" n=196" n=86" n=27"

0"
Strongly Agree" Agree" Neutral" Disagree" Strongly Disagree"

“How much do you agree or disagree with the following statement: 
‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

2/15/13
7

Some Majors Did Not Disagree at All, But
Small Numbers Not Definitive"
Program! Percent!
SE (Nuclear) – Electrical (n=3)" 100"
45"
SE (Nuclear) – Mechanical (n=1)" 100"
40" Systems Analysis (n=27)" 42"
Cost Estimating and Analysis (n=12)" 27"
35"
Electronic Sys. Engineering (n=47)" 23"

Program! Percent! MSES (n=47)" 23"

30"
Contract Management (n=3)" 0" EMBA (mil & civ) (n=148)" 21"
Percent!

25" HSI (N=9)" 0" Systems Engineering (n=230)" 16"

Software Engineering (n=4)" 0" Program Management (n=68)" 10"

20"
Space Systems Operations (n=6)" 0" Systems Eng. Management (n=31)" 3"

15"

10"

5"

n=95" n=218" n=196" n=86" n=27"

0"
Strongly Agree" Agree" Neutral" Disagree" Strongly Disagree"

“How much do you agree or disagree with the following statement: 
‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

2/15/13
8

Neutrals Are Largely Outside 

(Hard) Engineering Disciplines"
Program! Percent!
MSES (n=47)"
•  “I believe that the NPS library is an essential51"
45"
tool that
Contract should always
Management be available to DL 47"
(n=18)"
students.”"
40" HSI (n=9)"
•  “The NPS Library is a valuable resource. Not 37"
every(mil
EMBA class requires
& civ) need for use, however,35"
(n=148)"
35" some do.”"
Program
•  “I have Management
used the NPS (n=68)"
Library for a good deal34"
of my day
Systems job work (n=230)"
Engineering as well. It has provided 32"
30" me an easy avenue to periodicals and
Electronic
journals Systems Engineering
that are often difficult(n=47)"
to get access21"
to through local research
Analysisstreams.”"
Percent!

25" Cost Estimating and (n=12)" 18"

Systems Analysis (n=27)" 17"
20"
Systems Eng. Management (n=31)" 10"

15"

10"

5"

n=95" n=218" n=196" n=86" n=27"

0"
Strongly Agree" Agree" Neutral" Disagree" Strongly Disagree"

“How much do you agree or disagree with the following statement: 
‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

2/15/13
9

Most Disciplines Had Substantial
Percentage of Students Who Agreed"
Program! Percent!
Space Systems
•  “I believe that (n=6)"
the NPS library is an essential 100"
45"
tool thatEng.
Systems should always be(n=31)"
Management available to DL 87"
students.”"
40" Space
•  “The Systems Operations
NPS Library (n=6)" resource. Not
is a valuable 83"
HSIevery class requires need for use, however,63"
(n=9)"
35" some do.”"
Program
•  “I have Management
used the NPS (n=68)"
Library for a good deal 56"
of my
Cost day job and
Estimating workAnalysis
as well.(n=12)"
It has provided 55"
30" me an easy avenue to periodicals and
Electronic
journals Systems Engineering
that are often difficult(n=47)"
to get access53"
to through local research streams.”"
Percent!

25" Contract Management (n=18)" 53"

Systems Engineering (n=230)" 50"
20"
EMBA (mil & civ) (n=148)" 43"

15" Systems Analysis (n=27)" 38"

MSES (n=47)" 22"
10"

5"

n=95" n=218" n=196" n=86" n=27"

0"
Strongly Agree" Agree" Neutral" Disagree" Strongly Disagree"

“How much do you agree or disagree with the following statement: 
‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

2/15/13
10

On Displaying Margins of Error"

•  “Do I have to display margins of error on every plot?”!

–  No, sometimes it’s overkill and/or distracting"
–  But they should be communicated somehow "
•  If not included on every plot and table, give the
reader/audience some general guidelines: "
"“For analyses of the entire DL student population, the
margins of error in this survey are approximately!
–  two percent for questions with a binary scale (e.g., yes/no), !
–  five percent for questions with a Likert scale (e.g., strongly
agree, agree, neutral, disagree, strongly disagree).!
!When analyzing smaller groups the margins of error
will be larger, perhaps substantially.”"

2/15/13
11

Barcharts and Histograms Not Optimal for
Comparing Between Groups or Subsets"
•  Neither plot particularly good at allowing
visual comparison between groups"
Presidio Health Clinic Rating

700
Very Satisfied
Satisfied
Neutral
600
Unsatisfied
Very Unsatisfied
500
400
Count

300
200
100
0

Married and collocated with family Single Married and a geographic bachelor Single with dependents Other
Presidio Health Clinic Rating
300

Very Unsatisfied
Unsatisfied
Neutral
Satisfied
250

Very Satisfied
200
Count

150
100
50
0

Married and collocated with family Single Married and a geographic bachelor Single with dependents Other
2/15/13
12

Barcharts and Histograms Not Optimal for
Comparing Between Groups or Subsets"
•  Converting to percentages does not really
help:"
Presidio Health Clinic Rating

1.0
0.8
0.6
Fraction

0.4
0.2
0.0

Married and collocated with family Single Married and

Presidio a geographic
Health Clinicbachelor
Rating Single with dependents Other
0.5
0.4
0.3
Fraction

0.2
0.1
0.0

Married and collocated with family Single Married and a geographic bachelor Single with dependents Other
2/15/13
13

 Likert-scale Data:

Diverging Stacked Bar Charts"

2/15/13
14

Source: Robbins, N.B., and R.M. Heiberger, Plotting Likert and Other Rating Scales, Proceedings of the 2011 Joint Statistical Meetings, 1058-1066.

 Compare to Traditional Bar Charts"

•  Much harder to distinguish differences:"

–  Divided bar chart:"

–  Side-by-side bar chart"

2/15/13
15

Source: Robbins, N.B., and R.M. Heiberger, Plotting Likert and Other Rating Scales, Proceedings of the 2011 Joint Statistical Meetings, 1058-1066.

 Creating Diverging Stacked Bar Charts"

•  In R, use the likert() function in the HH package"

•  Examples from QOL survey results:"
Presidio Health Clinic Rating Presidio Health Clinic Rating

Married and collocated with family Married and collocated with family 725

Single Single 210

Row Count Totals

Family Status

Family Status
Married and a geographic bachelor Married and a geographic bachelor 70

Single with dependents Single with dependents 21

Other Other 6

200 0 200 400 40 20 0 20 40 60

Count Percent
Likert_Scale Likert_Scale
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied

2/15/13
16

Diverging Stacked Bar Charts"

2/15/13
17

Diverging Stacked Bar Charts"

2/15/13
18

Other Thoughts on Survey Briefings"

•  Goal is to communicate to decision maker what the

data say about the survey objective"
–  Don’t make it a data dump"
•  Focus on effective graphical communication"
–  Use graphics that effectively communicate the quantitative
results"
•  See Cleveland (1994, 1993) and Tufte (1990, 2001)"
–  Save the mathematics, modeling, and technical details for
the back-up slides / report appendix"
•  But do communicate the necessary details to
convince the audience that the survey was done
effectively and rigorously"
–  Response rate (presumably high), margin(s) of error, etc."

2/15/13
19

Calculating the Response Rate"

•  In theory, the response rate is simple:"

Number of completed surveys
Response Rate =
Number of surveys sent out
•  In practice, it can be more complicated to
calculate "
–  What counts as a “completed survey”?"
–  What to do with those who could not be reached,
say due to incorrect contact information?"
–  Etc."

2/15/13
20

Other Potential Response Rate 

Calculation Complications"
•  When must screen frame members to
determine sample eligibility"
–  Hard then to determine denominator for response
rate calculation"
•  When sample frame consists of clustered
elements and full cluster nonrespondent"
–  Unclear how many sample elements were really
nonrespondent"
•  When using unequal sampling probabilities"
–  Unclear whether to use weights in response rate
calculation"

2/15/13
21

One (Conservative) Approach"

I
Response rate =
I + R + NC + O + e × U
where "
"I = number of complete surveys"
"R = number of refusals and break-offs"
"NC = number of non-contacts"
"O = number of other eligible"
"U = number of unknown eligibility"
"e = estimated proportion of eligibility"
2/15/13
22

If Only News Organizations (and Many
Others) Followed These Suggestions…"

2/15/13
23

What We Have Just Learned"

•  Discussed a bit about how to display survey

data and results in a briefing"
–  How to structure slides"
–  What to include"
–  Displaying margins of error"
•  Learned about a couple of useful new plots"
•  Discussed how to calculate response rates
and possible issues"
"

2/15/13
24

References on Good Graphics"

•  Cleveland, W.S. (1993). Visualizing Data,

Hobart Press."
•  Cleveland, W.S. (1994). The Elements of
Graphing Data, Hobart Press."
•  Tufte, E.R. (2001). The Visual Display of
Quantitative Information, 2nd ed., Graphics
Press."
•  Tufte, E.R. (1990). Envisioning Information,
Graphics Press."

2/15/13
25