SUMMARY AND ANALYSIS

OF EXTENSION
EDUCATION PROGRAM
EVALUATION IN R

SALVATORE S. MANGIAFICO
Rutgers Cooperative Extension
New Brunswick, NJ

VERSION 1.6.5

i
 ©2016 by Salvatore S. Mangiafico.

Non-commercial reproduction of this content, with attribution, is permitted.

For-profit reproduction without permission is prohibited.

If you use the code or information in this site in a published work, please cite it as a
source. Also, if you are an instructor and use this book in your course, please let me know.
[email protected]

Mangiafico, S.S. 2016. Summary and Analysis of Extension Program Evaluation in R, version
1.6.5.
rcompanion.org/documents/RHandbookProgramEvaluation.pdf .

Web version: rcompanion.org/handbook/

ii
List of Chapters

List of Chapters
List of Chapters.......................................................................................................................... iii
Table of Contents ...................................................................................................................... vi
Introduction ...............................................................................................................................1
Purpose of This Book.......................................................................................................................... 1
About the Author of this Book ............................................................................................................ 2
Using R .............................................................................................................................................. 2
Statistics Textbooks and Other Resources ........................................................................................... 9
Statistics for Educational Program Evaluation ........................................................................... 13
Why Statistics? ................................................................................................................................ 13
Evaluation Tools and Surveys ........................................................................................................... 14
Variables, Descriptive Statistics, and Plots ................................................................................ 20
Types of Variables ............................................................................................................................ 20
Descriptive Statistics ........................................................................................................................ 24
Confidence Intervals ........................................................................................................................ 56
Basic Plots ....................................................................................................................................... 67
Understanding Statistics and Hypothesis Testing .................................................................... 105
Hypothesis Testing and p-values......................................................................................................105
Reporting Results of Data and Analyses ...........................................................................................121
Choosing a Statistical Test ...............................................................................................................126
Independent and Paired Values .......................................................................................................134
Likert Data ............................................................................................................................. 140
Introduction to Likert Data ..............................................................................................................140
Descriptive Statistics for Likert Data ................................................................................................143
Descriptive Statistics with the likert Package ...................................................................................164
Confidence Intervals for Medians ....................................................................................................169
Converting Numeric Data to Categories ...........................................................................................177
Traditional Nonparametric Tests............................................................................................. 188
Introduction to Traditional Nonparametric Tests .............................................................................188
One-sample Wilcoxon Signed-rank Test ...........................................................................................190
Sign Test for One-sample Data ........................................................................................................195

iii
 Two-sample Mann–Whitney U Test .................................................................................................197
Mood’s Median Test for Two-sample Data ......................................................................................203
Two-sample Paired Rank-sum Test ..................................................................................................205
Sign Test for Two-sample Paired Data..............................................................................................212
Kruskal–Wallis Test .........................................................................................................................215
Mood’s Median Test .......................................................................................................................226
Friedman Test .................................................................................................................................231
Quade Test .....................................................................................................................................242
Nonparametric Regression ..............................................................................................................250
Nonparametric Regression for Time Series ......................................................................................259
Permutation Tests for Ordinal Data ........................................................................................ 266
Introduction to Permutation Tests...................................................................................................266
One-way Permutation Test of Independence for Ordinal Data..........................................................270
One-way Permutation Test of Symmetry for Paired Ordinal Data .....................................................279
Concepts for Linear Models ..................................................................................................... 289
Introduction to Linear Models .........................................................................................................289
Using Random Effects in Models......................................................................................................292
What are Least Square Means? .......................................................................................................297
Least Square Means for Multiple Comparisons ................................................................................299
Factorial ANOVA: Main Effects, Interaction Effects, and Interaction Plots .........................................303
p-values and R-square Values for Models ........................................................................................315
Ordinal Tests with Cumulative Link Models ............................................................................. 328
Introduction to Cumulative Link Models (CLM) for Ordinal Data .......................................................328
Two-sample Ordinal Test with CLM .................................................................................................330
Two-sample Paired Ordinal Test with CLMM ...................................................................................334
One-way Ordinal ANOVA with CLM .................................................................................................339
One-way Repeated Ordinal ANOVA with CLMM ..............................................................................346
Two-way Ordinal ANOVA with CLM .................................................................................................355
Two-way Repeated Ordinal ANOVA with CLMM ..............................................................................371
Tests for Nominal Data ........................................................................................................... 385
Introduction to Tests for Nominal Variables .....................................................................................385
Confidence Intervals for Proportions ...............................................................................................390
Goodness-of-Fit Tests for Nominal Variables....................................................................................395

iv
 Association Tests for Nominal Variables ..........................................................................................402
Tests for Paired Nominal Data .........................................................................................................410
Cochran–Mantel–Haenszel Test for 3-Dimensional Tables ................................................................420
Cochran’s Q Test for Paired Nominal Data .......................................................................................424
Association Tests for Ordinal Tables ................................................................................................434
Models for Nominal Data ................................................................................................................445
Parametric Tests..................................................................................................................... 463
Introduction to Parametric Tests .....................................................................................................463
One-sample t-test ...........................................................................................................................483
Two-sample t-test ...........................................................................................................................487
Paired t-test ....................................................................................................................................493
One-way ANOVA ............................................................................................................................499
One-way ANOVA with Blocks ..........................................................................................................509
One-way ANOVA with Random Blocks .............................................................................................518
Two-way ANOVA ............................................................................................................................527
Repeated Measures ANOVA ............................................................................................................538
Correlation and Linear Regression ...................................................................................................549
Advanced Parametric Methods .......................................................................................................567
Transforming Data ..........................................................................................................................588
Analysis of Count Data and Percentage Data .......................................................................... 607
Hermite and Poisson Regression for Count Data ..............................................................................607
Beta Regression for Percent and Proportion Data ............................................................................625

v
Table of Contents

Table of Contents
List of Chapters _________________________________________________________________ iii
Table of Contents _______________________________________________________________ vi
Introduction ____________________________________________________________________ 1
Purpose of This Book __________________________________________________________________ 1
Goals of this book ____________________________________________________________________________ 1
Specific learning goals _________________________________________________________________________ 1
Pre-requisites________________________________________________________________________________ 1
What this book will not cover ___________________________________________________________________ 1
Designing and conducting surveys _____________________________________________________________ 2
Advanced statistical analyses _________________________________________________________________ 2
About the Author of this Book __________________________________________________________ 2
Using R _____________________________________________________________________________ 2
R and RStudio________________________________________________________________________________ 2
Using the RStudio environment _________________________________________________________________ 3
Installation _______________________________________________________________________________ 3
Using the R Console environment ________________________________________________________________ 3
What if I don’t have my own computer? __________________________________________________________ 3
Portable installation on a usb drive ____________________________________________________________ 3
Using university computers __________________________________________________________________ 3
Using R online _____________________________________________________________________________ 3
apps.rutgers.edu___________________________________________________________________________ 3
Tests for package installation ___________________________________________________________________ 4
Required readings ____________________________________________________________________________ 5
About R __________________________________________________________________________________ 5
Obtaining R _______________________________________________________________________________ 6
A Few Notes to Get Started with R ____________________________________________________________ 6
Avoiding Pitfalls in R ________________________________________________________________________ 6
Help with R _______________________________________________________________________________ 6
R Tutorials ________________________________________________________________________________ 6
References for this chapter _____________________________________________________________________ 6
Exercises A __________________________________________________________________________________ 6
Statistics Textbooks and Other Resources _________________________________________________ 9
Some free general statistics books _______________________________________________________________ 9
Books on statistical analyses and tests ___________________________________________________________ 10
Analyses in R _______________________________________________________________________________ 10
Online resources __________________________________________________________________________ 10
Text books _______________________________________________________________________________ 10
Videos __________________________________________________________________________________ 10
Other useful resources for understanding statistics ________________________________________________ 11
Videos __________________________________________________________________________________ 11
Blogs ___________________________________________________________________________________ 11
Online Learning Modules and Massive Open Online Courses (MOOC’s) ________________________________ 11

vi
Statistics for Educational Program Evaluation ________________________________________ 13
Why Statistics? ______________________________________________________________________ 13
The importance of statistics in program evaluation ________________________________________________ 13
Applicability of statistical tests to other fields and situations _________________________________________ 14
Evaluation Tools and Surveys __________________________________________________________ 14
Extension programs __________________________________________________________________________ 14
Designing a questionnaire _____________________________________________________________________ 14
Some principles of questionnaire design for program evaluation ___________________________________ 15
Examples of evaluation tools ________________________________________________________________ 17
Optional Readings ___________________________________________________________________________ 18
Optional additional resources __________________________________________________________________ 19
References for this chapter ____________________________________________________________________ 19
Acknowledgements __________________________________________________________________________ 19

Variables, Descriptive Statistics, and Plots ___________________________________________ 20

Types of Variables ___________________________________________________________________ 20
Organizing data: observations and variables ______________________________________________________ 20
Long-format and wide-format data ___________________________________________________________ 20
Types of variables ___________________________________________________________________________ 21
Nominal data ____________________________________________________________________________ 21
Ordinal data _____________________________________________________________________________ 21
Interval/ratio data ________________________________________________________________________ 22
Levels of measurement ____________________________________________________________________ 22
Required readings ___________________________________________________________________________ 23
Optional readings ___________________________________________________________________________ 23
Exercises B _________________________________________________________________________________ 23
Descriptive Statistics _________________________________________________________________ 24
Packages used in this chapter __________________________________________________________________ 24
Descriptive statistics _________________________________________________________________________ 24
Descriptive statistics for interval/ratio data _______________________________________________________ 25
Statistics of location for interval/ratio data ____________________________________________________ 26
Statistics of variation for interval/ratio data ____________________________________________________ 27
Statistics for grouped interval/ratio data ______________________________________________________ 30
Summaries for data frames _________________________________________________________________ 32
Dealing with missing values _________________________________________________________________ 33
Statistics of shape for interval/ratio data ______________________________________________________ 37
Descriptive statistics for ordinal data ____________________________________________________________ 38
Example of descriptive statistics for ordinal data ________________________________________________ 39
Statistics of location for ordinal data __________________________________________________________ 41
Statistics of variation for ordinal data _________________________________________________________ 41
Statistics for grouped ordinal data____________________________________________________________ 41
Statistics for ordinal data treated as nominal data _______________________________________________ 42
Descriptive statistics for nominal data ___________________________________________________________ 44
Example of descriptive statistics for nominal data _______________________________________________ 45
Required readings ___________________________________________________________________________ 46
Optional readings ___________________________________________________________________________ 48
Optional note on reporting summary statistics honestly ____________________________________________ 49
Optional analyses ___________________________________________________________________________ 50
Robust estimators: trimmed mean and Winsorized mean ________________________________________ 50
Geometric mean __________________________________________________________________________ 52

vii
 Harmonic mean __________________________________________________________________________ 54
Exercises C _________________________________________________________________________________ 54
Confidence Intervals _________________________________________________________________ 56
Packages used in this chapter __________________________________________________________________ 56
Understanding confidence intervals _____________________________________________________________ 56
Populations and samples ___________________________________________________________________ 57
Interpretation of confidence intervals _________________________________________________________ 57
Confidence intervals as an alternative to some tests _____________________________________________ 57
Example for confidence intervals _______________________________________________________________ 58
Traditional confidence intervals for means _____________________________________________________ 59
Traditional confidence intervals for means for grouped data _______________ Error! Bookmark not defined.
Optional Analyses: confidence interval for the mean by bootstrapping ______________________________ 63
Required readings ___________________________________________________________________________ 64
Optional readings ___________________________________________________________________________ 64
References _________________________________________________________________________________ 64
Optional analyses ___________________________________________________________________________ 64
Confidence intervals for geometric mean ______________________________________________________ 64
Confidence intervals for geometric means for groups ____________________________________________ 65
Exercises D _________________________________________________________________________________ 66
Basic Plots__________________________________________________________________________ 67
Packages used in this chapter __________________________________________________________________ 67
The need to understand plots __________________________________________________________________ 67
Some advice on producing plots ________________________________________________________________ 67
Producing clear and informative plots _________________________________________________________ 68
Misleading plots __________________________________________________________________________ 68
Using software to produce plots _____________________________________________________________ 69
Exporting plots ___________________________________________________________________________ 69
Examples of basic plots for interval/ratio and ordinal data ___________________________________________ 69
Histogram _______________________________________________________________________________ 70
Box plots ________________________________________________________________________________ 74
Plot of means and interaction plots ___________________________________________________________ 78
Bar plot of means _________________________________________________________________________ 84
Scatter plot ______________________________________________________________________________ 88
Examples of basic plots for nominal data _________________________________________________________ 89
Bar plot for counts of a nominal variable ______________________________________________________ 90
Mosaic plot ______________________________________________________________________________ 92
Optional analyses: Describing histogram shapes ___________________________________________________ 93
Optional analyses: Misleading and disorienting plots _______________________________________________ 96
The need to include measures of variation _____________________________________________________ 96
For bar charts, start the y-axis at zero _________________________________________________________ 97
Avoid disorienting the audience _____________________________________________________________ 98
Use the best model_______________________________________________________________________ 100
References ________________________________________________________________________________ 102
Required readings __________________________________________________________________________ 102
Optional readings __________________________________________________________________________ 102
Exercises E ________________________________________________________________________________ 103

Understanding Statistics and Hypothesis Testing _____________________________________ 105

Hypothesis Testing and p-values _______________________________________________________ 105
Statistical inference _________________________________________________________________________ 105
Hypothesis testing __________________________________________________________________________ 105

viii
 The null and alternative hypotheses _________________________________________________________ 105
p-value definition ________________________________________________________________________ 106
Theory and practice of using p-values __________________________________________________________ 108
Wait, does this make any sense? ____________________________________________________________ 108
Statistics is like a jury? ____________________________________________________________________ 109
Errors in inference _______________________________________________________________________ 109
The 0.05 alpha value is not dogma __________________________________________________________ 110
Philosophy of statistical analyses ______________________________________________________________ 111
Statistics is not like a trial __________________________________________________________________ 111
Practical significance and statistical significance________________________________________________ 111
p-values and effect sizes __________________________________________________________________ 112
Preplanned tests _________________________________________________________________________ 113
Exploratory data analysis __________________________________________________________________ 114
p-value hacking __________________________________________________________________________ 114
A few of xkcd comics ________________________________________________________________________ 114
Significant ______________________________________________________________________________ 114
Null hypothesis __________________________________________________________________________ 114
P-values ________________________________________________________________________________ 115
Experiments, sampling, and causation __________________________________________________________ 115
Types of experimental designs ______________________________________________________________ 115
Quasi-experiment designs _________________________________________________________________ 115
Observational studies _____________________________________________________________________ 115
Sampling _______________________________________________________________________________ 115
Plan ahead and be consistent ______________________________________________________________ 116
Optional discussion: Alternative methods to the Null Hypothesis Significance Test ______________________ 116
The NHST controversy ____________________________________________________________________ 117
Alternatives to the NHST approach __________________________________________________________ 118
References and further reading _______________________________________________________________ 118
Exercises F ________________________________________________________________________________ 119
Reporting Results of Data and Analyses _________________________________________________ 121
Packages used in this chapter _________________________________________________________________ 122
Reporting analyses, results, and assumptions ____________________________________________________ 122
Procedures _____________________________________________________________________________ 122
Results _________________________________________________________________________________ 122
Notes on different data and analyses ________________________________________________________ 123
Advice on tables and plots ___________________________________________________________________ 123
Plots __________________________________________________________________________________ 123
Tables _________________________________________________________________________________ 123
Table headings and plot captions ___________________________________________________________ 124
Example description of statistical analysis and results _____________________________________________ 124
Procedures _____________________________________________________________________________ 124
Results _________________________________________________________________________________ 125
Optional technical note on effect sizes _______________________________________________________ 125
References ________________________________________________________________________________ 126
Choosing a Statistical Test ____________________________________________________________ 126
Plan your experimental design before you collect data ____________________________________________ 126
What is the hypothesis? _____________________________________________________________________ 127
What number and type of variables do you have? ________________________________________________ 127
References ________________________________________________________________________________ 134
Optional readings __________________________________________________________________________ 134
Independent and Paired Values _______________________________________________________ 134

ix
 Packages used in this chapter _________________________________________________________________ 135
An example of paired and unpaired data ________________________________________________________ 135
Box plot and summary statistics by group _____________________________________________________ 136
Bar plot to show paired differences__________________________________________________________ 137
Paired t-test and unpaired test _____________________________________________________________ 138
Histogram of differences with normal curve ___________________________________________________ 138

Likert Data ___________________________________________________________________ 140

Introduction to Likert Data ___________________________________________________________ 140
Likert data ________________________________________________________________________________ 140
Numbers of responses ____________________________________________________________________ 140
Symmetry ______________________________________________________________________________ 140
Neutral responses ________________________________________________________________________ 140
Form of responses _______________________________________________________________________ 140
Opt-out answers _________________________________________________________________________ 141
Examples of Likert item responses __________________________________________________________ 141
Likert items and Likert scales _________________________________________________________________ 141
Analysis of Likert data _______________________________________________________________________ 141
Likert data should be treated as ordinal data __________________________________________________ 142
Some people treat Likert data as interval/ratio data ____________________________________________ 142
Analysis of Likert scale data ________________________________________________________________ 142
Analysis of Likert data ____________________________________________________________________ 143
Optional reading ___________________________________________________________________________ 143
Descriptive Statistics for Likert Data ____________________________________________________ 143
Median ________________________________________________________________________________ 143
Count of responses and bar plot ____________________________________________________________ 144
Quartiles and percentiles __________________________________________________________________ 144
Packages used in this chapter _________________________________________________________________ 144
Descriptive statistics for one-sample data _______________________________________________________ 145
One-sample data ________________________________________________________________________ 145
Variables and functions in this example ______________________________________________________ 145
Summary treating Likert data as nominal data _________________________________________________ 146
Summary treating Likert data as numeric data _________________________________________________ 147
Descriptive statistics for one-way or multi-way data_______________________________________________ 149
One-way data ___________________________________________________________________________ 149
Multi-way data __________________________________________________________________________ 149
Variables and functions in this example ______________________________________________________ 149
Summary treating Likert data as nominal data _________________________________________________ 151
Summary treating Likert data as numeric data _________________________________________________ 154
Interaction plot using medians and quartiles __________________________________________________ 157
Interaction plot using medians and confidence intervals _________________________________________ 159
Descriptive statistics with opt-out responses ____________________________________________________ 160
Analysis for vector data ___________________________________________________________________ 160
Analysis for long-form data frame ___________________________________________________________ 161
Summary for whole data frame _____________________________________________________________ 162
Exercises G ________________________________________________________________________________ 163
Descriptive Statistics with the likert Package _____________________________________________ 164
Packages used in this chapter _________________________________________________________________ 164
The likert package __________________________________________________________________________ 164
Summary statistics and plots with the likert package ______________________________________________ 165
Plots that treat Likert data like factor data ____________________________________________________ 166

x
 Plots that treat Likert data like numeric data __________________________________________________ 168
References ________________________________________________________________________________ 169
Confidence Intervals for Medians ______________________________________________________ 169
Packages used in this chapter _________________________________________________________________ 169
Medians, quantiles, and confidence intervals for one-sample data ___________________________________ 170
Produce median with median, summary, and Summarize functions ________________________________ 171
Confidence interval for medians with the wilcox.test function ____________________________________ 172
Confidence interval for median by bootstrap __________________________________________________ 172
Median and confidence interval with the DescTools package _____________________________________ 173
groupwiseMedian function to produce medians and confidence intervals ___________________________ 173
Medians, quantiles, and confidence intervals for grouped data ______________________________________ 174
Summarize function in FSA package for grouped data ___________________________________________ 175
groupwiseMedian function to produce medians and confidence intervals for grouped data ____________ 175
References ________________________________________________________________________________ 176
Exercises H ________________________________________________________________________________ 176
Converting Numeric Data to Categories _________________________________________________ 177
Categorizing data by a range of values __________________________________________________________ 177
Categorizing data by percentiles _______________________________________________________________ 178
Categorizing data with clustering ______________________________________________________________ 178
Packages used in this chapter _________________________________________________________________ 178
Examples for converting numeric data to categories ______________________________________________ 178
Categorize data by range of values __________________________________________________________ 179
Categorize data by percentile ______________________________________________________________ 181
Categorize data by clustering _______________________________________________________________ 182

Traditional Nonparametric Tests __________________________________________________ 188

Introduction to Traditional Nonparametric Tests__________________________________________ 188
Nonparametric test assumptions ______________________________________________________________ 188
Advantages of nonparametric tests ____________________________________________________________ 188
Interpretation of nonparametric tests __________________________________________________________ 188
Using permutation tests and ordinal regression for ordinal data _____________________________________ 189
Using traditional nonparametric tests with ordinal data ____________________________________________ 189
Using traditional nonparametric tests with interval/ratio data ______________________________________ 189
References ________________________________________________________________________________ 189
One-sample Wilcoxon Signed-rank Test _________________________________________________ 190
Packages used in this chapter _________________________________________________________________ 190
One-sample Wilcoxon signed-rank test example __________________________________________________ 191
Summarize data treating Likert scores as factors _______________________________________________ 192
Summarize data treating Likert scores as numeric ______________________________________________ 193
One-sample Wilcoxon signed-rank test _______________________________________________________ 193
Exercises I_________________________________________________________________________________ 193
Sign Test for One-sample Data ________________________________________________________ 195
Packages used in this chapter _________________________________________________________________ 195
One-sample sign test example ________________________________________________________________ 196
Sign test with the BSDA package ______________________________________________________________ 196
Sign test with the DescTools package ___________________________________________________________ 197
Two-sample Mann–Whitney U Test ____________________________________________________ 197
When to use this test _______________________________________________________________________ 197
Packages used in this chapter _________________________________________________________________ 199

xi
 Two-sample Mann–Whitney U test example _____________________________________________________ 199
Summarize data treating Likert scores as factors _______________________________________________ 200
Histograms of data by group _______________________________________________________________ 201
Summarize data treating Likert scores as numeric ______________________________________________ 201
Two-sample Mann–Whitney U test example __________________________________________________ 201
Exercises J ________________________________________________________________________________ 202
Mood’s Median Test for Two-sample Data ______________________________________________ 203
Packages used in this chapter _________________________________________________________________ 204
Example using the RVAideMemoire package _____________________________________________________ 204
Mood’s Median Test ________________________________________________________________________ 205
Two-sample Paired Rank-sum Test _____________________________________________________ 205
When to use this test _______________________________________________________________________ 205
Packages used in this chapter _________________________________________________________________ 206
Two-sample paired rank-sum test example ______________________________________________________ 206
Plot the paired data ______________________________________________________________________ 208
Two-sample paired sign-rank test ___________________________________________________________ 210
Exercises K ________________________________________________________________________________ 211
Sign Test for Two-sample Paired Data __________________________________________________ 212
Packages used in this chapter _________________________________________________________________ 212
Sign test for paired two-sample data example ___________________________________________________ 213
Two-sample sign test with BSDA package _____________________________________________________ 214
Two-sample sign test with DescTools package _________________________________________________ 214
Kruskal–Wallis Test _________________________________________________________________ 215
Packages used in this chapter _________________________________________________________________ 216
Kruskal–Wallis test example __________________________________________________________________ 216
Summarize data treating Likert scores as factors _______________________________________________ 218
Histograms by group _____________________________________________________________________ 218
Summarize data treating Likert scores as numeric ______________________________________________ 219
Kruskal–Wallis test example _______________________________________________________________ 219
Post-hoc test: Dunn test for multiple comparisons of groups _____________________________________ 220
Post-hoc test: pairwise Mann–Whitney U-tests for multiple comparisons ___________________________ 221
Plot of medians and confidence intervals _______________________________________________________ 223
Exercises L ________________________________________________________________________________ 224
Mood’s Median Test ________________________________________________________________ 226
Packages used in this chapter _________________________________________________________________ 227
Example using the RVAideMemoire package _____________________________________________________ 227
Mood’s median test ______________________________________________________________________ 228
Post-hoc test: pairwiseMedianTest function ___________________________________________________ 229
Post-hoc test: pairwiseMedianMatrix function _________________________________________________ 230
Friedman Test______________________________________________________________________ 231
Packages used in this chapter _________________________________________________________________ 233
Friedman test example ______________________________________________________________________ 233
Summarize data treating Likert scores as factors _______________________________________________ 235
Histograms by group _____________________________________________________________________ 235
Bar plots of differences between groups______________________________________________________ 236
pairwiseDifferences to produce bar plots of differences between all groups _________________________ 237
Summarize data treating Likert scores as numeric ______________________________________________ 238
Friedman test example ____________________________________________________________________ 239
Post-hoc test: pairwise sign test for multiple comparisons of groups _______________________________ 239

xii
 Post-hoc Conover test ____________________________________________________________________ 241
Quade Test ________________________________________________________________________ 242
Packages used in this chapter _________________________________________________________________ 243
Quade test example ________________________________________________________________________ 244
Summarize data treating Likert scores as factors _______________________________________________ 245
Histograms by group _____________________________________________________________________ 246
Summarize data treating Likert scores as numeric ______________________________________________ 247
Quade test example ______________________________________________________________________ 248
Post-hoc test: pairwise two-sample paired rank-sum test for multiple comparisons ___________________ 248
Post-hoc test: Quade post-hoc test __________________________________________________________ 249
Nonparametric Regression ___________________________________________________________ 250
Packages used in this chapter _________________________________________________________________ 250
Nonparametric correlation ___________________________________________________________________ 251
Nonparametric regression examples ___________________________________________________________ 251
Kendall–Theil Sen Siegel nonparametric linear regression __________________________________________ 252
Plot with statistics________________________________________________________________________ 253
Quantile regression _________________________________________________________________________ 254
Plot with statistics________________________________________________________________________ 255
Local regression ____________________________________________________________________________ 256
Plot ___________________________________________________________________________________ 257
Generalized additive models__________________________________________________________________ 257
Plot with statistics________________________________________________________________________ 258
Nonparametric Regression for Time Series ______________________________________________ 259
Packages used in this chapter _________________________________________________________________ 260
Nonparametric regression examples ___________________________________________________________ 260
Decomposing time series objects ______________________________________________________________ 262
Mann–Kendall trend test ____________________________________________________________________ 263
Sen’s slope for time series data _______________________________________________________________ 264
Pettitt’s test for change in value _______________________________________________________________ 264

Permutation Tests for Ordinal Data _______________________________________________ 266

Introduction to Permutation Tests _____________________________________________________ 266
The coin package ________________________________________________________________________ 266
References ________________________________________________________________________________ 270
Packages used in this chapter _________________________________________________________________ 267
Permutation test example ___________________________________________________________________ 267
Box plot ________________________________________________________________________________ 268
Scatter plot with one-to-one line ____________________________________________________________ 269
Permutation test of independence __________________________________________________________ 269
Permutation test of symmetry ______________________________________________________________ 270
One-way Permutation Test of Independence for Ordinal Data _______________________________ 270
When to use this test _______________________________________________________________________ 270
Packages used in this chapter _________________________________________________________________ 271
One-way ordinal permutation test example _____________________________________________________ 272
Summarize data treating Likert scores as factors _______________________________________________ 273
Histograms by group _____________________________________________________________________ 274
Summarize data treating Likert scores as numeric ______________________________________________ 274
One-way ordinal permutation test __________________________________________________________ 275
Post-hoc test: pairwise permutation tests ____________________________________________________ 275
Plot of medians and confidence intervals _____________________________________________________ 277

xiii
 Comparison of methods for independence tests for ordinal data ____________________________________ 278
One-way Permutation Test of Symmetry for Paired Ordinal Data ____________________________ 279
When to use this test _______________________________________________________________________ 279
Packages used in this chapter _________________________________________________________________ 280
One-way ordinal permutation test of symmetry example __________________________________________ 280
Summarize data treating Likert scores as factors _______________________________________________ 282
Histograms by group _____________________________________________________________________ 282
Summarize data treating Likert scores as numeric ______________________________________________ 283
Permutation symmetry test ________________________________________________________________ 283
Post-hoc test: pairwise permutation tests ____________________________________________________ 284
Plot of medians and confidence intervals _____________________________________________________ 286
Comparison of methods for symmetry tests for ordinal data ________________________________________ 288

Concepts for Linear Models ______________________________________________________ 289

Introduction to Linear Models_________________________________________________________ 289
Types of linear models ______________________________________________________________________ 289
Formulae for specifying models in R ____________________________________________________________ 290
Data in data frames ______________________________________________________________________ 290
Random effects __________________________________________________________________________ 291
Extracting model information from R ___________________________________________________________ 291
References ________________________________________________________________________________ 292
Using Random Effects in Models _______________________________________________________ 292
Packages used in this chapter _________________________________________________________________ 293
An example of a mixed model ________________________________________________________________ 294
Mixed model with lmer ___________________________________________________________________ 295
Mixed model with nlme ___________________________________________________________________ 296
What are Least Square Means? ________________________________________________________ 297
Packages used in this chapter _________________________________________________________________ 298
Least square means example _________________________________________________________________ 298
Least square means ______________________________________________________________________ 299
Least Square Means for Multiple Comparisons ___________________________________________ 299
p-value adjustments for multiple comparisons _________________________________________________ 300
Ignore values of lsmeans for clm and clmm models _____________________________________________ 300
Packages used in this chapter _________________________________________________________________ 300
Multiple comparisons with lsmeans ____________________________________________________________ 300
Factorial ANOVA: Main Effects, Interaction Effects, and Interaction Plots ______________________ 303
Packages used in this chapter _________________________________________________________________ 304
Two-way ANOVA example with interaction effect ________________________________________________ 305
Simple interaction plot ____________________________________________________________________ 306
Specify the linear model and conduct an analysis of variance _____________________________________ 307
Post-hoc testing with lsmeans ______________________________________________________________ 307
Extended example with additional country ____________________________________________________ 308
Simple interaction plot ____________________________________________________________________ 309
Specify the linear model and conduct an analysis of variance _____________________________________ 310
Post-hoc testing with lsmeans ______________________________________________________________ 310
Interaction plot with error bars using ggplot2 _________________________________________________ 311
Interaction plot with the phia package _______________________________________________________ 314
p-values and R-square Values for Models________________________________________________ 315
p-values for models ______________________________________________________________________ 315

xiv
 R-squared and pseudo R-squared ___________________________________________________________ 316
Packages used in this chapter _________________________________________________________________ 316
Example of model p-value, R-squared, and pseudo R-squared _______________________________________ 317
Linear model ____________________________________________________________________________ 319
Generalized linear model __________________________________________________________________ 320
Optional analyses: Confidence intervals for p-values and R-squared values ____________________________ 322
Linear model ____________________________________________________________________________ 323
Generalized linear model __________________________________________________________________ 325
References ________________________________________________________________________________ 328

Ordinal Tests with Cumulative Link Models _________________________________________ 328

Introduction to Cumulative Link Models (CLM) for Ordinal Data _____________________________ 328
The ordinal package ______________________________________________________________________ 328
Analysis of deviance ______________________________________________________________________ 329
Model assumptions for CLM _______________________________________________________________ 329
References ________________________________________________________________________________ 329
Two-sample Ordinal Test with CLM ____________________________________________________ 330
Packages used in this chapter _________________________________________________________________ 330
Two-sample ordinal model example ___________________________________________________________ 331
Summarize data treating Likert scores as factors _______________________________________________ 332
Histograms of data by group _______________________________________________________________ 332
Summarize data treating Likert scores as numeric ______________________________________________ 333
Two-sample ordinal model example _________________________________________________________ 333
Two-sample Paired Ordinal Test with CLMM _____________________________________________ 334
Packages used in this chapter _________________________________________________________________ 335
Two-sample paired ordinal model example ______________________________________________________ 335
Plot the paired data ______________________________________________________________________ 336
Two-sample paired ordinal model ___________________________________________________________ 337
One-way Ordinal ANOVA with CLM ____________________________________________________ 339
Packages used in this chapter _________________________________________________________________ 340
One-way ordinal ANOVA example _____________________________________________________________ 340
Summarize data treating Likert scores as factors _______________________________________________ 341
Histograms by group _____________________________________________________________________ 342
Summarize data treating Likert scores as numeric ______________________________________________ 342
One-way ordinal ANOVA __________________________________________________________________ 343
Optional analyses __________________________________________________________________________ 345
Post-hoc test: pairwise ordinal tests for multiple comparisons of groups ____________________________ 345
One-way Repeated Ordinal ANOVA with CLMM __________________________________________ 346
Packages used in this chapter _________________________________________________________________ 347
One-way repeated ordinal ANOVA example _____________________________________________________ 347
Summarize data treating Likert scores as factors _______________________________________________ 349
Histograms by group _____________________________________________________________________ 349
Summarize data treating Likert scores as numeric ______________________________________________ 350
One-way repeated ordinal ANOVA __________________________________________________________ 350
Post-hoc comparisons with lsmeans _________________________________________________________ 352
Optional analyses __________________________________________________________________________ 353
Post-hoc test: pairwise paired ordinal tests for multiple comparisons ______________________________ 353
Two-way Ordinal ANOVA with CLM ____________________________________________________ 355
Packages used in this chapter _________________________________________________________________ 356

xv
 Two-way ordinal ANOVA example _____________________________________________________________ 357
Summarize data treating Likert scores as factors _______________________________________________ 358
Histograms by group _____________________________________________________________________ 359
Summarize data treating Likert scores as numeric ______________________________________________ 362
Produce interaction plot with medians and quantiles ___________________________________________ 363
Two-way ordinal ANOVA __________________________________________________________________ 364
Post-hoc tests with lsmeans for multiple comparisons of groups __________________________________ 365
Optional analyses __________________________________________________________________________ 368
Post-hoc test: pairwise ordinal tests for multiple comparisons of groups ____________________________ 368
Two-way Repeated Ordinal ANOVA with CLMM __________________________________________ 371
Packages used in this chapter _________________________________________________________________ 372
Two-way repeated ordinal ANOVA example _____________________________________________________ 373
Summarize data treating Likert scores as factors _______________________________________________ 374
Histograms by group _____________________________________________________________________ 375
Summarize data treating Likert scores as numeric ______________________________________________ 376
Plot Likert scores before and after___________________________________________________________ 377
Produce interaction plot with medians and confidence intervals __________________________________ 378
Two-way repeated ordinal ANOVA __________________________________________________________ 379
Post-hoc tests with lsmeans for multiple comparisons of groups __________________________________ 382
Interaction plot with group separation letters _________________________________________________ 383

Tests for Nominal Data _________________________________________________________ 385

Introduction to Tests for Nominal Variables _____________________________________________ 385
Descriptive statistics and plots for nominal data __________________________________________________ 385
Contingency tables _________________________________________________________________________ 385
Bar plots _______________________________________________________________________________ 386
Mosaic plots ____________________________________________________________________________ 387
Optional analyses: converting long-format data to a matrix _________________________________________ 388
Long-format with each row as an observations ________________________________________________ 388
Long-format with counts of observations _____________________________________________________ 389
Optional analyses: obtaining information about a matrix object _____________________________________ 390
Confidence Intervals for Proportions ___________________________________________________ 390
Packages used in this chapter _________________________________________________________________ 391
Example of confidence intervals for a binomial proportion _________________________________________ 391
Example of confidence intervals for a multinomial proportion _______________________________________ 393
Optional analysis: confidence intervals for a difference in proportions ________________________________ 394
References ________________________________________________________________________________ 395
Goodness-of-Fit Tests for Nominal Variables _____________________________________________ 395
Packages used in this chapter _________________________________________________________________ 396
Goodness-of-fit tests for nominal variables example ______________________________________________ 396
Exact tests for goodness-of-fit ______________________________________________________________ 397
G-test for goodness-of-fit __________________________________________________________________ 398
Chi-square test for goodness-of-fit __________________________________________________________ 398
Multinomial test example with plot and confidence intervals _______________________________________ 399
Optional readings __________________________________________________________________________ 402
References ________________________________________________________________________________ 402
Association Tests for Nominal Variables _________________________________________________ 402
Low cell counts __________________________________________________________________________ 402
Fisher’s exact test ________________________________________________________________________ 403
Choosing among tests ____________________________________________________________________ 403

xvi
 Packages used in this chapter _________________________________________________________________ 404
Association tests for nominal variables example __________________________________________________ 404
Reading the data as a matrix _______________________________________________________________ 404
Fisher Exact test of association ________________________________________________________________ 404
Post-hoc analysis ________________________________________________________________________ 405
G-test of association ________________________________________________________________________ 407
Post-hoc analysis ________________________________________________________________________ 407
Chi-square test of association _________________________________________________________________ 408
Post-hoc analysis ________________________________________________________________________ 408
Optional readings __________________________________________________________________________ 409
References ________________________________________________________________________________ 409
Exercises M _______________________________________________________________________________ 409
Tests for Paired Nominal Data_________________________________________________________ 410
Packages used in this chapter _________________________________________________________________ 411
McNemar and McNemar–Bowker tests ______________________________________________________ 411
Exact tests ______________________________________________________________________________ 411
Example of tests for paired data nominal data ___________________________________________________ 412
Exact tests for symmetry __________________________________________________________________ 413
McNemar and McNemar–Bowker chi-square tests for symmetry __________________________________ 414
An example without repeated measures, comparing test of symmetry with test of association __________ 414
Optional analysis: a 4 x 4 example with several 0’s _____________________________________________ 415
Optional analyses: conducting exact tests for symmetry _________________________________________ 417
Exercises N ________________________________________________________________________________ 419
Cochran–Mantel–Haenszel Test for 3-Dimensional Tables __________________________________ 420
Packages used in this chapter _________________________________________________________________ 421
C–M–H test example: long-format with counts ___________________________________________________ 421
C–M–H test example: table format ____________________________________________________________ 424
Cochran’s Q Test for Paired Nominal Data _______________________________________________ 424
Packages used in this chapter _________________________________________________________________ 426
Cochran’s Q test example: long-format _________________________________________________________ 426
Cochran’s Q test example: short-format in matrix ________________________________________________ 432
Association Tests for Ordinal Tables ____________________________________________________ 434
Packages used in this chapter _________________________________________________________________ 434
Linear-by-linear test for ordered contingency tables ______________________________________________ 435
Example of linear-by-linear test 1 ___________________________________________________________ 435
Example of linear-by-linear test 2 ___________________________________________________________ 437
Extended Cochran–Armitage test ______________________________________________________________ 437
Example of extended Cochran–Armitage test 1 ________________________________________________ 438
Example of extended Cochran–Armitage test 2 ________________________________________________ 440
Long-form data and three-dimensional contingency tables _________________________________________ 441
Two-dimensional table example ____________________________________________________________ 443
Three-dimensional table example ___________________________________________________________ 443
References ________________________________________________________________________________ 444
Models for Nominal Data ____________________________________________________________ 445
Packages used in this chapter _________________________________________________________________ 445
Log-linear models for tests of association _______________________________________________________ 445
Log-linear model example _________________________________________________________________ 446
Logistic regression __________________________________________________________________________ 448
Logistic regression example ________________________________________________________________ 449
Multinomial logistic regression ________________________________________________________________ 453

xvii
 VGAM package __________________________________________________________________________ 453
mlogit and nnet packages _________________________________________________________________ 453
Multinomial regression example ____________________________________________________________ 454
Mixed-effects logistic regression example _______________________________________________________ 457
Other tools for categorical analysis ____________________________________________________________ 461

Parametric Tests _______________________________________________________________ 463

Introduction to Parametric Tests ______________________________________________________ 463
Packages used in this chapter _________________________________________________________________ 463
When to use parametric tests _________________________________________________________________ 463
Count data may not be appropriate for common parametric tests _________________________________ 463
Percentage and proportion data ____________________________________________________________ 464
Assumptions in parametric statistics ___________________________________________________________ 464
Random sampling ________________________________________________________________________ 465
Independent observations _________________________________________________________________ 465
Normal distribution of data or residuals ______________________________________________________ 465
Homogeneity of variance __________________________________________________________________ 467
Outliers ________________________________________________________________________________ 468
Parametric tests are somewhat robust _______________________________________________________ 468
Assessing model assumptions _________________________________________________________________ 469
Using formal tests to assess normality of residuals _____________________________________________ 470
Skew and kurtosis ________________________________________________________________________ 470
Using visual inspection to assess the normality of residuals ______________________________________ 470
Steps to handle violations of assumptions ____________________________________________________ 476
Descriptive Statistics for Parametric Statistics ____________________________________________________ 477
Optional readings __________________________________________________________________________ 477
References ________________________________________________________________________________ 477
Optional analyses: formal tests for normality ____________________________________________________ 477
Shapiro–Wilk normality test________________________________________________________________ 479
Anderson-Darling normality test ____________________________________________________________ 479
One-sample Kolmogorov-Smirnov test _______________________________________________________ 479
D'Agostino Normality Test _________________________________________________________________ 479
Optional analyses: formal tests for homogeneity of variance ________________________________________ 480
Levene’s test for homogeneity of variance ____________________________________________________ 481
Brown–Forsythe or robust Levene’s test ______________________________________________________ 482
Fligner-Killeen test _______________________________________________________________________ 482
One-sample t-test __________________________________________________________________ 483
Packages used in this chapter _________________________________________________________________ 483
One-sample t-test example ___________________________________________________________________ 483
Histogram of data ________________________________________________________________________ 484
Normal quantile plot of data _______________________________________________________________ 485
One-sample t-test ________________________________________________________________________ 485
Reference’s _______________________________________________________________________________ 486
Exercises O ________________________________________________________________________________ 486
Two-sample t-test __________________________________________________________________ 487
Packages used in this chapter _________________________________________________________________ 488
Two-sample t-test example __________________________________________________________________ 488
Summarize data by group _________________________________________________________________ 489
Histograms for data by group ______________________________________________________________ 490
Box plots for data by group ________________________________________________________________ 490
Two-sample unpaired t-test ________________________________________________________________ 491

xviii
 Optional readings __________________________________________________________________________ 491
References ________________________________________________________________________________ 491
Exercises P ________________________________________________________________________________ 491
Paired t-test _______________________________________________________________________ 493
Packages used in this chapter _________________________________________________________________ 493
Paired t-test example _______________________________________________________________________ 493
Histogram of difference data _______________________________________________________________ 495
Plot the paired data ______________________________________________________________________ 495
Paired t-test ____________________________________________________________________________ 497
Optional readings __________________________________________________________________________ 498
References ________________________________________________________________________________ 498
Exercises Q ________________________________________________________________________________ 498
One-way ANOVA ___________________________________________________________________ 499
Packages used in this chapter _________________________________________________________________ 500
One-way ANOVA example ___________________________________________________________________ 500
Summarize data by group _________________________________________________________________ 502
Box plots for data by group ________________________________________________________________ 503
Plot of means and confidence intervals_______________________________________________________ 503
Define linear model ______________________________________________________________________ 504
Conduct analysis of variance _______________________________________________________________ 504
Histogram of residuals ____________________________________________________________________ 505
Post-hoc analysis: mean separation tests _____________________________________________________ 505
References ________________________________________________________________________________ 507
Exercises R ________________________________________________________________________________ 507
One-way ANOVA with Blocks _________________________________________________________ 509
Packages used in this chapter _________________________________________________________________ 510
One-way ANOVA with blocks example __________________________________________________________ 510
Use Summarize to check if cell sizes are balanced ______________________________________________ 512
Define linear model ______________________________________________________________________ 513
Conduct analysis of variance _______________________________________________________________ 513
Histogram and plot of residuals _____________________________________________________________ 513
Post-hoc analysis: mean separation tests _____________________________________________________ 514
Plot of LS means and confidence intervals ____________________________________________________ 515
Exercises S ________________________________________________________________________________ 516
One-way ANOVA with Random Blocks __________________________________________________ 518
Packages used in this chapter _________________________________________________________________ 519
One-way ANOVA with random blocks example ___________________________________________________ 519
Mixed-effects model with lmer _____________________________________________________________ 521
Mixed-effects model with nlme _____________________________________________________________ 524
Comparison of results from one-way ANOVA with blocks __________________________________________ 526
Two-way ANOVA ___________________________________________________________________ 527
Packages used in this chapter _________________________________________________________________ 528
Two-way ANOVA example ___________________________________________________________________ 528
Use Summarize to check if cell sizes are balanced ______________________________________________ 530
Define linear model ______________________________________________________________________ 530
Conduct analysis of variance _______________________________________________________________ 531
Histogram of residuals ____________________________________________________________________ 531
Post-hoc analysis: mean separation tests _____________________________________________________ 532
Plot of means and confidence intervals for main effects _________________________________________ 533
Plot of LS means and confidence intervals for interaction ________________________________________ 535

xix
 Exercises T ________________________________________________________________________________ 537
Repeated Measures ANOVA __________________________________________________________ 538
Packages used in this chapter _________________________________________________________________ 539
Repeated measures ANOVA example___________________________________________________________ 539
Optional analysis: determining autocorrelation in residuals _________________________________________ 547
Optional discussion on specifying formulae for repeated measures analysis ___________________________ 548
Specifying random effects in models _________________________________________________________ 548
Indicating time and subject variables ________________________________________________________ 548
Correlation and Linear Regression _____________________________________________________ 549
Packages used in this chapter _________________________________________________________________ 550
Examples for correlation and linear regression ___________________________________________________ 550
Visualizing correlated variables _____________________________________________________________ 552
Pearson correlation ______________________________________________________________________ 555
Kendall correlation _______________________________________________________________________ 557
Spearman correlation _____________________________________________________________________ 557
Linear regression ________________________________________________________________________ 558
Polynomial regression ____________________________________________________________________ 561
A few of xkcd comics ________________________________________________________________________ 565
Correlation _____________________________________________________________________________ 565
Extrapolating ____________________________________________________________________________ 565
Cat proximity ___________________________________________________________________________ 565
Exercises U ________________________________________________________________________________ 565
References ________________________________________________________________________________ 567
Advanced Parametric Methods ________________________________________________________ 567
Packages used in this chapter _________________________________________________________________ 567
More complex experimental designs ___________________________________________________________ 568
Analysis of co-variance ______________________________________________________________________ 568
Nonlinear regression and curvilinear regression __________________________________________________ 568
Multiple regression _________________________________________________________________________ 569
Logistic regression __________________________________________________________________________ 569
Analysis of count data _______________________________________________________________________ 569
Robust techniques __________________________________________________________________________ 569
Example of robust linear regression _________________________________________________________ 570
Contrasts in Linear Models ___________________________________________________________________ 574
Linear plateau and quadratic plateau models ____________________________________________________ 574
Examples of linear plateau and quadratic plateau models ________________________________________ 574
Cate–Nelson analysis ________________________________________________________________________ 587
Transforming Data __________________________________________________________________ 588
Transforming data __________________________________________________________________________ 589
Packages used in this chapter _________________________________________________________________ 589
Example of transforming skewed data __________________________________________________________ 590
Square root transformation ________________________________________________________________ 591
Cube root transformation _________________________________________________________________ 591
Log transformation _______________________________________________________________________ 592
Tukey’s Ladder of Powers transformation _______________________________________________________ 593
Example of Tukey-transformed data in ANOVA ________________________________________________ 594
Box–Cox transformation _____________________________________________________________________ 599
Box–Cox transformation for a single variable __________________________________________________ 600
Example of Box–Cox transformation for ANOVA model __________________________________________ 601
Conclusions _______________________________________________________________________________ 606

xx
Analysis of Count Data and Percentage Data ________________________________________ 607
Hermite and Poisson Regression for Count Data __________________________________________ 607
Introduction _______________________________________________________________________________ 607
Count data _____________________________________________________________________________ 607
Regression approaches for count data _______________________________________________________ 607
Cautionary note _________________________________________________________________________ 607
Generalized linear regression ______________________________________________________________ 607
Packages used in this chapter _________________________________________________________________ 607
Hermite regression _________________________________________________________________________ 608
Hermite regression example _______________________________________________________________ 608
Hermite regression _______________________________________________________________________ 610
Histograms _____________________________________________________________________________ 611
Post-hoc analysis: Medians and confidence intervals ____________________________________________ 611
Post-hoc analysis: Pairwise models __________________________________________________________ 613
Poisson regression example __________________________________________________________________ 615
Negative binomial regression example _________________________________________________________ 616
Zero-inflated regression example ______________________________________________________________ 618
Robust Poisson regression example ____________________________________________________________ 619
Quasi-Poisson regression ____________________________________________________________________ 620
Optional code for chi-square goodness-of-fit test _________________________________________________ 621
Omnibus test ___________________________________________________________________________ 621
Post-hoc chi-square tests __________________________________________________________________ 622
Optional analysis: Vuong test to compare Poisson, negative binomial, and zero-inflated models ___________ 622
References ________________________________________________________________________________ 624
References for count data ____________________________________________________________________ 625
Beta Regression for Percent and Proportion Data _________________________________________ 625
Introduction _______________________________________________________________________________ 625
Proportion data _________________________________________________________________________ 625
Beta regression __________________________________________________________________________ 626
Packages used in this chapter _________________________________________________________________ 627
Beta regression example with discrete counts ___________________________________________________ 627
Beta regression example __________________________________________________________________ 628
Alternate logistic regression________________________________________________________________ 630
Beta regression example with inherently proportional data _________________________________________ 631
Beta regression __________________________________________________________________________ 633
References ________________________________________________________________________________ 637

xxi
 PURPOSE OF THIS BOOK SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Introduction

Purpose of This Book

Goals of this book

The goal of this book is to introduce to students interested in extension education, outreach, and
public education to the quantitative methods used to assess the evaluation of these activities.
Extension education includes a diverse collection of subject matter, including environmental science,
home horticulture, agriculture, youth development, nutrition, and financial literacy.

Tools for evaluating educational programs may include in-class surveys that measure the knowledge
gain of students in a course or follow-up surveys to determine the behaviors adopted by course
participants. Evaluation may also include any number of measured variables, perhaps the age of youth
participants, the number of calories eaten daily by students in a nutrition program, or the organic
matter content of farm fields managed by participating farmers.

The examples and methods here are chosen specifically to be applicable to the evaluation of extension
education programs. That being said, these methods are some of the most common used in the
analysis of experiments—techniques used from diverse disciplines from manufacturing to
environmental science to psychology, though each of these disciplines has additional methods used in
specific situations.

Specific learning goals

One goal of this book is to give readers the skills and abilities to be able to understand the graphs and
statistics that you might encounter in a publication such as the Journal of Extension or other academic
reports of program results. As examples, students will be able to answer the questions: What can I
conclude from this bar plot? How do I interpret this p-value? What is an r-squared value?

A second goal is for readers to be able to design and analyze their own program evaluation
experiments in order to document the impacts of their extension teaching or research. What analysis
would I use to assess knowledge gain with before-and-after surveys? What statistics should I report to
convey the results of this analysis? Can I explain the results with a graph?

Pre-requisites

This book is written for students at the undergraduate level with no prior knowledge of the analysis of
experiments, and with no prior knowledge of computer programming. This being said, students with
no background in these areas will need to apply care and dedication in order to understand the
material and the computer code used in examples. These students may also need to explore the
optional readings to obtain a better foundation in statistical thinking and theory.

What this book will not cover

1
 ABOUT THE AUTHOR OF THIS BOOK SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Designing and conducting surveys

There are many skills and considerations that go into conducting competent assessments of education
programs. This book will not cover these many of these topics in any depth. For example, good survey
design and effective survey questions will be touched on only very briefly. Conducting surveys well,
for example by avoiding sampling bias, will also not be covered in any significant way.

Advanced statistical analyses

There are variety of relatively advanced statistical analyses that are used in even relatively simple
studies. This book focuses on only the most basic analyses for common designs used in extension
evaluation. A solid understanding of these analyses will give the reader the foundation for exploring
more complicated analyses as the student wishes or the situation calls for.

About the Author of this Book

This book is written with the intention of providing users simple and complete examples of analyses
common in the evaluation of extension education programs using the R Project for Statistical
Computing. I have chosen the computer code to both be as clear as possible for a beginner to follow,
but also to demonstrate some of the options available in conducting analyses or producing plots of
data. In some cases the code is more difficult to follow than in others.

Likewise, discussion of the assumptions and theoretical considerations for the statistical analyses
included here is limited. Readers are encouraged to understand these considerations in more depth
when using these analyses.

I am neither a statistician nor an R programmer, so all advice and code in this book comes without
guarantee. I am happy to accept corrections or suggestions. Send correspondence to
[email protected].

Using R
R and RStudio

This book will use the software package R Project for Statistical Computing to create plots and conduct
statistical analyses. It is free to install on a Windows, Mac, or Linux computer. Although it is not
required, I also recommend using RStudio, which is also free.

Links to websites where the software can be obtained are included in the “Obtaining R” section in the
“Required Readings” below.

2
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Using the RStudio environment

RStudio provides a nice work environment because it presents several windows on the screen that
make it easy to view code, results, and plots at once.

Program code can be worked on in the upper left Script window. If that window isn’t displayed, it can
be opened with File > New File > R Script. Code in the Script window is selected, and the Run button
is used to run the code. The results are reported in the Console window on the lower right. Code can
be saved as an .r or .R file, and those files should subsequently open automatically with RStudio.

As an alternative, code can be pasted directly in the Console window.

The lower right window will usually show either plot results or help results.

Installation
As far as I know, R should be installed first. And then when RStudio is installed, you will tell RStudio
where R is installed on the machine. Links to obtain this software are in the Obtaining R in the
Required Readings below.

Using the R Console environment

If you are not using RStudio, code is pasted directly into the R GUI Console. Results are produced in the
Console as code is entered, and plots will open in a separate window.

What if I don’t have my own computer?

If you don’t have your own computer on which to install the software, there are a few options.

Portable installation on a usb drive

One solution is to install R Portable on a portable usb drive. You can then run this software on
university or other computers directly from that drive, if the computer is set up to give you permission
to do so. You should check to see if you will have permission to run portable software from a usb drive
on these computers.

Using university computers

R and RStudio are installed on Rutgers University computers in computer laboratories. You should
check with the individual computer lab, and make sure you will have permission to install additional R
packages on those machines.

Using R online
There are websites on which you can run R in an online environment. r-fiddle.org is one such site.
However, I have had trouble using some R packages used in this book on r-fiddle.

apps.rutgers.edu
R and RStudio are also included in the apps.rutgers.edu environment. You can log in at
apps.rutgers.edu with your Rutgers Net ID. Then, Desktop > Menu > Development > R Studio. In

3
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

general, I have not found this to be the most convenient environment to work in. Working on a laptop,
I have found that I had to zoom out with the browser zoom to see the whole virtual desktop on my
screen. To transfer text to the desktop environment, paste the text into the clipboard using the
clipboard icon at the upper right of the screen. Also, I have had trouble trying to install additional R
packages in this environment.

Tests for package installation

If you are unsure if you can install additional R packages in the environment you are working in, try
the two examples below. The psych and FSA packages may take a while for their initial installation.
The code for you to run is in blue, and the output is in red. The output is truncated here.

Don’t worry too much about what the code is doing at this point. The main point is to see if you can
get output from both the psych and FSA packages.

The code assigns a vector of numbers to Score, and a vector of text strings to Student. It then combines
those two into a data frame called Data, which is then printed. The summary function counts the
values in Student, and determines the median and other statistics for Score. The psych package is
installed, then loaded with the library function, and then is used to output summary statistics for Score
for each Student. The same is then done with the FSA package.

Remember to run only the blue code. The red code is the (truncated) output R should produce.
Score = c(10, 9, 8, 7, 7, 8, 9, 10, 6, 5, 4, 9, 10, 9, 10)
Student = c("Bugs", "Bugs", "Bugs", "Bugs","Bugs",
"Daffy", "Daffy", "Daffy", "Daffy", "Daffy",
"Taz", "Taz", "Taz", "Taz", "Taz")

Data = data.frame(Student, Score)

Data

Student Score
1 Bugs 10
2 Bugs 9
3 Bugs 8
4 Bugs 7
5 Bugs 7
6 Daffy 8
7 Daffy 9
8 Daffy 10
9 Daffy 6
10 Daffy 5
11 Taz 4
12 Taz 9
13 Taz 10
14 Taz 9
15 Taz 10

summary(Data)

4
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Student Score
Bugs :5 Min. : 4.000
Daffy:5 1st Qu.: 7.000
Taz :5 Median : 9.000
Mean : 8.067
3rd Qu.: 9.500
Max. :10.000

if(!require(psych)){install.packages("psych")}

library(psych)

describeBy(x = Score,
group = Student)

group: Bugs
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 5 8.2 1.3 8 8.2 1.48 7 10 3 0.26 -1.96 0.58
------------------------------------------------------------------------
group: Daffy
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 5 7.6 2.07 8 7.6 2.97 5 10 5 -0.11 -2.03 0.93
------------------------------------------------------------------------
group: Taz
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 5 8.4 2.51 9 8.4 1.48 4 10 6 -0.97 -1.04 1.12

if(!require(FSA)){install.packages("FSA")}

library(FSA)

Summarize(Score ~ Student,
data=Data)

Student n mean sd min Q1 median Q3 max percZero

1 Bugs 5 8.2 1.303840 7 7 8 9 10 0
2 Daffy 5 7.6 2.073644 5 6 8 9 10 0
3 Taz 5 8.4 2.509980 4 9 9 10 10 0

Required readings

The following readings are required for this chapter. You can read them at the individual links below,
or as chapters in the pdf version of the R Companion to the Handbook of Biological Statistics
(rcompanion.org/documents/RCompanionBioStatistics.pdf).

About R
rcompanion.org/rcompanion/a_04.html

5
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Obtaining R
rcompanion.org/rcompanion/a_05.html

A Few Notes to Get Started with R

rcompanion.org/rcompanion/a_06.html

Avoiding Pitfalls in R
rcompanion.org/rcompanion/a_07.html

Help with R
rcompanion.org/rcompanion/a_08.html

R Tutorials
rcompanion.org/rcompanion/a_09.html

References for this chapter

Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological Statistics, version 1.09.
rcompanion.org/rcompanion/. (Pdf version:
rcompanion.org/documents/RCompanionBioStatistics.pdf.)

Exercises A

1. Install R and RStudio on your computer, or determine how you will access this software. Be sure
that you will be able to install additional R packages (as in “Tests for package installation” above).

2. Run the sample code snippets in the “Required readings” above, and in the “Tests for package
installation above”. Try understanding the code. Modify the data in the examples and examine the
output. You should be comfortable running short programs in R and examining the results before
continuing with this book.

3. If you haven't installed the FSA package, do so with the following command, or use Tools > Install
packages in RStudio.

if(!require(FSA)){install.packages("FSA")}

The package contains a data set called ChinookArg that has the lengths and weights of Chinook salmon
at three locations.

It is not necessary but the following will load the data set into memory.

library(FSA)

data(ChinookArg)

6
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

We can find some information about the data set with

?ChinookArg

We can see the first and last rows of the data with

library(FSA)

headtail(ChinookArg)

Note that the variable tl is length, w is weight, and loc is location. This is explained in the help file
pulled up with

?ChinookArg

We can then get some summary statistics about the data.

library(FSA)

Summarize(tl ~ loc,
data = ChinookArg)

This will provide summary statistics about length at each location.

In the output, n is the number of observations for that group.

This example uses formula notation, where tl is measurement variable and loc is the grouping variable,
and they are separated with a tilde. We could also think of tl as the dependent variable and loc as the
independent variable.

Some functions accept formula notation and some do not. Usually asking for help about the function
will help you determine what input is required and what other arguments can be passed to the
function, e.g

?Summarize

Answer the following:

• What is the number of observations for Petrohue?

• What is the mean length for Chinook in Petrohue?

• The minimum length in Petrohue?

• The maximum length in Petrohue?

4. Install the ggplot2 package with following command, or use Tools > Install packages in RStudio.

7
 USING R SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(ggplot2)){install.packages("ggplot2")}

The following code will plot Chinook length vs. weight. It will add a smooth line to the plot.

library(ggplot2)

qplot(x = w,
y = tl,
data = ChinookArg,
geom = c("point", "smooth"),
xlab = "Weight (kg)",
ylab = "Length (cm)",
main = "Chinook plot by Sal")

• In the code above, change "Sal" to your name.

• Export the plot and embed it in your assignment.

In RStudio, in the Plot window, you can try the Export menu, and save as image, save as pdf, or copy to
clipboard.

I tend to use the pdf option and then edit the file with Photoshop or GIMP.

If you don't use RStudio, or would like to make custom image files, you can create an image file with

png(filename = "Rplot%03d.png",
width = 4,
height = 3,
units = "in",
res = 600)

qplot(x = w,
y = tl,
data = ChinookArg,
geom = c("point", "smooth"),
xlab = "Weight (kg)",
ylab = "Length (cm)",
main = "Chinook plot by Sal")

dev.off()

You may use the following to see where the file was saved, if you didn’t specify a path in the filename
argument.

getwd()

8
 STATISTICS TEXTBOOKS AND OTHER RESOURCES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

5. In the qplot function above, switch the variable for weight, w, with the variable for location, loc.
Remove the whole line with the geom argument. And change the xlab argument to something
appropriate.

• Export the plot and embed it in your assignment.

6. There's another data set in the FSA package called WhitefishLC.

Using the command ?WhitefishLC,

• What are the units for fish length in this data set?

Using this data set, summarize the length of whitefish (tl) by their age (scale1).

• Do you observe anything interesting in these results?

Plot length vs. age for whitefish.

• Embed this plot with your assignment. Include appropriate axis labels and be sure the units
of length are correct.

Statistics Textbooks and Other Resources

This book is not intended to be a substitute for an introductory course or text in statistics. The
introductory chapters will briefly cover certain key concepts mostly through readings and videos from
external sources. Other concepts that are important to having an appreciation for the theory and use
of statistical tests are not addressed. For this understanding, readers are encouraged to pursue an
introductory undergraduate course, appropriate MOOC, or careful reading of an introductory
textbook.

Some free general statistics books

Luckily there are free resources available that cover the key concepts in an introductory statistics
course. Some of these will be suggested as recommended reading in this book.

Diez, D.M., C.D. Barr, and M. Çetinkaya-Rundel. 2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.
Types of data, plots, experimental design, sampling, probability, hypothesis testing, confidence
limits, t-test, analysis of variance, chi-square test, linear regression, multiple regression,
logistic regression.

9
 STATISTICS TEXTBOOKS AND OTHER RESOURCES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Openstax College. 2013. Introductory Statistics. Rice University.

openstaxcollege.org/textbooks/introductory-statistics.
Sampling, descriptive statistics, probability, distributions, confidence intervals, hypothesis
testing, Type I and Type II errors, t-test, chi-square, linear regression, analysis of variance.

Lane, D.M. (ed.). No date. Introduction to Statistics, v. 2.0. onlinestatbook.com/.

Descriptive statistics, plots, correlation, probability, experimental design, confidence intervals,
type I and type II errors, t-test, regression, analysis of variance, chi-square test, nonparametric
tests, power.

Stockburger, D.W. 2013. Introductory Statistics: Concepts, Models, and Applications, 3rd web edition.
www.psychstat.missouristate.edu/IntroBook3/sbk.htm.
Distributions, regression, correlation, hypothesis testing, t-test, chi-square test, analysis of
variance.

Books on statistical analyses and tests

McDonald, J.H. 2014. Handbook of Biological Statistics, 3rd edition.

www.biostathandbook.com/index.html, www.biostathandbook.com/HandbookBioStatThird.pdf
Data analysis steps, kinds of variables, probability, hypothesis testing, confounding variables,
descriptive statistics, chi-square test, t-test, analysis of variance, nonparametric tests, linear
regression, logistic regression. Analyses in SAS programming language.

Analyses in R

Online resources

Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological Statistics.

rcompanion.org/rcompanion/, rcompanion.org/documents/RCompanionBioStatistics.pdf.
Using R, descriptive statistics, chi-square test, t-test, analysis of variance, nonparametric tests,
linear regression, logistic regression. Analyses in R programming language.

Kabacoff, R. 2014. Quick-R: Accessing the Power of R. www.statmethods.net/

Data input, data management, basic statistics, advanced statistics, basic graphs, advanced
graphs.

Venables, W.N., D.M. Smith, and R Core Team. 2015. An Introduction to R. cran.r-
project.org/doc/manuals/r-release/R-intro.pdf.

Text books

Crawley, M.J. 2012. The R Book. ISBN: 978-0-470-97392-9.

Videos

10
 STATISTICS TEXTBOOKS AND OTHER RESOURCES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Joseph, M.B. The IQUIT R video series. www.r-bloggers.com/the-iquit-r-video-series/ or

www.youtube.com/channel/UCXSzyk3li4bh9ODis_B8daQ.

Other useful resources for understanding statistics

Videos

Statistics Learning Centre (Videos with Dr. Nic). YouTube channel.

www.youtube.com/channel/UCG32MfGLit1pcqCRXyy9cAg.

Blogs

Learn and Teach Statistics and Operations Research (Articles by Dr. Nic).
learnandteachstatistics.wordpress.com/summary/.

Online Learning Modules and Massive Open Online Courses (MOOC’s)

Online learning modules and online courses offer another alternative to learning basic statistical
concepts, programming in R, or advanced statistical techniques.

Swirl
swirlstats.com/
Swirl has free, brief lessons for beginners to R programming and analysis. Topics include R
programming, data analysis, data visualization, data manipulation, statistical inference.

DataCamp
www.datacamp.com/
DataCamp has short online modules covering topics such as an introduction to R, data manipulation,
data visualization, and statistics with R. Limited parts of several courses are free. For the complete
courses, there is a subscription charge. There is a discount for students.

Coursera
www.coursera.org/
Coursera hosts MOOC’s from various universities on a range of academic topics, including introductory
statistics using R. Courses are free, though there is an option to pay for a certificate in a data science
specialization.

edX
www.edx.org/
edX also hosts free MOOC’s from universities on a variety of topics, including a course on introduction
to R programming.

Udemy
www.udemy.com/
Udemy hosts relatively inexpensive online courses on a variety of subjects, including programming
and statistics with R. Some are free.

11
STATISTICS TEXTBOOKS AND OTHER RESOURCES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

12
 WHY STATISTICS? SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Statistics for Educational Program Evaluation

Why Statistics?
Statistics is a general term. Every time you report a mean, median, or standard deviation of your data,
you are reporting a statistic. Specifically, this type of statistics are called descriptive statistics. We can
also use statistical tests and report the statistics from these tests, such as p-values or r-squared values.
In order to convey the implications of these statistics and the underlying data, we might present them
in a plot or table.

The importance of statistics in program evaluation

There are several reasons why we want to conduct statistical analyses when looking at our program
evaluation data.

Communication
Descriptive statistics and statistical tests can be used to communicate with our colleagues. Once we
understand statistical tests—as well common statistics such as p-values, confidence intervals, and r-
squared values—we can communicate our results in a shared language. A single well designed plot
with appropriate statistics is an effective tool to convey our findings.

Communication is a two-way street. We also need to be able to understand others when they share
their results with us, whether it’s in a departmental presentation, a presentation at a professional
meeting, in extension literature, or in a peer-reviewed journal article.

Adding rigor
Sometimes educators will simply report the change in median scores before and after a course, or
report a best fit line for bivariate data. However, without applying a statistical tests and reporting p-
values or other appropriate statistics, it is not clear to the reader (or probably the author!) if the
reported effect is real.

Getting things right

In order for your statistical analyses to provide correct results, you need to understand when they are
appropriate and what the results mean. Also you will want to understand what tests and techniques
are available in order to best be able to understand and explain your results.

Presenting results well

Knowledge of statistical tests and plots can inform the best ways to summarize and present results
from your data.

Academic recognition
Correct and accepted statistical methods are usually required to get results published in academic
journals or proceedings from professional meetings.

13
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Program evaluation
Appropriate statistics help to evaluate programs, for example determining if there was an increase in
student knowledge scores was statistically significant or if one teaching technique is better than
another. Such information can guide future programming efforts.

Applicability of statistical tests to other fields and situations

The statistical techniques presented in this book are applicable to a wide variety of disciplines, and are
some of the most common used across fields. While the analyses presented in this book are common
and relatively simple, understanding of these techniques will serve as a basis for more advanced
analyses.

Evaluation Tools and Surveys

A complete discussion of the proper design and administration of program evaluation tools and
surveys is far beyond the scope of this book. This chapter is intended only to give the reader some
context for the remainder of the book, which discusses the analysis of data from these survey tools and
other sources.

Extension programs

An extension program is an organized educational endeavor delivered to some segment of the public,
such as famers, students in 4-H, or Master Gardeners. It is based on an assessments of needs for that
group for the specific topic of the program, and is based on objectives for what knowledge students
should improve, skills they should learn, or behaviors they should adopt.

An extension program will have multiple activities associated with these goals. The extension
educator may be organizing educational events, delivering lectures, writing factsheets, or reporting on
research.

Each of these components is evaluated in light of the objectives of the program and the objectives of
the individual activities. In each case objectives should be measureable.

In some cases, the evaluator can collect “hard data” about the results of some program: number of
attendees, number of downloads, measured differences in water quality.

In other cases evaluation tools such as surveys will be developed to evaluate how well program
objectives will be met. These might ask participants about an increase in their knowledge, a change in
their behavior, or adoption of specific practices, along with other relevant information.

Designing a questionnaire

14
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

There is much to be learned about good questionnaire design, and good practices for conducting a
survey. This chapter will only scratch the surface of this topic.

Some principles of questionnaire design for program evaluation

Length and audience-appropriateness

Your questionnaire should be short enough and simple enough so that your respondents will be likely
to complete it, and complete it thoughtfully.

On the other hand, it should be long enough to gain valuable information. For example, it might not
want to ask simply, “Did you learn something?”, but ask how much was learned in each topic area.
Identifying areas where teaching was successful or not successful helps to improve the program in the
future.

It is also important to take into account your audience. Young children may be able to answer only a
few questions, simple in scope, and without too many options. On the other end of the spectrum,
expert audiences can much more critically assess programs and program component. Non-expert
adult audiences fall in the middle of this spectrum: They are capable of assessing their own changes in
knowledge, skills, attitudes, behaviors, and implemented practices. Still, questionnaires for them
should still be short and clear, with the possibility of some open-ended questions.

Types of impacts
A questionnaire may assess a single type of impact or several types depending on what is appropriate.
It may seek to assess knowledge gain, changes in attitude, gained skills, changes in behavior.

These can be assessed for an extension activity as short as single lecture, a day-long workshop, or after
a program lasting several years.

Assessment techniques of a questionnaire

Knowledge gain: before-and-after tests

For knowledge gain questions, respondents could be tested in a before-and-after manner, where some
specific question is asked on a pre-test before the course, and then on a post-test after the course.

It is helpful if the same respondent can be identified on both tests, so that the before responses and
after responses can be paired for each respondent.

Personally, I find before-and-after assessment rather patronizing and annoying for adult audiences.
Instead, I prefer a self-assessment knowledge gain format, discussed below.

As an example, Bakacs et al. (2013) on Table 1, reported an increase in knowledge by respondents for
individual questions about using rain barrels, conserving water, and reducing stormwater runoff. For
example, they asked workshop attendees the following question, both before and after the workshop.
They asked respondents for their initials and favorite number for each test so that responses could be
paired for each respondent.

15
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

10. A 55 gallon rain barrel, when full will weigh approximately

A. 200 lbs.
B. 300 lbs.
C. 400 lbs.
D. Not sure

Knowledge gain: self-assessment questions

Another approach for assessing knowledge gain after a program or event is to allow respondents to
assess their own knowledge gain. This allows self-reflection, and may therefore be more meaningful
than looking at correct and incorrect answers on before-and-after tests.

One form of self-assessment question asks the respondent to assess their knowledge on a topic both
before and after the course, but is answered only once, after the course.

As an example, Mangiafico et al. (2011) (Table1) reported self-assessment changes in knowledge gain
by respondents for individual questions about environmentally-friendly lawn care. The survey was
done following a lecture, so that respondents reflected about their knowledge before and after the
lecture from the perspective of having just listened to the lecture.
Before After

1 2 3 4 5 I understand how to determine how much 1 2 3 4 5

phosphorus should be applied to my lawn.

Behavior change and adopted practices

One way to approach assessing changes in behavior is to use follow-up surveys (discussed below) at
some time after a program or workshop to see if participants adopted practices or behaviors.

A second method is to question participants at the conclusion of an event concerning their anticipated
changes in behaviors based on what they learned at the event. The obvious drawback to this approach
is that respondents are likely to anticipate that they will improve their behaviors, and this anticipation
may never materialize in reality. However, this method is often used because it is easier and allows
collecting data from a captive audience.

As an example, Mangiafico et al. (2011) (Table2) reported self-assessment anticipated behaviors

concerning environmentally-friendly lawn care.
17. I will test my soil or have my soil tested to determine the need to adjust pH
and apply phosphorus fertilizer.

disagree neutral agree

1 2 3 4 5 6 7 8 9 10 Don’t know Not applicable

16
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Follow-up surveys
Follow-up surveys are conducted at some time after a course or event, and are useful to assess if
specific practices were implemented or certain behaviors were changed, to give some time reflection
by respondents to consider changes in attitudes or knowledge.

As an example, Bakacs et al. (2013) on Table 3, used a follow-up survey to determine if workshop
participants installed rain barrels at their homes or businesses after building one at the workshop.

The follow-up survey in New Jersey was e-mailed to participants three to six months after the
workshops, and were completed online.

One drawback to this approach is that it is necessary to collect contact information from participants
and take the time re-survey participants. Another problem is that response rates may be lower than
response rates from in-classroom assessments.

Rating of course quality and materials

It is often useful to have program participants rate the quality of instruction, presentation, visual aids,
handouts, etc.

Open-ended questions
Open-ended questions can be very valuable to solicit critical comments about a course or program, or
assess knowledge gain or other impacts that aren’t covered by other assessment questions.

Opened-ended questions can attempt to solicit critical comments—“What could be done to improve
this program?”; assess knowledge gain—“What did you learn from this program?”; or gain other
information—“Where did you learn about this course?”.

These questions can be very valuable, but you should also be cautious since responses may be sparse
and not representative of program participants. In particular, critical responses may be
unrepresentatively positive or negative. In general, participants may or may not put a lot effort into
answering open-ended questions, depending on a variety of factors not necessarily related to program
content or quality.

Demographic data
Finally, collecting demographic data of questionnaire respondents can be useful for interpreting and
understanding responses for other questions. Demographic data may include profession or
stakeholder group; current practices; or age, grade, sex, etc.

Forms of responses
Answers to questions may be in the form of yes/no, multiple choice, ordered response (Likert), or
open-ended.

The statistical analysis of these different forms of answers will vary. This will be unpacked over the
course of this book.

Examples of evaluation tools

17
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Rutgers Cooperative Extension Program Evaluation, Youth Audience, GRK-3

njaes.rutgers.edu/evaluation/youth/docs/RCEYOUTHEVALUATION_GRK-3_FORM.pdf
Notes:
• As a questionnaire written for younger children, most responses to questions are 3- to 5-
point Likert items with face emoticons for support, and yes/no questions.
• Questions 1 and 2 are self-assessment knowledge gain. 3-point Likert items.
• Question 3 is anticipated behavior change. Yes/no.
• Questions 5 and 6 are rating of the instructor and program. 5-point Likert items.
• Question 7 is a question about demographics.
• Question 8 is an open-ended question.

Rutgers Cooperative Extension Turf Management for a Healthier Lawn, Program Evaluation Form
rcompanion.org/documents/TurfProgramEvaluation.pdf
Notes:
• Questions 1–11 are self-assessment questions of knowledge gain and attitudes, using before-
and-after in a single sitting. 5-point Likert items.
• Questions 12–13 are ratings of information and educational materials. 10-point Likert items,
with two opt-out options (Don’t know, and Not applicable)
• Questions 17 – 20 are anticipated behaviors. 10-point Likert items, with two opt-out options
(Don’t know, and Not applicable)

Rutgers Cooperative Extension Program Evaluation, Older Youth Audience, GR4-13A

njaes.rutgers.edu/evaluation/youth/docs/RCEYOUTHEVALUATION_GR4-13A_FORM.pdf
Notes:
• Questions 1–2 are open-ended questions assessing knowledge gain and behavior change.
• Questions 4–5 are ratings of presenter and program. 5-point Likert items.
• Question 6 is a question about demographics.
• Question 7 is an open-ended question.

Rutgers Cooperative Extension Program Evaluation - Older Youth Audience, GR4-13B

njaes.rutgers.edu/evaluation/youth/docs/RCEYOUTHEVALUATION_GR4-13B_FORM.pdf
Notes:
• Questions 1–3 are self-assessment questions of knowledge gain, using before-and-after in a
single sitting. 4-point Likert items.
• Question 4 is an open-ended question about behaviors.
• Questions 6–7 are ratings of presenter and program. 5-point Likert items.
• Question 8 is a question about demographics.
• Question 9 is an open-ended question.

Optional resource
Poling, R. L. 1999. Example Extension Program Evaluation Tools. Agricultural Extension
Service Institute of Agriculture University of Tennessee. web.utk.edu/~aee/evaltools.pdf.

Optional Readings

[Video] Statistics Learning Center (Dr. Nic). 2011. “Designing a Questionnaire”.

18
 EVALUATION TOOLS AND SURVEYS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

www.youtube.com/watch?v=FkX-t0Pgzzs.

Optional additional resources

Barkman, S. J. 2001. A Field Guide to Designing Quantitative Instruments to Measure Program Impact.
Purdue University.
www.northskynonprofitnetwork.org/sites/default/files/documents/Field%20Guide%20to%20Devel
oping%20Quantiative%20Instruments.pdf.

Chavez, C. No date. Survey Design. Loyola Marymount University.

www.lmu.edu/Assets/Academic+Affairs+Division/Assessment+and+Data+Analysis/Christine$!27s+F
older/Surveys+Website/Survey+Design+Resource.pdf.

Rutgers Cooperative Extension. 2015. Program Evaluation. njaes.rutgers.edu/evaluation/.

Walonick, D.S. 2010. Designing and Using Questionnaires. StatPac. statpac.com/surveys/surveys.pdf.

References for this chapter

Bakacs, M., M. Haberland, S.S. Mangiafico, A. Winquist, C.C Obropta, A. Boyajian A., and S. Mellor. 2013.
Rain Barrels: A Catalyst for Change? Journal of Extension 51(3), article 3RIB6.
www.joe.org/joe/2013june/rb6.php

Mangiafico, S.S., C.C. Obropta, and E. Rossi-Griffin. 2011. A Lawn Care Education Program to Address
Water Conservation and Water Pollution Prevention in New Jersey. Journal of the National Association
of County Agricultural Agents 4(2). www.nacaa.com/journal/index.php?jid=108.

Acknowledgements

I would like to thank Dan Kluchinski of Rutgers Cooperative Extension and the Rutgers Department of
Agricultural and Resource Management Agents for supplying some resources for this chapter.

19
 TYPES OF VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Variables, Descriptive Statistics, and Plots

Types of Variables
Organizing data: observations and variables

In general, collected raw data is organized according to observations and variables. Variables
represent a single measurement or characteristic for each observation. In the Statistics Learning
Center video in the Required Readings below, Dr. Nic gives an example of a survey where each
observation is a separate person, and the variables are age, sex, and chocolate preference for each
person.

In education evaluation, observations will commonly be a student or an instructor, but could also be
any other experimental unit, such as a single farmer’s field.

In reality, a single observation may be more specific. For example, the rating for one instructor by one
student on one date. For example, here is part of the example from the Friedman Test chapter with
some additional hypothetical data.

Each row represents an observation. So each observation contains the rating by a single student on a
single date for an instructor. The variables are Instructor, Date, Student, and Rating.
Instructor Date Student Rating
Bob Belcher 2015-01-01 a 4
Bob Belcher 2015-01-01 b 5
Bob Belcher 2015-01-01 c 4
Bob Belcher 2015-01-01 d 6
Bob Belcher 2015-02-05 e 6
Bob Belcher 2015-02-05 f 6
Bob Belcher 2015-02-05 g 10
Bob Belcher 2015-02-05 h 6
Linda Belcher 2015-01-01 a 8
Linda Belcher 2015-01-01 b 6
Linda Belcher 2015-01-01 c 8
Linda Belcher 2015-01-01 d 8
Linda Belcher 2015-02-05 e 8
Linda Belcher 2015-02-05 f 7
Linda Belcher 2015-02-05 g 10
Linda Belcher 2015-02-05 h 9

Long-format and wide-format data

The example above has ratings for each instructor on each of two dates and by each of eight students.
Because each observation has just one rating value, this data is in long format.

In general, it is best to keep data in long format for summary and analyses.

20
 TYPES OF VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

However, to conduct certain analyses in R, or to produce certain plots, data will need to be in wide
format. The following is an example of the same data, translated into wide format, with a focus on the
ratings by date, and in this case ignoring student.
Instructor ——————— Rating ———————
2015-01-01 2015-02-05
Bob Belcher 4 6
Bob Belcher 5 6
Bob Belcher 4 10
Bob Belcher 6 6
Linda Belcher 8 8
Linda Belcher 6 7
Linda Belcher 8 10
Linda Belcher 8 9

Types of variables

The most common variables used in data analysis can be classified as one of three types of variables:
nominal, ordinal, and interval/ratio.

Understanding the differences in these types of variables is critical, since the variable type will
determine which statistical analysis will be valid for that data. In addition, the way we summarize
data with statistics and plots will be determined by the variable type.

Nominal data
Nominal variables are data whose levels are labels or descriptions, and which cannot be ordered.
Examples of nominal variables are sex, school, and yes/no questions. They are also called “categorical”
or “qualitative” variables, and the levels of a variable are sometimes called “classes” or “groups”.

The levels of categorical variables cannot be ordered. For the variable sex, it makes no sense to try to
put the levels “female”, “male”, and “other” in any numerical order. If levels are numbered for
convenience, the numbers are arbitrary, and the variable can’t be treated as a numeric variable.

Ordinal data
Ordinal variables can be ordered, or ranked in logical order, but the interval between levels of the
variables are not necessarily known. Subjective measurements are often ordinal variables. One
example would be having people rank four items by preference in order from one to four. A different
example would be having people assess several items based on a Likert ranking scale: “On a scale of
one to five, do you agree or disagree with this statement?” A third example is level of education for
adults, considering for example “less than high school”, “high school”, “associate’s degree”, etc.

Critically, in each case we can order the responses: My first favorite salad dressing is better than
second favorite, which is better than my third favorite. But we cannot know if the interval between
the levels is equal. For example, the distance between your favorite salad dressing and your second
favorite salad dressing may be small, where there may be a large gap between your second and third
choices.

21
 TYPES OF VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

We can logically assign numbers to levels of an ordinal variable, and can treat them in order, but
shouldn’t treat them as numeric: “strongly agree” and “neutral” may not average out to an “agree.”

For the purposes of this book, we will consider such Likert data to be ordinal data under most
circumstances.

Interval/ratio data
Interval/ratio variables are measured or counted values: age, height, weight, number of students. The
interval between numbers is known to be equal: the interval between one kilogram and two
kilograms is the same as between three kilograms and four kilograms. Interval/ratio data are also
called “quantitative” data, although ordinal data are also quantitative.

Discrete and continuous variables

A further division of interval/ratio data is between discrete variables, whose values are necessarily
whole numbers or other discrete values, such as population or counts of items. Continuous variables
can take on any value within an interval, and so can be expressed as decimals. They are often
measured quantities. For example, in theory a weight could be measured as 1 kg, 1.01 kg, or 1.009 kg,
and so on. Age could also be considered a continuous variable, though we often treat it as a discrete
variable, by rounding it to the most recent birthday.

Optional technical note

There is a technical difference between interval and ratio data. For interval data, the interval between
measurements is the same, but ratio between measurements is not known. A common case of this is
temperature measured in degrees Fahrenheit. The difference between 5 ° F and 10 ° F is the same as
that between 10 ° F and 15 ° F. But 10 ° F is not twice 5 ° F. This is because the definition of 0 ° F is
arbitrary. 0 ° F does not equal 0 ° C.

Measurements where there is a natural zero, such as length or height, or where a zero can be honestly
defined, such as time since an event, are considered ratio data.

For the most part, ratio and interval data are considered together. In general, just be careful not to
make senseless statements with interval data, such as saying, “The mean temperature in Greenhouse 1
was twice the mean temperature of Greenhouse 2.”

Levels of measurement
In general it is advantageous to treat variables as the highest level of measurement for which they
qualify. That is, we could treat education level as a categorical variable, but usually we will want to
treat it as an ordinal variable. This is because treating it as an ordinal variable retains more of the
information carried in the data. If we were to reduce it to a categorical variable, we would lose the
order of the levels of the variable. By using a higher level of measurement, we will have more options
in the way we analyze, summarize, and present data.

This being said, there may be cases when it is advantageous to treat ordinal or count data as
categorical. One case is if there are few levels of the variable, or if it makes sense to condense the
variable into a couple of broad categories. Another example of when we choose a lower level of
measurement is when we use nonparametric statistical analyses which treat interval/ratio data as
ordinal, or ranked, data.
22
 TYPES OF VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Required readings

[Video] “Types of Data: Nominal, Ordinal, Interval/Ratio” from Statistics Learning Center (Dr. Nic).
2011. www.youtube.com/watch?v=hZxnzfnt5v8.

Optional readings

“Types of biological variables” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/variabletypes.html.

“Frequency, frequency tables, and levels of measurement”, Chapter 1.3 in Openstax College. 2013.
Introductory Statistics. Rice University. openstaxcollege.org/textbooks/introductory-statistics.

“Data basics”, Chapter 1.2 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012. OpenIntro
Statistics, 2nd ed. www.openintro.org/.

Exercises B

1. For the following variables and levels identify the variable as nominal, ordinal, or ratio/interval.
• Political affiliation (very liberal, liberal, independent, conservative, very conservative)
• Political affiliation (Democrat, Republican, unaffiliated)
• Student age
• Favorite school subject (Science, History, Math, Physical Education, English Literature)
• How often do you complete your homework (never, sometimes, often, always)
• Sex (1-female, 2-male, 3-other)
• Level of education (1-elementary school, 2-junior high school, 3-high school)
• Dairy cow weight
• Number of meals and snacks eaten in a day
• (Yes, No)
• (Yes, No, Don’t know)
• Favorite salad dressing (1st, 2nd, 3rd, 4th)
• Favorite salad dressing (Italian, French, Caesar, etc.)

2. Are the following terms associated with nominal, ordinal, or ratio/interval data.
• Likert
• Categorical
• Continuous
• Qualitative
• Discrete

3. For the following table, identify:

• Number of observations
• Number of variables
• Type of each variable (nominal, ordinal, or interval/ratio)

23
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Student Grade Age Sex Height Calories Attitude Class

eaten rank
A 5 10 M 137 2000 5 4
B 5 11 M 140 1500 4 2
C 4 9 F 120 1200 3 5
D 4 10 F 140 1400 4 1
E 6 12 Other 147 1800 5 6
F 5 10 M 135 1600 4 8
G 4 9 M 130 1200 4 3
H 4 10 F 140 1800 3 7

Descriptive Statistics
Packages used in this chapter

The packages used in this chapter include:

• psych
• DescTools
• Rmisc
• FSA
• plyr
• boot

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(Rmisc)){install.packages("Rmisc")}
if(!require(FSA)){install.packages("FSA")}
if(!require(plyr)){install.packages("plyr")}
if(!require(boot)){install.packages("boot")}

Descriptive statistics

Descriptive statistics are used to summarize data in a way that provides insight into the information
contained in the data. This might include examining the mean or median of numeric data or the
frequency of observations for nominal data. Plots can be created that show the data and indicating
summary statistics.

Choosing which summary statistics are appropriate depend on the type of variable being examined.
Different statistics should be used for interval/ratio, ordinal, and nominal data.

In describing or examining data, you will typically be concerned with measures of location, variation,
and shape.

24
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Location is also called central tendency. It is a measure of the values of the data. For example, are the
values close to 10 or 100 or 1000? Measures of location include mean and median, as well as
somewhat more exotic statistics like M-estimators or Winsorized means.

Variation is also called dispersion. It is a measure of how far the data points lie from one another.
Common statistics include standard deviation and coefficient of variation. For data that aren’t
normally-distributed, percentiles or the interquartile range might be used.

Shape refers to the distribution of values. The best tools to evaluate the shape of data are histograms
and related plots. Statistics include skewness and kurtosis, though they are less useful than visual
inspection. We can describe data shape as normally-distributed, log-normal, uniform, skewed, bi-
modal, and others.

Descriptive statistics for interval/ratio data

For this example, imagine that Ren and Stimpy have each held eight workshops educating the public
about water conservation at home. They are interested in how many people showed up to the
workshops.

Because the data are housed in a data frame, we can use the convention Data$Attendees to access the
variable Attendees within the data frame Data.
Input = ("
Instructor Location Attendees
Ren North 7
Ren North 22
Ren North 6
Ren North 15
Ren South 12
Ren South 13
Ren South 14
Ren South 16
Stimpy North 18
Stimpy North 17
Stimpy North 15
Stimpy North 9
Stimpy South 15
Stimpy South 11
Stimpy South 19
Stimpy South 23
")

Data = read.table(textConnection(Input),header=TRUE)

Data ### Will output data frame called Data

str(Data) ### Will be discussed later,

### but shows the structure of the data frame

summary(Data) ### Will be discussed later,

25
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### but summarizes variables in the data frame

Functions sum and length

The sum of a variable can be found with the sum function, and the number of observations can be
found with the length function.
sum(Data$Attendees)

232

length(Data$Attendees)

16

Statistics of location for interval/ratio data

Mean
The mean is the arithmetic average, and is a common statistic used with interval/ratio data. It is
simply the sum of the values divided by the number of values. The mean function in R will return the
mean.
sum(Data$Attendees) / length(Data$Attendees)

14.5

mean(Data$Attendees)

14.5

Caution should be used when reporting mean values with skewed data, as the mean may not be
representative of the center of the data. For example, imagine a town with 10 families, nine of whom
have an income of less than $50, 000 per year, but with one family with an income of $2,000,000 per
year. The mean income for families in the town would be $233,000, but this may not be a reasonable
way to summarize the income of the town.
Income = c(49000, 44000, 25000, 18000, 32000, 47000, 37000, 45000, 36000, 2000000)

mean(Income)

233300

Median
The median is defined as the value below which are 50% of the observations. To find this value
manually, you would order the observations, and separate the lowest 50% from the highest 50%. For
data sets with an odd number of observations, the median is the middle value. For data sets with an
even number of observations, the median falls half-way between the two middle values.

26
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The median is a robust statistic in that it is not affected by adding extreme values. For example, if we
changed Stimpy’s last Attendees value from 23 to 1000, it would not affect the median.
median(Data$Attendees)

15

### Note that in this case the mean and median are close in value
### to one another. The mean and median will be more different
### the more the data are skewed.

The median is appropriate for either skewed or unskewed data. The median income for the town
discussed above is $40,500. Half the families in the town have an income above this amount, and half
have an income below this amount.

Income = c(49000, 44000, 25000, 18000, 32000, 47000, 37000, 45000, 36000, 2000000)

median(Income)

40500

Note that medians are sometimes reported as the “average person” or “typical family”. Saying, “The
average American family earned $54,000 last year” means that the median income for families was
$54,000. The “average family” is that one with the median income.

Mode
The mode is a summary statistic that is used rarely in practice, but is normally included in any
discussion of mean and medians. When there are discreet values for a variable, the mode is simply the
value which occurs most frequently. For example, in the Statistics Learning Center video in the
Required Readings below, Dr. Nic gives an example of counting the number of pairs of shoes each
student owns. The most common answer was 10, and therefore 10 is the mode for that data set.

For our Ren and Stimpy example, the value 15 occurs three times and so is the mode.

The Mode function can be found in the package DescTools.

library(DescTools)

Mode(Data$Attendees)

15

Statistics of variation for interval/ratio data

Standard deviation
The standard deviation is a measure of variation which is commonly used with interval/ratio data. It’s
a measurement of how close the observations in the data set are to the mean.

27
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

There’s a handy rule of thumb that—for normally distributed data—68% of data points fall within the
mean ± 1 standard deviation, 95% of data points fall within the mean ± 2 standard deviations, and
99.7% of data points fall within the mean ± 3 standard deviations.

Because the mean is often represented with the letter mu, and the standard deviation is represented
with the letter sigma, saying someone is “a few sigmas away from mu” indicates they are rather a rare
character. (I originally heard this joke on an episode of Car Talk, for which I cannot find a reference or
transcript.)
sd(Data$Attendees)

4.830459

Standard deviation may not be appropriate for skewed data.

Standard error of the mean

Standard error of the mean is a measure that estimates how close a calculated mean is likely to be to
the true mean of that population. It is commonly used in tables or plots where multiple means are
presented together. For example, we might want to present the mean attendees for Ren with the
standard error for that mean and the mean attendees for Stimpy with the standard error that mean.

The standard error is the standard deviation of a data set divided by the square root of the number of
observations. It can also be found in the output for the describe function in the psych package, labelled
se.

sd(Data$Attendees) /
sqrt(length(Data$Attendees))

1.207615

library(psych)

describe(Data$Attendees)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 16 14.5 4.83 15 14.5 4.45 6 23 17 -0.04 -0.88 1.21

### se indicates the standard error of the mean

Standard error of the mean may not be appropriate for skewed data.

Five-number summary, quartiles, percentiles

The median is the same as the 50th percentile, because 50% of values fall below this value. Other
percentiles for a data set can be identified to provide more information. Typically, the 0th, 25th, 50th,
75th, and 100th percentiles are reported. This is sometimes called the five-number summary.

28
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

These values can also be called the minimum, 1st quartile, 2nd quartile, 3rd quartile, and maximum.

The five-number summary is a useful measure of variation for skewed interval/ratio data or for
ordinal data. 25% of values fall below the 1st quartile and 25% of values fall above the 3rd quartile.
This leaves the middle 50% of values between the 1st and 3rd quartiles, giving a sense of the range of
the middle half of the data. This range is called the interquartile range (IQR).

Percentiles and quartiles are relatively robust, as they aren’t affected much by a few extreme values.
They are appropriate for both skewed and unskewed data.
summary(Data$Attendees)

Min. 1st Qu. Median Mean 3rd Qu. Max.

6.00 11.75 15.00 14.50 17.25 23.00

### The five-number summary and the mean

Optional technical note on calculating percentiles

It may have struck you as odd that the 3rd quartile for Attendees was reported as 17.25. After all, if you
were to order the values of Attendees, the 75th percentile would fall between 17 and 18. But why does
R go with 17.25 and not 17.5?
sort(Data$Attendees)

6 7 9 11 12 13 14 15 15 15 16 17 18 19 22 23

The answer is that there are several different methods to calculate percentiles, and they may give
slightly different answers. For details on the calculations, see ?quantiles.

For Attendees, the default type 7 calculation yields a 75th percentile value of 17.25, whereas the type 2
calculation simply splits the difference between 17 and 18 and yields 17.5. The type 1 calculation
doesn’t average the two values, and so just returns 17.

quantile(Data$Attendees, 0.75, type=7)

75%
17.25

quantile(Data$Attendees, 0.75, type=2)

75%
17.5

quantile(Data$Attendees, 0.75, type=1)

75%
17

29
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Percentiles other than the 25th, 50th, and 75th can be calculated with the quantiles function. For
example, to calculate the 95th percentile:
quantile(Data$Attendees, .95)

95%
22.25

Confidence intervals
Confidence intervals are covered in the next chapter.

Statistics for grouped interval/ratio data

In many cases, we will want to examine summary statistics for a variable within groups. For example,
we may want to examine statistics for the workshops lead by Ren and those lead by Stimpy.

Summarize in FSA
The Summarize function in the FSA package returns the number of observations, mean, standard
deviation, minimum, 1st quartile, median, 3rd quartile, and maximum for grouped data.

Note the use of formula notation: Attendees is the dependent variable (the variable you want to get the
statistics for); and Instructor is the independent variable (the grouping variable). Summarize allows
you to summarize over the combination of multiple independent variables by listing them to the right
of the ~ separated by a plus sign (+).

library(FSA)

Summarize(Attendees ~ Instructor,
data=Data)

Instructor n nvalid mean sd min Q1 median Q3 max percZero

1 Ren 8 8 13.125 5.083236 6 10.75 13.5 15.25 22 0
2 Stimpy 8 8 15.875 4.454131 9 14.00 16.0 18.25 23 0

Summarize(Attendees ~ Instructor + Location,

data=Data)

Instructor Location n nvalid mean sd min Q1 median Q3 max percZero

1 Ren North 4 4 12.50 7.505554 6 6.75 11.0 16.75 22 0
2 Stimpy North 4 4 14.75 4.031129 9 13.50 16.0 17.25 18 0
3 Ren South 4 4 13.75 1.707825 12 12.75 13.5 14.50 16 0
4 Stimpy South 4 4 17.00 5.163978 11 14.00 17.0 20.00 23 0

summarySE in Rmisc
The summarySE function in the Rmisc package outputs the number of observations, mean, standard
deviation, standard error of the mean, and confidence interval for grouped data. The summarySE

30
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

function allows you to summarize over the combination of multiple independent variables by listing
them as a vector, e.g. c("Instructor", "Student").
library(Rmisc)

summarySE(data=Data,
"Attendees",
groupvars="Instructor",
conf.interval = 0.95)

Instructor N Attendees sd se ci
1 Ren 8 13.125 5.083236 1.797195 4.249691
2 Stimpy 8 15.875 4.454131 1.574773 3.723747

summarySE(data=Data,
"Attendees",
groupvars = c("Instructor", "Location"),
conf.interval = 0.95)

Instructor Location N Attendees sd se ci

1 Ren North 4 12.50 7.505553 3.7527767 11.943011
2 Ren South 4 13.75 1.707825 0.8539126 2.717531
3 Stimpy North 4 14.75 4.031129 2.0155644 6.414426
4 Stimpy South 4 17.00 5.163978 2.5819889 8.217041

describeBy in psych
The describeBy function in the psych package returns the number of observations, mean, median,
trimmed means, minimum, maximum, range, skew, kurtosis, and standard error of the mean for
grouped data. describeBy allows you to summarize over the combination of multiple independent
variables by combining terms with a colon (:).

library(psych)

describeBy(Data$Attendees,
group = Data$Instructor,
digits= 4)

group: Ren
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 8 13.12 5.08 13.5 13.12 2.97 6 22 16 0.13 -1.08 1.8
-------------------------------------------------------------------------
group: Stimpy
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 8 15.88 4.45 16 15.88 3.71 9 23 14 -0.06 -1.26 1.57

describeBy(Data$Attendees,
group = Data$Instructor : Data$Location,
digits= 4)

31
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

group: Ren:North
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 4 12.5 7.51 11 12.5 6.67 6 22 16 0.26 -2.14 3.75
-------------------------------------------------------------------------
group: Ren:South
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 4 13.75 1.71 13.5 13.75 1.48 12 16 4 0.28 -1.96 0.85
-------------------------------------------------------------------------
group: Stimpy:North
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 4 14.75 4.03 16 14.75 2.22 9 18 9 -0.55 -1.84 2.02
-------------------------------------------------------------------------
group: Stimpy:South
vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 4 17 5.16 17 17 5.93 11 23 12 0 -2.08 2.58

Summaries for data frames

Often we will want to summarize variables in a whole data frame, either to get some summary
statistics for individual variables, or to check that variables have the values we expected in inputting
the data, to be sure there wasn’t some error.

The str function

The str function in the native utils package will list variables for a data frame, with their types and
levels. Data is the name of the data frame, created above.

str(Data)

'data.frame': 16 obs. of 3 variables:

$ Instructor: Factor w/ 2 levels "Ren","Stimpy": 1 1 1 1 1 1 1 1 2 2 ...
$ Location : Factor w/ 2 levels "North","South": 1 1 1 1 2 2 2 2 1 1 ...
$ Attendees : int 7 22 6 15 12 13 14 16 18 17 ...

### Instructor is a factor (nominal) variable with two levels.

### Location is a factor (nominal) variable with two levels.
### Attendees is an integer variable.

The summary function

The summary function in the native base package will summarize all variables in a data frame, listing
the frequencies for levels of nominal variables; and for ratio/interval data, the minimum, 1st quartile,
median, mean, 3rd quartile, and maximum.

summary(Data)

Instructor Location Attendees

Ren :8 North:8 Min. : 6.00
Stimpy:8 South:8 1st Qu.:11.75
Median :15.00
Mean :14.50
3rd Qu.:17.25

32
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Max. :23.00

The headTail function in psych

The headTail function in the psych package reports the first and last observations for a data frame.
library(psych)

headTail(Data)

Instructor Location Attendees

1 Ren North 7
2 Ren North 22
3 Ren North 6
4 Ren North 15
... <NA> <NA> ...
13 Stimpy South 15
14 Stimpy South 11
15 Stimpy South 19
16 Stimpy South 23

The describe function in psych

The describe function in the psych package reports number of observations, mean, standard deviation,
trimmed means, minimum, maximum, range, skew, kurtosis, and standard error for variables in a data
frame.

Note that factor variables are labeled with an asterisk (*), and the levels of the factors are coded as 1,
2, 3, etc.

library(psych)

describe(Data)

vars n mean sd median trimmed mad min max range skew kurtosis se
Instructor* 1 16 1.5 0.52 1.5 1.5 0.74 1 2 1 0.00 -2.12 0.13
Location* 2 16 1.5 0.52 1.5 1.5 0.74 1 2 1 0.00 -2.12 0.13
Attendees 3 16 14.5 4.83 15.0 14.5 4.45 6 23 17 -0.04 -0.88 1.21

Dealing with missing values

Sometimes a data set will have missing values. This can occur for a variety of reasons, such as a
respondent not answering a specific question, or a researcher being unable to make a measurement
due to temporarily malfunctioning equipment.

In R, a missing value is indicated with NA.

By default, different functions in R will handle missing values in different ways. But most have options
to change how they treat missing data.

33
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In general, you should scan your data for missing data, and think carefully about the best way to
handle observations with missing values.
Input = ("
Instructor Location Attendees
Ren North 7
Ren North 22
Ren North 6
Ren North 15
Ren South 12
Ren South 13
Ren South NA
Ren South 16
Stimpy North 18
Stimpy North 17
Stimpy North NA
Stimpy North 9
Stimpy South 15
Stimpy South 11
Stimpy South 19
Stimpy South 23
")

Data2 = read.table(textConnection(Input),header=TRUE)

### Note: This data frame will be called Data2 to distinguish it

### from Data above.

Data2

The na.rm option for missing values with a simple function

Many common functions in R have a na.rm option. If this option is set to FALSE, the function will
return an NA result if there are any NA’s in the data values passed to the function. If set to TRUE,
observations with NA values will be discarded, and the function will continue to calculate its output.

Note that the na.rm option operates only on the data values actually passed to the function. In the
following example with median, only Attendees is passed to the function; if there were NA’s in other
variables, this would not affect the function.

Not all functions have the same default for the na.rm option. To determine the default, use e.g.
?median, ?mean, ?sd.
median(Data2$Attendees,
na.rm = FALSE)

NA

### na.rm=FALSE. Since there is an NA in the data, report NA.

median(Data2$Attendees,
na.rm = TRUE)

34
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

15

### na.rm=TRUE. Drop observations with NA and then calculate the median.

Missing values with Summarize in FSA

Summarize in FSA will indicate invalid values, including NA’s, with the count of valid observations
outputted as the variable nvalid.
library(FSA)

Summarize(Attendees ~ Instructor,
data=Data2)

Instructor n nvalid mean sd min Q1 median Q3 max percZero

1 Ren 8 7 13 5.477226 6 9.5 13 15.5 22 0
2 Stimpy 8 7 16 4.795832 9 13.0 17 18.5 23 0

### This function removes missing values, but indicates the number of
### missing values by not including them in the count for nvalid.

Missing values indicated with summary function

The summary function counts NA’s for numeric variables.
summary(Data2)

Instructor Location Attendees

Ren :8 North:8 Min. : 6.00
Stimpy:8 South:8 1st Qu.:11.25
Median :15.00
Mean :14.50
3rd Qu.:17.75
Max. :23.00
NA's :2

### Indicates two NA’s in Attendees.

Missing values in the describe function in psych

The describe function is the psych package removes NA’s by default.
library(psych)

describe(Data2$Attendees)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 14 14.5 5.19 15 14.5 5.19 6 23 17 -0.04 -1.17 1.39

### Note that two NA’s were removed by default, reporting an n of 14.

35
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Missing values in the summarySE function in Rmisc

By default, the summarySE function does not remove NA’s, but can be made to do so with the
na.rm=TRUE option.
library(Rmisc)

summarySE(data=Data2,
"Attendees")

.id N Attendees sd se ci
1 <NA> 16 NA NA NA NA

### Note an N of 16 is reported, and statistics are reported as NA.

library(Rmisc)

summarySE(data=Data2,
"Attendees",
na.rm=TRUE)

.id N Attendees sd se ci
1 <NA> 14 14.5 5.185038 1.38576 2.993752

### Note an N of 14 is reported, and statistics are calculated with

### NA’s removed.

Advanced techniques

Discarding subjects
Certain psychology studies use scales that are calculated from responses of several questions. Because
answers to all questions are needed to reliably calculate the scale, any observation (subject, person)
with missing answers will simply be discarded. Some types of analyses in other fields also follow this
approach.

Subjects can be deleted with the subset function. The following code creates a new data frame, Data3,
with all observations with NA in the variable Attendees removed from Data2.

Data3 = subset(Data2,
!is.na(Attendees))

Data3

Imputation of values
Missing values can be assigned a likely value through the process of imputation. An algorithm is used
that determines a value based on the values of other variables for that observation relative to values
for other variables. The mice package in R can perform imputation.

36
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional code: removing missing values in vectors

If functions don’t have a na.rm option, it is possible to remove NA observations manually. Here we’ll
create a vector called valid that just contains those values of Attendees that are not NA’s.

The ! operator is a logical not. The brackets serve as a list of observations to include for the preceding
variable. So, the code essentially says, “Define a vector valid as the values of Attendees where the
values of Attendees are not NA.”

valid = Data2$Attendees[!is.na(Data2$Attendees)]

summary(valid)

Min. 1st Qu. Median Mean 3rd Qu. Max.

6.00 11.25 15.00 14.50 17.75 23.00

Statistics of shape for interval/ratio data

The most common statistics for shape are skewness and kurtosis.

Understanding skewness and kurtosis are important as they are ways in which a distribution of data
varies from a normal distribution. This will be important in assessing the assumptions of certain
statistical tests.

However, I rarely see skewness and kurtosis values reported. Instead, normality is usually assessed
visually with plot, or using certain statistical tests. One problem with using skewness and kurtosis
values is that there is not agreement in what values constitute meaningful deviations from the normal
curve.

Skewness
Skewness indicates the degree of asymmetry in a data set. If there are relatively more values that are
far greater than the mean, the distribution is positively skewed or right skewed, with a tail stretching
to the right. Negative or left skew is the opposite.

A symmetric distribution has a skewness of 0. The skewness value for a positively skewed distribution
is positive, and a negative value for a negatively skewed distribution. Sometimes a skew with an
absolute value greater than 1 or 1.5 or 2 is considered highly skewed. There is not agreement on this
interpretation.

There are different methods to calculate skew and kurtosis. The describe function in the psych package
has three options for how to calculate them.

library(psych)

describe(Data$Attendees,
type=3) ### Type of calculation for skewness and kurtosis

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 16 14.5 4.83 15 14.5 4.45 6 23 17 -0.04 -0.88 1.21

37
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Skewness and kurtosis among other statistics

The normal curve is symmetrical around its center. The positively-skewed

distribution has a longer, thicker tail to the right. The negatively-skewed
distribution has a longer, thicker tail to the left.

For more information on the normal curve, see the video on “Normal Distribution” from Statistics
Learning Center in the “Optional Readings” section. Additional thoughts on the normal distribution
and real-world data distributions are in the article by Dr. Nic in the “Optional Readings” section.

Kurtosis

Kurtosis measures the degree to which the distribution is either more sharply peaked or more squat
relative to a normal distribution. In general, the higher the kurtosis, the sharper the peak and the
longer the tails. This is called leptokurtic, and is indicated by positive kurtosis values. The opposite—
platykurtosis—has negative kurtosis values. Sometimes a kurtosis with an absolute value greater than
2 or 3 is considered a high deviation from being mesokurtic. There is not agreement on this
interpretation.

Descriptive statistics for ordinal data

Descriptive statistics for ordinal data are more limited than those for interval/ratio data. You’ll
remember that for ordinal data, the levels can be ordered, but we can’t say that the intervals between
the levels are equal. For example, we can’t say that an Associate’s degree and Master’s degree

38
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

somehow average out to a Bachelor’s degree. This concept is discussed in more detail in the chapters
on Likert data.

Because of this fact, several common descriptive statistics are usually inappropriate for use with
ordinal data. These include mean, standard deviation, and standard error of the mean.

Ordinal data can be described by either: 1) treating the data as numeric and using appropriate
statistics such as median and quartiles; or, 2) treating the data as nominal, and looking at counts of the
data for each level.

A more-complete discussion of descriptive statistics for ordinal data can be found in the Descriptive
Statistics for Likert Data chapter.

Example of descriptive statistics for ordinal data

For this example, imagine that Arthur and Baxter have each held a workshop educating the public
about preventing water pollution at home. They are interested in the education level of people who
attended the workshops. We can consider education level an ordinal variable. They also collected
information on the sex and county of participants.

For this data, we have manually coded Education with numbers in a separate variable Ed.code. These
numbers list the education in order: High school < Associate’s < Bachelor’s < Master’s < Ph.D.

Of course we could have had R do this coding for us, but the code is slightly messy, so I did the coding
manually, and listed the optional R code below.

Ideally we would want to treat Education as an ordered factor variable in R. But unfortunately most
common functions in R won’t handle ordered factors well. Later in this book, ordered factor data will
be handled directly with cumulative link models (CLM), permutation tests, and tests for ordered
tables.

Optional R code is shown below for converting a factor to an ordered factor.

Input = ("
Date Instructor Student Sex County Education Ed.code
'2015-11-01' 'Arthur Read' a female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' b female 'Bear Lake' PHD 5
'2015-11-01' 'Arthur Read' c male 'Elwood' BA 3
'2015-11-01' 'Arthur Read' d female 'Elwood' MA 4
'2015-11-01' 'Arthur Read' e male 'Elwood' HS 1
'2015-11-01' 'Arthur Read' f female 'Bear Lake' MA 4
'2015-11-01' 'Arthur Read' g male 'Elwood' HS 1
'2015-11-01' 'Arthur Read' h female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' i female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' j female 'Elwood' BA 3
'2015-12-01' 'Buster Baxter' k male 'Elwood' MA 4
'2015-12-01' 'Buster Baxter' l male 'Bear Lake' MA 4
'2015-12-01' 'Buster Baxter' m female 'Elwood' AA 2
'2015-12-01' 'Buster Baxter' n male 'Elwood' AA 2
'2015-12-01' 'Buster Baxter' o other 'Elwood' BA 3

39
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'2015-12-01' 'Buster Baxter' p female 'Elwood' BA 3

'2015-12-01' 'Buster Baxter' q female 'Bear Lake' PHD 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

Data

str(Data)

'data.frame': 17 obs. of 7 variables:

$ Date : Factor w/ 2 levels "2015-11-01","2015-12-01": 1 1 1 1 1 1 1 1 1 1
$ Instructor: Factor w/ 2 levels "Arthur Read",..: 1 1 1 1 1 1 1 1 1 1 ...
$ Student : Factor w/ 17 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
$ Sex : Factor w/ 3 levels "female","male",..: 1 1 2 1 2 1 2 1 1 1 ...
$ County : Factor w/ 2 levels "Bear Lake","Elwood": 2 1 2 2 2 1 2 2 2 2 ...
$ Education : Factor w/ 5 levels "AA","BA","HS",..: 2 5 2 4 3 4 3 2 2 2 ...
$ Ed.code : int 3 5 3 4 1 4 1 3 3 3 ...

summary(Data)

Date Instructor Student Sex County

2015-11-01:10 Arthur Read :10 a : 1 female:10 Bear Lake: 4
2015-12-01: 7 Buster Baxter: 7 b : 1 male : 6 Elwood :13
c : 1 other : 1
d : 1
e : 1
f : 1
(Other):11
Education Ed.code
AA :2 Min. :1.000
BA :7 1st Qu.:3.000
HS :2 Median :3.000
MA :4 Mean :3.118
PHD:2 3rd Qu.:4.000
Max. :5.00

### Remove unnecessary objects

rm(Input)

Optional code to assign values to a variable based on another variable

Data$Ed.code[Data$Education=="HS"] = 1
Data$Ed.code[Data$Education=="AA"] = 2
Data$Ed.code[Data$Education=="BA"] = 3
Data$Ed.code[Data$Education=="MA"] = 4
Data$Ed.code[Data$Education=="PHD"] = 5

40
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional code to change a factor variable to an ordered factor variable

Data$Education.ordered = factor(Data$Education,
ordered = TRUE,
levels = c("HS", "AA", "BA", "MA", "PHD")
)
str(Data$Education.ordered)

summary(Data$Education.ordered)

HS AA BA MA PHD
2 2 7 4 2

median(Data$Education.ordered)

[1] "BA"

Statistics of location for ordinal data

Using the mean is usually not appropriate for ordinal data. Instead the median should be used as a
measure of location. We will use our numerically-coded variable Ed.code for the analysis.

median(Data$Ed.code)

### Remember that 3 meant Bachelor’s degree in our coding

Statistics of variation for ordinal data

Statistics like standard deviation and standard error of the mean are usually inappropriate for ordinal
data. Instead the five-number summary can be used.
summary(Data$Ed.code)

Min. 1st Qu. Median Mean 3rd Qu. Max.

1.000 3.000 3.000 3.118 4.000 5.000

### Remember to ignore the mean value for ordinal data.

### Remember that 1 meant High school, 3 meant Bachelor’s,

### 4 meant Master’s, and 5 meant Ph.D.

Statistics for grouped ordinal data

Because the Summarize function in the FSA package reports the five-number summary, it is a useful
function for descriptive statistics for grouped ordinal data.
library(FSA)

41
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize(Ed.code ~ Sex,
data=Data)

Sex n nvalid mean sd min Q1 median Q3 max percZero

1 female 10 10 3.5 0.9718253 2 3.00 3.0 4.00 5 0
2 male 6 6 2.5 1.3784049 1 1.25 2.5 3.75 4 0
3 other 1 1 3.0 NA 3 3.00 3.0 3.00 3 0

### Remember to ignore the mean and sd values for ordinal data.

library(FSA)

Summarize(Ed.code ~ Sex + County,

data=Data)

Sex County n nvalid mean sd min Q1 median Q3 max percZero

1 female Bear Lake 3 3 4.666667 0.5773503 4 4.5 5 5 5 0
2 male Bear Lake 1 1 4.000000 NA 4 4.0 4 4 4 0
3 female Elwood 7 7 3.000000 0.5773503 2 3.0 3 3 4 0
4 male Elwood 5 5 2.200000 1.3038405 1 1.0 2 3 4 0
5 other Elwood 1 1 3.000000 NA 3 3.0 3 3 3 0

### Remember to ignore the mean and sd values for ordinal data.

Statistics for ordinal data treated as nominal data

The summary function in the native base package and the xtabs function in the native stats package
provide counts for levels of a nominal variable.

The prop.table function translates a table into proportions. The margin=1 option indicates that the
proportions are calculated for each row.

First we will order the levels of Education, otherwise R with report results in alphabetical order.
Data$Education = factor(Data$Education,
levels = c("HS", "AA", "BA", "MA", "PHD"))

### Order factors, otherwise R will alphabetize them

summary(Data$Education)

HS AA BA MA PHD
2 2 7 4 2

### Counts of each level of Education

XT = xtabs( ~ Education,
data=Data)

XT

42
DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Education
HS AA BA MA PHD
2 2 7 4 2

prop.table(XT)

Education
HS AA BA MA PHD
0.1176471 0.1176471 0.4117647 0.2352941 0.1176471

Grouped data
XT = xtabs( ~ Sex + Education,
data = Data)

XT

Education
Sex HS AA BA MA PHD
female 0 1 5 2 2
male 2 1 1 2 0
other 0 0 1 0 0

prop.table(XT,
margin = 1)

Education
Sex HS AA BA MA PHD
female 0.0000000 0.1000000 0.5000000 0.2000000 0.2000000
male 0.3333333 0.1666667 0.1666667 0.3333333 0.0000000
other 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000

### Proportion of responses for each row

Two-way grouped data

XT = xtabs(~ Sex + Education + County,
data=Data)

XT

### Note that the dependent variable, Education, is the middle

### of the variable list.

, , County = Bear Lake

Education
Sex HS AA BA MA PHD
female 0 0 0 1 2

43
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

male 0 0 0 1 0
other 0 0 0 0 0

, , County = Elwood

Education
Sex HS AA BA MA PHD
female 0 1 5 1 0
male 2 1 1 1 0
other 0 0 1 0 0

Descriptive statistics for nominal data

Descriptive statistics for nominal data consist of listing or plotting counts for levels of the nominal
data, often by levels of a grouping variable. Tables of this information are often called contingency
tables.

For this example, we will look again at Arthur and Buster’s data, but this time considering Sex to be the
dependent variable of interest.

If levels of a nominal variable are coded with numbers, remember that the numbers will be arbitrary.
For example, if you assign female = 1, and male = 2, and other = 3, it makes no sense to say that the
average sex in Arthur’s class was 1.3. Or that the average sex in Buster’s class was greater than that in
Arthur’s. It also makes no sense to say that the median sex was female.
Input = ("
Date Instructor Student Sex County Education Ed.code
'2015-11-01' 'Arthur Read' a female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' b female 'Bear Lake' PHD 5
'2015-11-01' 'Arthur Read' c male 'Elwood' BA 3
'2015-11-01' 'Arthur Read' d female 'Elwood' MA 4
'2015-11-01' 'Arthur Read' e male 'Elwood' HS 1
'2015-11-01' 'Arthur Read' f female 'Bear Lake' MA 4
'2015-11-01' 'Arthur Read' g male 'Elwood' HS 1
'2015-11-01' 'Arthur Read' h female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' i female 'Elwood' BA 3
'2015-11-01' 'Arthur Read' j female 'Elwood' BA 3
'2015-12-01' 'Buster Baxter' k male 'Elwood' MA 4
'2015-12-01' 'Buster Baxter' l male 'Bear Lake' MA 4
'2015-12-01' 'Buster Baxter' m female 'Elwood' AA 2
'2015-12-01' 'Buster Baxter' n male 'Elwood' AA 2
'2015-12-01' 'Buster Baxter' o other 'Elwood' BA 3
'2015-12-01' 'Buster Baxter' p female 'Elwood' BA 3
'2015-12-01' 'Buster Baxter' q female 'Bear Lake' PHD 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

44
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Example of descriptive statistics for nominal data

The summary function in the native base package and the xtabs function in the native stats package
provide counts for levels of a nominal variable.

The prop.table function translates a table into proportions. The margin=1 option indicates that the
proportions are calculated for each row.

One-sample data
summary(Data$Sex)

female male other

10 6 1

### Counts of each level of Sex

One-way data
xtabs(~ Date + Sex,
data=Data)

Sex
Date female male other
2015-11-01 7 3 0
2015-12-01 3 3 1

XT = xtabs(~ Date + Sex,

data=Data)

prop.table(XT,
margin = 1)

Sex
Date female male other
2015-11-01 0.7000000 0.3000000 0.0000000
2015-12-01 0.4285714 0.4285714 0.1428571

### Proportion of each level of Sex for each row

45
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

sum(XT)

[1] 17

### Sum of observation in the table

rowSums(XT)

2015-11-01 2015-12-01
10 7

### Sum of observation in each row of the table

colSums(XT)

female male other

10 6 1

### Sum of observation in each column of the table

Two-way data
xtabs(~ County + Sex + Date,
data=Data)

### Note that the dependent variable, Sex, is the middle

### of the variable list.

, , Date = 2015-11-01

Sex
County female male other
Bear Lake 2 0 0
Elwood 5 3 0

, , Date = 2015-12-01

Sex
County female male other
Bear Lake 1 1 0
Elwood 2 2 1

Levels for factor variables

Most of the time in R, nominal variables will be handled by the software as factor variables.

The order of levels of factor variables are important because most functions, including plotting
functions, will handle levels of the factor in order. The order of the levels can be changed, for example,
to change the order that groups are plotted in a plot, or which groups are at the top of the table.

46
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

By default, the read.table function in R interprets character data as factor variables. And by default R
alphabetizes the levels of the factors.

Looking at the Arthur and Buster data, note that Instructor, Student, Sex, among other variables, are
treated as factor variables.
str(Data)

'data.frame': 17 obs. of 7 variables:

$ Date : Factor w/ 2 levels "2015-11-01","2015-12-01": 1 1 1 1 1 1 1 1
$ Instructor: Factor w/ 2 levels "Arthur Read",..: 1 1 1 1 1 1 1 1 1 1 ...
$ Student : Factor w/ 17 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 .
$ Sex : Factor w/ 3 levels "female","male",..: 1 1 2 1 2 1 2 1 1 1 ...
$ County : Factor w/ 2 levels "Bear Lake","Elwood": 2 1 2 2 2 1 2 2 2 2 .
$ Education : Factor w/ 5 levels "AA","BA","HS",..: 2 5 2 4 3 4 3 2 2 2 ...
$ Ed.code : int 3 5 3 4 1 4 1 3 3 3 ...

Note also that the levels of the factor variables were alphabetized by default. That is, even though
Elwood was found in the data before Bear Lake, R treats Bear Lake as the first level in the variable
County.
summary(Data)

Date Instructor Sex County

2015-11-01:10 Arthur Read :10 female:10 Bear Lake: 4
2015-12-01: 7 Buster Baxter: 7 male : 6 Elwood :13
other : 1

We can order factor levels by the order in which they were read in the data frame.
Data$County = factor(Data$County,
levels=unique(Data$County))

levels(Data$County)

[1] "Elwood" "Bear Lake"

We can also order factor levels in an order of our choosing.

Data$Sex = factor(Data$Sex,
levels=c("male" , "female"))

levels(Data$Sex)

[1] "male" "female"

47
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that in the actions above, we are not changing the order in the data frame, simply which level is
treated internally by the software as "1" or "2", and so on.

Required readings

[Video] “Understanding Summary statistics: Mean, Median, Mode” from Statistics Learning Center
(Dr. Nic). 2015. www.youtube.com/watch?v=rAN6DBctgJ0.

Optional readings

Statistics of location: mean and median

“Statistics of central tendency” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/central.html.

"The median outclasses the mean" from Dr. Nic. 2013. Learn and Teach Statistics &
Operations Research. learnandteachstatistics.wordpress.com/2013/04/29/median/.

“Measures of the Location of the Data”, Section 2.3 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

“Measures of the Center of the Data”, Section 2.5 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

“Skewness and the Mean, Median, and Mode”, Section 2.6 in Openstax. 2013. Introductory
Statistics. openstaxcollege.org/textbooks/introductory-statistics.

Statistics of variation: standard deviation and range

“Statistics of dispersion” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/dispersion.html.

“Standard error of the mean” in McDonald, J.H. 2014. Handbook of Biological Statistics.
www.biostathandbook.com/standarderror.html.

“Measures of the Spread of the Data”, Section 2.7 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

The normal distribution

[Video] “Normal Distribution” from Statistics Learning Center (Dr. Nic). 2016.
www.youtube.com/watch?v=mtH1fmUVkfE.

"The normal distribution – three tricky bits" from Dr. Nic. 2016. Learn and Teach Statistics &
Operations Research. learnandteachstatistics.wordpress.com/2016/01/18/the-normal-
distribution/.

48
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional note on reporting summary statistics honestly

Every time we report a descriptive statistic or the result of a statistical test, we are condensing
information, which may have been a whole data set, into one or a few pieces of information. It is the
job of the analyst to choose the best way to present descriptive and graphical information, and to
choose the correct statistical test or method.

SAT score example

Imagine a set of changes in SAT scores from seven students who took a certain SAT preparation
course. The change in scores, Increase, represents the difference in score, that is the score after taking
the course minus their initial score.

Increase = c(50, 60, 120, -80, -10, 10, 0)

Our first instinct in assessing the course might be to look at the average (mean) change in score. The
result is a mean increase of 21 points, which could be considered a success, perhaps.
mean(Increase)

[1] 21.42857

But we would be remiss to not look at other summary statistics and plots.

Using the Summarize function in the FSA package, we find that the median increase was only 10 points.
The increase for the first quartile was negative, suggesting at least 25% of students got a lower score
afterwards.
library(FSA)

Summarize(Increase,
digits = 2)

n nvalid mean sd min Q1 median Q3 max

7.00 7.00 21.43 63.09 -80.00 -5.00 10.00 55.00 120.00

A histogram of the changes in scores suggests a slight right skew to the data, but that the mass of the
data sits not far from zero.
hist(Increase,
col="gray")

49
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Finally, we’ll compute the 95% confidence intervals for the mean change in score. Looking at the
results for the percentile method, the confidence interval includes zero, suggesting the change in
scores for this course were not statistically different from zero.
library(rcompanion)

Data = data.frame(Increase)

groupwiseMean(data = Data,
var = "Increase",
group = NULL,
traditional = FALSE,
percentile = TRUE)

.id n Mean Conf.level Percentile.lower Percentile.upper

1 <NA> 7 21.4 0.95 -21.4 65.7

Based on the data exploration, it seems that it would be irresponsible or dishonest to simply report
that the average increase in test scores was 20 points.

For descriptive statistics, we might report the mean and 95% confidence interval. Or perhaps the 5-
point summary for the change in scores, or show the histogram of values.

Optional analyses

Robust estimators: trimmed mean and Winsorized mean

Robust estimators of central tendency are used to describe the location of a data set without undue
influence of extreme values. Robust estimators include trimmed means, Winsorized means, and other
estimates like M-estimators (not discussed here).

50
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Example
The New Jersey Envirothon is a statewide competition for high school students covering various
aspects of natural resources science. In the Team Presentation station, student teams are judged by
five judges.

Currently, scorekeepers drop the highest and lowest scores for each team to avoid the effect of
aberrant low or high scores. This is an example of a trimmed mean. In this case, the mean is trimmed
to 60% (with 20% removed from each side).

Another option to ameliorate the effect of extreme scores is using Winsorized means. A Winsorized
mean removes extreme observations, but replaces them with the closest observations in terms of
magnitude.

In this example, two teams received five scores for their presentations that have the same mean and
median. We will look at robust estimators to determine if one team should be scored higher than the
other.

Team.A = c(100, 90, 80, 60, 20)

median(Team.A)

[1] 80

mean(Team.A)

[1] 70

mean(Team.A,
trim = .20) # This trims to the inner 60% of observations

[1] 76.66667

library(psych)

winsor(Team.A,
trim = 0.20) # This Winsorizes to the inner 60% of observations

[1] 92 90 80 60 52

### Note that the Winsorized values at the extremes appear to be calculated
### with a function analogous to the quantile(x, probs, type = 7) function.

winsor.mean(Team.A,
trim = 0.20) # This Winsorizes to the inner 60% of observations

[1] 74.8

51
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Team.B = c(80, 80, 80, 80, 30)

median(Team.B)

[1] 80

mean(Team.B)

[1] 70

mean(Team.B,
trim = .20) # This trims to the inner 60% of observations

[1] 80

library(psych)

winsor(Team.B,
trim = 0.20) # This Winsorizes to the inner 60% of observations

[1] 80 80 80 80 70

### Note that the Winsorized values at the extremes appear to be calculated
### with a function analogous to the quantile(x, probs, type = 7) function.

winsor.mean(Team.B,
trim = 0.20) # This Winsorizes to the inner 60% of observations

[1] 78

In this example, the means and medians for Team A and Team B were identical. However, the
trimmed mean for Team A was less than for Team B (77 vs. 80), and the Winsorized mean for A was
less than for Team B (75 vs. 78). According to either the trimmed mean method or the Winsorized
mean method, Team B had a higher score.

Note also that the Winsorized means were lower than for the trimmed means, for both teams. This is
because the Winsorized mean is better able to take into account the low scores for each team.

Geometric mean

The geometric mean is used to summarize certain measurements, such as average return for
investments, and for certain scientific measurements, such as bacteria counts in environmental water.
It is useful when data are log-normally distributed.

Practically speaking, using the geometric mean ameliorates the effect of outlying values in the data.

52
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

To get the geometric mean, the log of each value is taken, these values are averaged, and then the
result is the base of the log raised to this value.

Imagine a series of 9 counts of bacteria from lake water samples, called Bacteria here. The geometric
mean can be calculated with a nested log, mean, and exp functions. Or more simply, the
geometric.mean function in the psych package can be used.

Bacteria example
Bacteria = c(20, 40, 50, 60, 100, 120, 150, 200, 1000)

exp(mean(log(Bacteria)))

[1] 98.38887

library(psych)

geometric.mean(Bacteria)

[1] 98.38887

hist(Bacteria, ### Produce a histogram of values

col="darkgray",
breaks="FD")

Annual return example

Geometric means are also used to calculate the average annual return for investments. Each value in
this vector represents the percent return of the investment each year. The arithmetic mean will not
give the correct answer for average annual return.
Return = c(-0.80, 0.20, 0.30, 0.30, 0.20, 0.30)

library(psych)

geometric.mean(Return + 1)-1

[1] -0.07344666

### The geometric mean will give the correct answer for average
### annual return

This can also be calculated manually. If you start with 100 dollars with this investment, you will end
up with 63 dollars, equivalent to a return of – 0.07.
100 * (1-0.80) * 1.20 * 1.30 * 1.30 * 1.20 * 1.30

[1] 63.2736

53
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Harmonic mean

The harmonic mean is another type of average that is used in certain situations, such as averaging
rates or speeds.

To get the harmonic mean, the inverse of each value is taken, these values are averaged, and then the
inverse of this result is reported.

Imagine a series of 9 speeds, called Speed here. The harmonic mean can be calculated with a nested
1/x, mean, and 1/x functions. Or more simply, the harmonic.mean function in the psych package can be
used.
Speed = c(20, 40, 50, 60, 100, 120, 150, 200, 1000)

1/mean(1/Speed)

63.08411

library(psych)

harmonic.mean(Speed)

63.08411

Exercises C

1. Considering Ren and Stimpy's workshops,

• How many total attendees were there?

• How many total workshops were there?

• How many locations were there?

• What was the mean number of attendees?

• What was the median number of attendees?

2. Considering Ren and Stimpy's workshops,

• Which workshops had a higher mean number of attendees and what was the mean? Ren’s or
Stimpy’s?

• Which workshops had a higher mean number of attendees and what was the mean? Ren
North or Ren South?

3. Considering Arthur and Buster's workshops,

54
 DESCRIPTIVE STATISTICS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• How many students attended in total?

• How many distinct levels of education were reported?

• What was the median level of education reported?

• 75% of students had what level of education or lower?

• What was the median education level for females?

• What was the median education level for males?

• What was the median education level for females from Elwood County?

• What was the median education level for males from Elwood County?

4. Considering Arthur and Buster's workshops,

• How many students were male?

• What percentage is this of total students?

• How many students in the November workshop were male?

• What percentage is this of total students in the November workshop?

• How many male students in the November workshop were from Bear Lake County?

5. As part of a nutrition education program, extension educators had students keep diaries of what
they ate for a day and then calculated the calories students consumed. The following data are the
result. Rating indicates the score, from 1 to 5, that students gave as to the usefulness of the program.
It should be considered an ordinal variable.
Student Teacher Sex Calories Rating
a Tetsuo male 2300 3
b Tetsuo female 1800 3
c Tetsuo male 1900 4
d Tetsuo female 1700 5
e Tetsuo male 2200 4
f Tetsuo female 1600 3
g Tetsuo male 1800 3
h Tetsuo female 2000 3
i Kaneda male 2100 4
j Kaneda female 1900 5
k Kaneda male 1900 4
l Kaneda female 1600 4
m Kaneda male 2000 4
n Kaneda female 2000 5

55
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

o Kaneda male 2100 3

p Kaneda female 1800 4

a. What are the variables in this data set and what type of variable is each? (Types are “nominal”,
“ordinal”, and “interval/ratio”, not how their types are reported by R.)

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

b. How many students were involved in this data set?

c. What was the mean caloric intake?

d. What was the median caloric intake?

e. What was the standard deviation of caloric intake?

f. What was the mean caloric intake for females?

g. What was the mean caloric intake for females in Kaneda’s class?

h. How many males are in Kaneda’s class?

Confidence Intervals
Packages used in this chapter

The packages used in this chapter include:

• Rmisc
• DescTools
• plyr
• boot
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(Rmisc)){install.packages("Rmisc")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(plyr)){install.packages("plyr")}
if(!require(boot)){install.packages("boot")}
if(!require(rcompanion)){install.packages("rcompanion")}

Understanding confidence intervals

56
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Confidence intervals are used to indicate how confident we are that a reported statistic falls within a
certain range. Confidence intervals can be calculated for a variety of statistics, such as the mean,
median, or slope of a linear regression. This chapter will focus on confidences intervals for means.
This book contains a separate chapter, Confidence Intervals for Medians, which addresses confidence
intervals for medians. There is also a chapter Confidence Intervals for Proportions in this book.

The Statistics Learning Center video in the Required Readings below gives a good explanation of the
meaning of confidence intervals.

Populations and samples

Most of the statistics we use assume we are analyzing a sample which we are using to represent a
larger population. If extension educators want to know about the caloric intake of 7th graders, they
would be hard-pressed to get the resources to have every 7th grader in the U.S. keep a food diary.
Instead they might collect data from one or two classrooms, and then treat the data sample as if it
represents a larger population of students.

The mean caloric intake could be calculated for this sample, but this mean will not be exactly the same
as the mean for the larger population. If we collect a large sample and the values aren’t too variable,
then the sample mean should be close to the population mean. But if we have few observations, or the
values are highly variable, we are less confident our sample mean is close to the population mean.

We will use confidence intervals to give a sense of this confidence.

It’s best not to overthink the discussion on populations and samples. We aren’t necessarily actually
extending our statistics to a larger population. That is, we shouldn’t think our measurements from
two classrooms are actually indicative of the whole country. There are likely many factors that would
change the result school to school and region to region. But even if we are thinking about just the 7th
graders in just these two classrooms, most of our statistics will still be based on the assumption that
there is a whole large population of 7th graders in these two classrooms, and we are sampling just a
subset.

Interpretation of confidence intervals

If we were to sample the same population many times and calculated a mean each time, a 95%
confidence interval would indicate the range that 95% of those means would fall into.

As a technically-dubious, but useful, interpretation, we often say that we are then justified in thinking
that the true population mean likely lies within our calculated confidence interval.

In all cases, the confidence interval says something about the calculated statistic. That is, we are
saying something about the mean, not about the underlying data per se.

Confidence intervals as an alternative to some tests

Most of the statistical tests in this book will calculate a probability (p-value) of the likelihood of data
and draw a conclusion from this p-value. John McDonald, in the Optional Readings below, describes
how confidence intervals can be used as an alternative approach.

57
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For example, if we want to compare the means of two groups to see if they are statistically different,
we will use a t-test, or similar test, calculate a p-value, and draw a conclusion. An alternative approach
would be to construct 95% or 99% confidence intervals about the mean for each group. If the
confidence intervals of the two means don’t overlap, we are justified in calling them statistically
different.

Likewise, if the 95% confidence interval for some statistic includes zero, we can conclude that the
statistic is not significantly different from zero.

As a technical note, non-overlapping confidence intervals for means do not equate exactly to a t-test
with a p-value of 0.05. They are different methods to assess similar questions. The article by
Cumming and Finch in the “References” section gives more details on the relationship between
overlapping confidence intervals and p-values from statistical tests.

Example for confidence intervals

For this example, extension educators had students wear pedometers to count their number of steps
over the course of a day. The following data are the result. Rating is the rating each student gave
about the usefulness of the program, on a 1-to-10 scale.
Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8
n female Satsuki 8000 9
o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7
v female Totoro 6000 7
w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

58
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Check the data frame

Data

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Recommended procedures for confidence intervals for means

Confidence intervals for means can be calculated by various methods.

The traditional method is the most commonly encountered, and is appropriate for normally
distributed data or with large sample sizes. It produces an interval that is symmetric about the mean.

For skewed data, confidence intervals by bootstrapping may be more reliable.

For routine use, I recommend using bootstrapped confidence intervals, particularly the BCa or
percentile methods. For further discussion, see below Optional Analyses: confidence intervals for the
mean by bootstrapping.

groupwiseMean function for grouped and ungrouped data

The groupwiseMean function in the rcompanion package can produce confidence intervals both by
traditional and bootstrap methods, for grouped and ungrouped data.

The data must be housed in a data frame. By default, the function reports confidence intervals by the
traditional method.

In the groupwiseMean function, the measurement variable is indicated by the var argument, and the
grouping variable is indicated by the group argument. Variable names are in quotes.

The confidence level is indicated by, e.g., the conf = 0.95 argument. The digits option indicates the
number of significant digits to which the output is rounded. Note that in the output, the means and
other statistics are rounded to 3 significant figures.

Ungrouped data
Ungrouped data is indicated with the group = NULL argument.
library(rcompanion)

groupwiseMean(data = Data,
var = "Steps",
group = NULL,

59
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

conf = 0.95,
digits = 3)

.id n Mean Conf.level Trad.lower Trad.upper

1 <NA> 26 7690 0.95 7170 8210

### Trad.lower and Trad.upper indicate the confidence interval

### for the mean by traditional method.

One-way data
library(rcompanion)

groupwiseMean(data = Data,
var = "Steps",
group = "Sex",
conf = 0.95,
digits = 3)

Sex n Mean Conf.level Trad.lower Trad.upper

1 female 15 8200 0.95 7530 8870
2 male 11 7000 0.95 6260 7740

### Trad.lower and Trad.upper indicate the confidence interval

### for the mean by traditional method.

Two-way data
library(rcompanion)

groupwiseMean(data = Data,
var = "Steps",
group = c("Teacher", "Sex"),
conf = 0.95,
digits = 3)

Teacher Sex n Mean Conf.level Trad.lower Trad.upper

1 Catbus female 6 8000 0.95 6520 9480
2 Catbus male 4 7000 0.95 4400 9600
3 Satsuki female 4 8500 0.95 7580 9420
4 Satsuki male 3 7000 0.95 4520 9480
5 Totoro female 5 8200 0.95 6360 10000
6 Totoro male 4 7000 0.95 5700 8300

### Trad.lower and Trad.upper indicate the confidence interval

### for the mean by traditional method.

Bootstrapped means by group

In the groupwiseMean function, the type of confidence interval is requested by setting certain options
to TRUE. These options are traditional, normal, basic, percentile and bca. The boot option reports an

60
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

optional statistic, the mean by bootstrap. The R option indicates the number of iterations to calculate
each bootstrap statistic.
library(rcompanion)

groupwiseMean(data = Data,
var = "Steps",
group = "Sex",
conf = 0.95,
digits = 3,
R = 10000,
boot = TRUE,
traditional = FALSE,
normal = FALSE,
basic = FALSE,
percentile = FALSE,
bca = TRUE)

Sex n Mean Boot.mean Conf.level Bca.lower Bca.upper

1 female 15 8200 8200 0.95 7470 8670
2 male 11 7000 7000 0.95 6270 7550

library(rcompanion)

groupwiseMean(data = Data,
var = "Steps",
group = c("Teacher", "Sex"),
conf = 0.95,
digits = 3,
R = 10000,
boot = TRUE,
traditional = FALSE,
normal = FALSE,
basic = FALSE,
percentile = FALSE,
bca = TRUE)

Teacher Sex n Mean Boot.mean Conf.level Bca.lower Bca.upper

1 Catbus female 6 8000 8000 0.95 6830 8830
2 Catbus male 4 7000 6990 0.95 5500 8000
3 Satsuki female 4 8500 8500 0.95 8000 8750
4 Satsuki male 3 7000 7000 0.95 6000 7670
5 Totoro female 5 8200 8190 0.95 6800 9000
6 Totoro male 4 7000 7000 0.95 6250 7500

Optional: Other functions for traditional confidence intervals for means

Functions that produce confidence intervals for means by the traditional method include t.test, CI in
Rmisc, and MeanCI in DescTools.
t.test(Data$Steps,
conf.level=0.95)

61
CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One Sample t-test

95 percent confidence interval:

7171.667 8212.949

mean of x
7692.308

library(DescTools)

MeanCI(Data$Steps,
conf.level=0.95)

mean lwr.ci upr.ci

7692.308 7171.667 8212.949

library(Rmisc)

CI(Data$Steps,
ci=0.95)

upper mean lower

8212.949 7692.308 7171.667

library(Rmisc)

group.CI(Steps ~ Sex,
data=Data,
ci = 0.95)

Sex Steps.upper Steps.mean Steps.lower

1 female 8868.482 8200 7531.518
2 male 7735.930 7000 6264.070

group.CI(Steps ~ Teacher + Sex,

data=Data,
ci = 0.95)

Teacher Sex Steps.upper Steps.mean Steps.lower

1 Catbus female 9484.126 8000 6515.874
2 Satsuki female 9418.693 8500 7581.307
3 Totoro female 10041.685 8200 6358.315
4 Catbus male 9598.457 7000 4401.543
5 Satsuki male 9484.138 7000 4515.862
6 Totoro male 8299.228 7000 5700.772

62
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional Analyses: confidence intervals for the mean by bootstrapping

Bootstrapping is a method that samples the data many times, each time calculating a statistic, and then
determining a confidence interval or other statistic from these iterations.

Bootstrapped confidence intervals can be more reliable than those determined by the traditional
method for certain data sets. For more details on the different types of bootstrapped confidence
intervals, see the Carpenter and Bithel article in the “References” section of this chapter. The BCa
method (bias corrected, accelerated) is often cited as the best method, and the percentile method is
also cited as typically good.

The boot package can calculate confidence intervals for means by bootstrap. In the boot function, R
indicates the number of re-samplings.

The function groupwiseMean in the rcompanion package allows for calculating confidence intervals for
means for grouped data, using the bootstrap procedures from the boot package.

Note that for bootstrap procedures, your results may vary slightly from the results reported here.
library(boot)

Mboot = boot(Data$Steps,
function(x,i) mean(x[i]),
R=10000)

mean(Mboot$t[,1])

7693.381

### The mean based on the bootstrap method.

boot.ci(Mboot,
conf = 0.95,
type = c("norm", "basic" ,"perc", "bca")
)

Intervals :
Level Normal Basic
95% (7208, 8174 ) (7192, 8154 )

Level Percentile BCa

95% (7231, 8192 ) (7154, 8115 )
Calculations and Intervals on Original Scale

### Other information

Boot

hist(Boot$t[,1],
col = "darkgray")

63
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Required readings

[Video] “Understanding Confidence Intervals: Statistics Help” from Statistics Learning Center. (Dr.
Nic). 2013. www.youtube.com/watch?v=tFWsuO9f74o.

Optional readings

“Confidence limits” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/confidence.html.

[Video] “Calculating the Confidence interval for a mean using a formula” from Statistics Learning
Center. (Dr. Nic). 2013. www.youtube.com/watch?v=s4SRdaTycaw.

“Confidence Intervals”, Chapter 8 in Openstax. 2013. Introductory Statistics.

openstaxcollege.org/textbooks/introductory-statistics.

“Confidence intervals” , Chapter 4.2 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012.
OpenIntro Statistics, 2nd ed. www.openintro.org/.

References

Carpenter, J. and J. Bithel. 2000. “Bootstrap confidence intervals: when, which, what? A practical guide
for medical statisticians”. Statistics in Medicine 19:1141–1164.
www.tau.ac.il/~saharon/Boot/10.1.1.133.8405.pdf.

Cumming, G. and Finch, S. 2005. Inference by Eye:Confidence Intervals and How to Read Pictures of
Data. American Psychologist 60:170-180.

Optional analyses

Confidence intervals for geometric mean

The geometric mean was discussed in the previous chapter. Here, the CI function in Rmisc is used to
construct confidence intervals for the geometric mean.

Bacteria = c(20, 40, 50, 60, 100, 120, 150, 200, 1000)

library(Rmisc)

exp(CI(log(Bacteria), ci=0.95))

upper mean lower

233.85448 98.38887 41.39484

64
CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Confidence intervals for geometric means for groups

The function groupwiseGeometric in the rcompanion package produces the geometric mean and limits
for the geometric mean plus and minus the standard deviation, standard error, and confidence
interval.
Input = ("
Site Bacteria
A 20
A 40
A 50
A 60
A 100
A 120
A 150
A 200
A 1000

B 100
B 120
B 210
B 300
B 420
B 400
B 500
B 800
B 4000

C 10
C 30
C 40
C 60
C 110
C 100
C 160
C 210
C 1200
")

Data = read.table(textConnection(Input),header=TRUE)

library(rcompanion)

groupwiseGeometric(data = Data,
var = "Bacteria",
group = "Site", ### can use e.g. c("V1", "V2")
digits = 3,
na.rm = TRUE)

Site n Geo.mean sd.lower sd.upper se.lower se.upper ci.lower ci.upper

1 A 9 98.4 31.9 303 67.6 143 41.4 234
2 B 9 389.0 129.0 1170 269.0 561 167.0 906
3 C 9 88.1 22.7 341 56.1 138 31.1 249

65
 CONFIDENCE INTERVALS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Exercises D

1. Considering the Catbus and Satsuki data,

What was the mean of Steps?

What is the 95% confidence interval for Steps (traditional method)?

2. Considering the Catbus and Satsuki data,

What was the mean of Steps for females?

What is the 95% confidence interval for Steps for females (traditional method)?

3. Looking at the 95% confidence intervals for Steps for males and females, are we justified in claiming
that the mean Steps was statistically different for females and males? Why?

4. As part of a nutrition education program, extension educators had students keep diaries of what
they ate for a day and then calculated the calories students consumed.
Student Teacher Sex Calories Rating
a Tetsuo male 2300 3
b Tetsuo female 1800 3
c Tetsuo male 1900 4
d Tetsuo female 1700 5
e Tetsuo male 2200 4
f Tetsuo female 1600 3
g Tetsuo male 1800 3
h Tetsuo female 2000 3
i Kaneda male 2100 4
j Kaneda female 1900 5
k Kaneda male 1900 4
l Kaneda female 1600 4
m Kaneda male 2000 4
n Kaneda female 2000 5
o Kaneda male 2100 3
p Kaneda female 1800 4

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

a. What is the mean of Calories?

b. What is the 95% confidence interval for Calories (traditional method)?

c. What is the mean of Calories for females?

d. What is the 95% confidence interval for Calories for females (traditional method)?

66
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Basic Plots
Packages used in this chapter

The packages used in this chapter include:

• lattice
• plyr
• ggplot2
• FSA
• DescTools
• vcd
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(lattice)){install.packages("lattice")}
if(!require(plyr)){install.packages("plyr")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(FSA)){install.packages("FSA")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(vcd)){install.packages("vcd")}
if(!require(rcompanion)){install.packages("rcompanion")}

The need to understand plots

There are several reasons why it is important to be able to produce and interpret plots of data.

Visualize your data. Plots are essential for visualizing and exploring your data. Plotting a histogram
gives a sense of the range, center, and shape of the data. It shows if the data is symmetric, skewed,
bimodal, or uniform. Box plots and plots of means, medians, and measures of variation visually
indicate the difference in means or medians among groups. Scatter plots show the relationship
between two measured variables.

Communicate your results to others. Being able to choose and produce appropriate plots is important
for summarizing your data for professional or extension presentations and journal articles. Plots can
be very effective for summarizing data in a visual manner and can show statistical results with impact.

Understand other’s results. It is essential to understand the plots you find in extension and
professional literature.

Some advice on producing plots

67
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For some good advice on producing high-quality plots, read the “General tips” section of the McDonald
page listed in the “Optional readings I”.

His basic points:

• “Don’t clutter up your graph with unnecessary junk.”
• “Include all necessary information.”
• “Don’t use color in graphs for publication.”
• “Do use color in graphs for presentations.”

Producing clear and informative plots

There are many ways to make a bad plot. As McDonald notes, one way is to have a cluttered or unclear
plot, and another is to use color inappropriately.

I strive for clear, bold text, lines and points. In a presentation, it can be difficult for the audience to
understand busy plots, or see thin text, lines, or subtle distinctions of colors. Publications may or may
not accept color in plots, and subtle shading or hatching may not be preserved. Sometimes the
resolution of images of plots are downsampled, making small text or symbols unclear.

In general, a plot and its caption should more-or-less be able to stand alone. Obviously, you can’t
describe your whole study in a caption, but it is helpful to give enough information so that the figure is
informative if it were separated from the rest of the article or presentation. The amount of
information will be tempered by the style of medium you’re writing for, journal versus factsheet, for
example. In a scientific article, you may wish to include the subjects (“students in seventh grade who
had completed a 4-H program in robotics”), location, time period, and other relevant information.

Be sure to label axes as thoroughly as possible, and also accurately. For example, an “increase in
student scores” is not the same as “student scores.” Indicate if the values are cumulative, total,
monthly, etc.

When possible, plots should show some measures of variation, such as standard deviation, standard
error of the mean, or confidence interval. By their nature, histograms and box plots show the variation
in the data. Plots of means or medians will need to have error bars added to show dispersion or
confidence limits.

If applicable, usually the x-axis is the independent variable and the y-axis is the dependent variable of
an analysis.

Misleading plots
You should also avoid making misleading plots. Remember that you are trying to accurately
summarize your data and show the relationships among variables. It takes care to do this accurately
and in a way that won’t confuse or mislead your reader.

The “Optional analyses” section on “Misleading and disorienting plots” near the end of this chapter
gives some examples of plots I’ve seen that display their shortcomings.

68
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Using software to produce plots

The plots in this book will be produced using R. R has the capability to produce informative plots
quickly, which is useful for exploring data or for checking model assumptions. It also has the ability to
produce more refined plots with more options, quintessentially through using the package ggplot2.

Sometimes it is easier to produce plots using software with a graphic user interface like Microsoft
Excel, Apple Numbers, or free software like LibreOffice Calc.

In general, you should use whichever tool will best serve your purposes.

Exporting plots
RStudio makes exporting plots from R relatively easy. For lower-resolution images, plots can be
exported directly as image files like .jpg or .png. For higher-resolution images, images can be exported
as .pdf. See the Mangiafico reference in the “Optional readings I” section for some brief instructions.

Also, see “Exercises A” in this book for code on exporting higher resolution image files.

Examples of basic plots for interval/ratio and ordinal data

There are many types of plots, and this chapter will present only a few of the most common ones. In
later chapters, variants of these plots will be used to explore the data or to check the assumptions of
the analyses.

The type of plot you choose will depend upon on the number and types of variables you are trying to
present, and what you are trying to show.

For this example, extension educators had students wear pedometers to count their number of steps
over the course of a day. The following data are the result. Rating is the rating each student gave
about the usefulness of the program, on a 1-to-10 scale.

Note the code that orders the levels of the factor variables. In general, R will alphabetize levels of
factors, and this may be reflected on plots. The code will order the factor by the order that the levels
are encountered in the dataset.
Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8

69
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

n female Satsuki 8000 9

o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7
v female Totoro 6000 7
w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Order the factors, otherwise R with alphabetize them

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Data$Teacher = factor(Data$Teacher,
levels=unique(Data$Teacher))

### Check the data frame

Data

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Histogram
A histogram shows the frequency of observations broken down into ranges, to give a sense of the
range and distribution of the data.

• A single histogram is used for one interval/ratio or ordinal variable.

• Multiple histograms can be put on one plot to compare among groups.

• If you want to make a histogram for counts of a categorical variable in R, you will want to use
a bar plot.

70
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that col defines the colors of the bars; main defines the main plot title; xlab defines the label on
the x-axis.

Simple histogram
hist(Data$Steps,
col="gray",
main="",
xlab="Steps")

Frequency histogram of the number of steps taken by students.

Histogram with density line

This plot will add a curve for the density of the data in the histogram, which is a smoothed-out
representation of the distribution of the data.

The function plotDensityHistogram in the rcompanion package will produce this plot. Options include
the options for the hist function as well as adjust, bw, and kernel, which is passed to the density
function.
library(rcompanion)

plotDensityHistogram(Data$Steps,
adjust = 1) ### Decrease this number
### to make line less smooth

71
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histogram of the number of steps taken by students with the smoothed gaussian
kernel density shown as a black curve.

Histogram with normal curve

This plot will add a normal curve with the mean and standard deviation of the data in the histogram.
This type of plot will be useful to visually determine is a distribution of data is close to normal.

The function plotNormalHistogram in the rcompanion package will produce this plot. Options include
the options for the hist function as well as linecol for the line color.
library(rcompanion)

plotNormalHistogram(Data$Steps)

Histogram of the number of steps taken by students with a normal distribution of

the same mean and standard deviation shown in blue.

Histogram for one-way data

The lattice package can be used to put multiple histograms on one plot for grouped data. This is useful
to compare data distributions among groups. Notice the grammar used in the formula in the
histogram function, and the need to indicate into how many columns and rows the plots will be placed.

72
BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(lattice)

histogram(~ Steps | Sex,

data=Data,
layout=c(1,2) # columns and rows of individual plots
)

Histograms of the number of steps taken by male and female students.

Histogram for two-way data

library(lattice)

histogram(~ Steps | Teacher + Sex,

data=Data,
layout=c(3,2) # columns and rows of individual plots
)

73
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histograms of the number of steps taken by male and female students in each of
the classes of Catbus, Satsuki, and Totoro.

Box plots
Box plots graphically present the five-number summary. The main box indicates the 25th and 75th
percentile of the plotted data. The median is indicated with a heavy line through the box. The
whiskers extending from the box indicate the range of the data, but different software packages may
use somewhat different statistics for the extent of the whiskers. For this reason, when you are
presenting box plots, you should always specify what the length of whiskers indicate.

By default, the whiskers in the boxplot function indicate the range of the data, unless there are data
that are further away from the box than 1.5 times the length of the box. In this case, the whiskers
extend only 1.5 times the length of the box, and points beyond this distance are indicated with circles.
An example of box plots with circles indicating outlying values is shown in the “Box plot for one-way
data” section. If you wish to have the whiskers extend to the range of the data, and not display any
point with circles, the range=0 option can be used. See ?boxplots and ?boxplot.stats for more details.

The video from Statistics Learning Center in the “Optional readings II” section may be a helpful
introduction to box plots. The “Graphing Distributions” chapter in the Lane textbook in the “Optional
readings II” section also has a nice description of box plots.

• A single box plot is used for one interval/ratio or ordinal variable. Don’t use means for
ordinal data.

• Multiple box plots can be put on one plot to compare among groups.

74
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that xlab defines the label on the x-axis; ylab defines the label on the y-axis

Simple box plot

boxplot(Data$Steps,
ylab="Steps",
xlab="")

Box plot of the number of steps taken by students. The box indicates the 1st and
3rd quartiles, and the dark line indicates the median. Whiskers extend to the
maximum and minimum of the data.

Box plot with means

M = mean(Data$Steps)

boxplot(Data$Steps,
ylab="Steps",
xlab="")

points(M,
col="red",
pch="+", ### symbol to use, see ?points
cex=2) ### size of symbol

Box plot of the number of steps taken by students. The box indicates the 1st and
3rd quartiles, and the dark line indicates the median. The mean is indicated
with the red cross. Whiskers extend to the maximum and minimum of the data.

75
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Box plot for one-way data

boxplot(Steps ~ Sex,
data=Data,
ylab="Steps")

Box plots of the number of steps taken by male and female students. The boxes
indicate the 1st and 3rd quartiles, and the dark lines indicate the median.
Points more than 1.5 times the inter-quartile range away from the box are shown
with hollow circles.

Box plot for one-way data with means

M = tapply(Data$Steps,
INDEX = Data$Sex,
FUN = mean)

boxplot(Steps ~ Sex,
data=Data,
ylab="Steps")

points(M,
col="red",
pch="+",
cex=2)

Box plots of the number of steps taken by male and female students. The boxes
indicate the 1st and 3rd quartiles, and the dark lines indicate the median. The

76
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

means are indicated with red crosses. Points more than 1.5 times the inter-
quartile range away from the box are shown with hollow circles.

Box plot for two-way data

boxplot(Steps ~ Sex + Teacher,
data=Data,
ylab="Steps")

Box plots of the number of steps taken by male and female students in each class.
The boxes indicate the 1st and 3rd quartiles, and the dark lines indicate the
median. Points more than 1.5 times the inter-quartile range away from the box
are shown with hollow circles.

Box plot for two-way data with means

M = tapply(Data$Steps,
INDEX = Data$Teacher : Data$Sex,
FUN = mean)

boxplot(Steps ~ Sex + Teacher,

data=Data,
ylab="Steps")

points(M,
col="red",
pch="+",
cex=2)

77
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Box plots of the number of steps taken by male and female students in each class.
The boxes indicate the 1st and 3rd quartiles, and the dark lines indicate the
median. The means are indicated with red crosses. Points more than 1.5 times
the inter-quartile range away from the box are shown with hollow circles.

Plot of means and interaction plots

Means or medians for a group can be plotted with error bars showing standard deviation, standard
error, or confidence interval for the statistic.

For two-way data, these plots are sometimes called interaction plots.

• Each mean or median represents one group or treatment for an interval/ratio or ordinal
variable. Error bars are associated with each mean or median. Don’t use means for ordinal
data.

• Multiple means or medians can be plotted on the same plot, with groups from one or two
independent variables.

The package ggplot2 will be used for this type of plot. A simple interaction plot can be made with the
qplot function, and more refined plots can be made with the ggplot function.

The ggplot2 package is very powerful and flexible for making plots. The tradeoff is that the grammar
can be difficult to understand. When trying to get a plot to look a certain way, it may be necessary to
find an example plot online, and follow the code closely. Good resources include those by Hadley
Wickham, Winston Chang, and Robbins and Robbins in the “References” section.

Plot of means for one-way data with qplot

In this example, we will plot means and confidence intervals. These will first be calculated with the
function groupwiseMean. The output of the function is a data frame, we will call Sum. Sum will then be
passed to the qplot function, so the variables mentioned in the code are from the Sum data frame.

78
BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Steps",
group = "Sex",
conf = 0.95,
digits = 3)

Sum

Sex n Mean Conf.level Trad.lower Trad.upper

1 female 15 8200 0.95 7530 8870
2 male 11 7000 0.95 6260 7740

library(ggplot2)

qplot(x = Sex ,
y = Mean,
data = Sum) +

geom_errorbar(aes(
ymin = Trad.lower,
ymax = Trad.upper,
width = 0.15))

Plot of mean steps taken versus sex, for students in the classes of Catbus,
Satsuki, and Totoro. Error bars indicate the traditional 95% confidence
intervals for the mean.

79
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of means with ggplot

This example will plot the same statistics, but will make a more attractive plot using the ggplot
function. Note that the variables mentioned in the code are from the Sum data frame.

The code invokes the black-and-white visual theme with the option theme_bw.
library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Steps",
group = "Sex",
conf = 0.95,
digits = 3)

Sum

Sex n Mean Conf.level Trad.lower Trad.upper

1 female 15 8200 0.95 7530 8870
2 male 11 7000 0.95 6260 7740

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Sex,
y = Mean)) +
geom_errorbar(aes(ymin = Trad.lower,
ymax = Trad.upper),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +

ylab("Mean steps")

80
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of mean steps taken versus sex, for students in the classes of Catbus,
Satsuki, and Totoro. Error bars indicate the traditional 95% confidence
intervals for the mean.

Interaction plot of means with qplot

This example will plot means and standard errors for the interaction of two independent variables,
Teacher and Sex.

We will use the Summarize function to produce the data frame Sum, and will then add a variable se to
the data frame for the standard error.

Note that the default color palette of ggplot uses a pinkish color for the first group in the legend and a
blueish color for the second. I wasn’t intentionally matching females and males with their traditional
colors. These colors could be changed to avoid the appearance of gender essentialism.
library(FSA)

Sum = Summarize(Steps ~ Teacher + Sex,

data=Data)

Sum$se = Sum$sd / sqrt(Sum$n)

### This creates a new variable for standard error in our summary data frame
### Replace n with nvalid if there are missing values

Sum

Teacher Sex n mean sd min Q1 median Q3 max percZero se

1 Catbus female 6 8000 1414.2136 6000 7250 8000 8750 10000 0 577.3503
2 Satsuki female 4 8500 577.3503 8000 8000 8500 9000 9000 0 288.6751
3 Totoro female 5 8200 1483.2397 6000 8000 8000 9000 10000 0 663.3250

81
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

4 Catbus male 4 4 7000 1632.9932 5000 6500 7000 7500 9000 0 816.4966
5 Satsuki male 3 3 7000 1000.0000 6000 6500 7000 7500 8000 0 577.3503
6 Totoro male 4 4 7000 816.4966 6000 6750 7000 7250 8000 0 408.2483

library(ggplot2)

qplot(x = Teacher,
y = mean,
color = Sex,
data = Sum) +

geom_errorbar(aes(ymin = mean - se,

ymax = mean + se,
width = 0.15))

Interaction plot of mean steps taken versus teacher for males and females. Error
bars indicate the standard error of the mean.

Interaction plot of means with ggplot

As in the example above, this example will plot means and standard errors for the interaction of two
independent variables, Teacher and Sex. As noted above, the pink and blue colors are the default color
palette in ggplot.
library(FSA)

Sum = Summarize(Steps ~ Teacher + Sex,

data=Data)

Sum$se = Sum$sd / sqrt(Sum$n)

### This creates a new variable for standard error in our summary data frame

82
BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Replace n with nvalid if there are missing values

Sum

Teacher Sex n mean sd min Q1 median Q3 max percZero se

1 Catbus female 6 8000 1414.2136 6000 7250 8000 8750 10000 0 577.3503
2 Satsuki female 4 8500 577.3503 8000 8000 8500 9000 9000 0 288.6751
3 Totoro female 5 8200 1483.2397 6000 8000 8000 9000 10000 0 663.3250
4 Catbus male 4 7000 1632.9932 5000 6500 7000 7500 9000 0 816.4966
5 Satsuki male 3 7000 1000.0000 6000 6500 7000 7500 8000 0 577.3503
6 Totoro male 4 7000 816.4966 6000 6750 7000 7250 8000 0 408.2483

library(ggplot2)

pd = position_dodge(.2) ### How much to jitter the points on the plot

ggplot(Sum, ### The data frame to use.

aes(x = Teacher,
y = mean,
color = Sex)) +

geom_point(shape = 15,
size = 4,
position = pd) +

geom_errorbar(aes(ymin = mean - se,

ymax = mean + se),
width = 0.2,
size = 0.7,
position = pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +

ylab("Mean steps")

83
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interaction plot of mean steps taken versus teacher for males and females. Error
bars indicate the standard error of the mean.

Bar plot of means

Another common method to plot means or medians is with bar plots. Error bars can be added to each
bar.

• Each bar represents one group mean or median for an interval/ratio or ordinal variable.
Error bars are associated with each mean or median. Don’t use means for ordinal data.

• Multiple means or medians can be plotted on the same plot, with groups from one or two
independent variables.

The barplot function can produce a simple bar plot, from a table of numbers. For more complex bar
plots, or to add error bars, the ggplot2 package can be used.

Simple bar plot of means

library(FSA)

Sum = Summarize(Steps ~ Sex,

data=Data)

Sum

Sex n nvalid mean sd min Q1 median Q3 max percZero

1 female 15 15 8200 1207.122 6000 8000 8000 9000 10000 0
2 male 11 11 7000 1095.445 5000 6500 7000 7500 9000 0

84
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Table = as.table(Sum$mean)

rownames(Table) = Sum$Sex

Table

female male
8200 7000

barplot(Table,
ylab="Mean steps",
xlab="Sex")

Bar plot of the mean steps taken by female and male students.

Bar plot of means for one-way data with qplot

In this example, we will plot means and confidence intervals. These will first be calculated with the
function groupwiseMean. The output of the function is a data frame, which we will call Sum. Sum will
then be passed to the qplot function, so that the variables mentioned in the code are from the Sum data
frame.
library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Steps",
group = "Sex",
conf = 0.95,
digits = 3)

Sum

Sex n Mean Conf.level Trad.lower Trad.upper

1 female 15 8200 0.95 7530 8870
2 male 11 7000 0.95 6260 7740

85
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(ggplot2)

qplot(x = Sex ,
y = Mean,
data = Sum) +

geom_bar(stat ="identity",
fill = "gray") +

geom_errorbar(aes(ymin = Trad.lower,
ymax = Trad.upper,
width = 0.15))

Bar plot of the mean steps taken by female and male students. Error bars
indicate traditional 95% confidence intervals of the means.

Interaction bar plot of means with ggplot

This example will plot means and standard errors for the interaction of two independent variables,
Teacher and Sex. We will use the Summarize function to produce the data frame Sum, and will then
add a variable se to the data frame for the standard error.

Note that the default color palette of ggplot uses a pinkish color for the first group in the legend and a
blueish color for the second. These colors can be changed.
library(FSA)

Sum = Summarize(Steps ~ Teacher + Sex,

data=Data)

Sum$se = Sum$sd / sqrt(Sum$n)

### This creates a new variable for standard error in our summary data frame
### Replace n with nvalid if there are missing values

Sum

86
BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Teacher Sex n mean sd min Q1 median Q3 max percZero se

1 Catbus female 6 8000 1414.2136 6000 7250 8000 8750 10000 0 577.3503
2 Satsuki female 4 8500 577.3503 8000 8000 8500 9000 9000 0 288.6751
3 Totoro female 5 8200 1483.2397 6000 8000 8000 9000 10000 0 663.3250
4 Catbus male 4 7000 1632.9932 5000 6500 7000 7500 9000 0 816.4966
5 Satsuki male 3 7000 1000.0000 6000 6500 7000 7500 8000 0 577.3503
6 Totoro male 4 7000 816.4966 6000 6750 7000 7250 8000 0 408.2483

library(ggplot2)

pd = position_dodge(1) ### How much to jitter the bars on the plot,

### 0.5 for overlapping bars

ggplot(Sum, ### The data frame to use.

aes(x = Teacher,
y = mean,
fill = Sex)) +

geom_bar(stat = "identity",
color = "black",
position = pd) +

geom_errorbar(aes(ymin = mean - se,

ymax = mean + se),
width = 0.2,
size = 0.7,
position = pd,
color = "black"
) +

theme_bw() +
theme(axis.title = element_text(face = "bold")) +

ylab("Mean steps")

87
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plots of the mean steps taken by female and male students in each class.
Error bars indicate the standard error of the means.

Scatter plot
A scatter plot is used for bivariate data, to show the relationship between two interval/ratio or ordinal
variables. Each point represents a single observation with one measured variable on the x-axis, and
one measured variable is y-axis.

• Data are two interval/ratio or ordinal variables, paired by observation.

• Multiple plots can be shown together to investigate the relationships among multiple
measured variables. (Not shown in this chapter.)

Simple scatter plot

The plot function can produce a simple bivariate scatter plot. The xlab and ylab options indicate the
labels for the x- and y-axes. The jitter function is needed only in cases where individual points will
overlap on the plot.
plot(Rating ~ jitter(Steps), # jitter offsets points so you can see them all
data=Data,
pch = 16, # shape of points
cex = 1.0, # size of points
xlab="Steps",
ylab="Rating")

88
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of program rating versus steps taken by students.

Scatter plot with qplot

The jitter function is needed only in cases where individual points will overlap on the plot.
qplot(x = jitter(Steps),
y = Rating,
data = Data,
xlab = "Steps per day",
ylab = "Rating (1 – 10)")

Plot of program rating versus steps taken by students.

Examples of basic plots for nominal data

Plots for nominal data show the counts of observation for each category of the variable.

89
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plot for counts of a nominal variable

The xtabs function is used to count observations for each level of the nominal variables, producing a
contingency table.
XT = xtabs(~ Sex + Teacher, data=Data)

XT
Teacher
Sex Catbus Satsuki Totoro
female 6 4 5
male 4 3 4

barplot(XT,
beside = TRUE,
legend = TRUE,
ylim = c(0, 8), # adjust to remove legend overlap
xlab = "Teacher",
ylab = "Count")

Bar plot for frequencies of observations for female and male students in each
class.

Bar plot of counts and confidence intervals with ggplot

This example will plot the counts of observations for the interaction of two independent variables,
Teacher and Sex. We will use the Summarize function to produce the data frame Sum, and will use the
variable n as the count of observations.

It’s not necessary you understand the code completely, but in order to demonstrate error bars on this
plot, 95% confidence intervals for the counts will be produced with the MultinomialCI function in the
DescTools package. We call the results of this function MCI. These results will then be converted from
proportions to counts and added to the Sum data frame.

We will use the scale_fill_manual option at the end of the code to change the fill on the bars to grays.

90
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that this example uses two-way data in a data frame, and extracts the count of observations from
the Summarize function. A similar plot, using one-way data, and starting with a vector of counts, is
shown in the “Multinomial test example with plot and confidence intervals” in the Goodness-of-Fit Tests
for Nominal Variables chapter.
library(FSA)

Sum = Summarize(Steps ~ Teacher + Sex,

data=Data)

library(DescTools)

MCI = MultinomCI(Sum$n,
conf.level=0.95)

Total = sum(Sum$n)

Sum$lower = MCI[,'lwr.ci'] * Total

Sum$upper = MCI[,'upr.ci'] * Total

### The few lines above create new variables in the data frame Sum
## for the multinomial confidence intervals for n

Sum

Teacher Sex n nvalid mean sd min Q1 median Q3 max percZero lower upper
1 Catbus female 6 6 8000 1414.2136 6000 7250 8000 8750 10000 0 2 11.27107
2 Satsuki female 4 4 8500 577.3503 8000 8000 8500 9000 9000 0 0 9.27107
3 Totoro female 5 5 8200 1483.2397 6000 8000 8000 9000 10000 0 1 10.27107
4 Catbus male 4 4 7000 1632.9932 5000 6500 7000 7500 9000 0 0 9.27107
5 Satsuki male 3 3 7000 1000.0000 6000 6500 7000 7500 8000 0 0 8.27107
6 Totoro male 4 4 7000 816.4966 6000 6750 7000 7250 8000 0 0 9.27107

library(ggplot2)

pd = position_dodge(0.5) ### How much of a gap for the bars on the plot,
### 1 for non-overlapping bars

ggplot(Sum, ### The data frame to use.

aes(x = Teacher,
y = n,
fill = Sex)) +

geom_bar(stat = "identity",
color = "black",
position = pd) +

geom_errorbar(aes(ymin = lower,
ymax = upper),
width = 0.2,
size = 0.7,
position = pd,
color = "black"

91
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +

scale_fill_manual(values= c("gray80","gray50","green")) +

ylab("Count of observations")

Bar plots of the counts of observations by female and male students in each class.
Error bars indicate the 95% multinomial confidence intervals for each count (Sison
and Glaz method).

Mosaic plot
A mosaic plot can be constructed with the mosaic function in the vcd package.

Interpretation of mosaic plots is discussed in the Introduction to Tests for Nominal Variables chapter.
library(vcd)

mosaic(~ Sex + Teacher,

data = Data,
shade = TRUE,
legend = TRUE)

92
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Mosaic plot for frequencies of observations for female and male students in each
class.

Ordering plot categories

The order in which factor variable levels are plotted can often be controlled by ordering the levels of
the factor, for example,
Data$Teacher = factor(Data$Teacher,
levels=c("Totoro", "Catbus", "Satsuki"))

levels(Data$Teacher)

[1] "Totoro" "Catbus" "Satsuki"

If you are going to be using other variables from the data frame, for example for error bars or mean
separation letters, it is sometimes best to reorder the whole data frame according to the factor
variable you want to use, and then reorder the levels of that variable as well. Sometimes this is
overkill, but it prevents any mismatching of the observations from the different variables.
library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Steps",
group = "Teacher",
conf = 0.95,
digits = 3,
traditional = FALSE,
percentile = TRUE)

Sum

93
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Teacher n Mean Conf.level Percentile.lower Percentile.upper

1 Totoro 9 7670 0.95 6890 8440
2 Catbus 10 7600 0.95 6700 8400
3 Satsuki 7 7860 0.95 7140 8570

Sum = Sum[order(factor(Sum$Teacher,
levels= c("Catbus", "Satsuki", "Totoro"))),]

Sum$Teacher = factor(Sum$Teacher,
levels= c("Catbus", "Satsuki", "Totoro"))

Sum

Teacher n Mean Conf.level Percentile.lower Percentile.upper

2 Catbus 10 7600 0.95 6700 8500
3 Satsuki 7 7860 0.95 7140 8570
1 Totoro 9 7670 0.95 6890 8440

Optional analyses: Describing histogram shapes

The following plots are histograms of water quality data for U.S. Geological Survey station 1467016,
Rancocas Creek at Willingboro, NJ, from approximately 1970 to 1974 (USGS, 2015).

The distribution of pH follows a normal distribution relatively well. It is rather symmetric around a
mean of about 6. A normal curve with the same mean and standard deviation as the data has been
superimposed as a blue line.

The distribution of conductance shows a slight right skew, but otherwise is relatively symmetric
around a mean of about 120.

I would describe the distribution of dissolved oxygen as bimodal. It seems to have relatively many
observations clustered around a value of 3 and around a value of 9.

I’m not sure how I would describe the distribution of temperature. If pressed I might say it is relatively
uniform. A histogram of a uniform distribution would have all bars of equal height. You might see
temperature as bimodal, or of indescribable shape.

Results from the describe function in the psych package are shown below the histograms, and kernel
density plots of these distributions are shown below those results.

For examples of skewed histograms, see the “Skewness” sub-section of the “Statistics of shape for
interval/ratio data” section in the Descriptive Statistics chapter.

94
BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### describe(pH)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 366 6.02 0.24 6 6.01 0.15 5.2 6.7 1.5 0.11 0.84 0.01

### describe(Conductance)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 366 121.95 14.08 121 121.33 13.34 94 162 68 0.38 -0.26 0.74

### describe(DO)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 354 6.32 3.43 6.35 6.25 4.82 0.5 12.4 11.9 0.09 -1.42 0.18

### describe(Temperature)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 366 14.48 7.98 14.55 14.54 11.64 1.5 27.1 25.6 0.02 -1.43 0.42

95
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional analyses: Misleading and disorienting plots

The need to include measures of variation

The following plot mimics one I’ve seen, showing median salary for non-tenure-track university
faculty by race or ethnicity. One minor complaint I have about it is that the x-axis label doesn’t really
describe the x-axis. Really, it should be labeled “Race or ethnicity”, and the categories should be listed
along the x-axis.

The larger complaint I have is that there are no measures of variation indicated for the medians. As
presented, one might say, “Blacks have a higher median salary than do Whites, whose salary is higher
still than that of Hispanics.” But I suspect that the variation within each demographic category is much
greater than the variation between them. If we had error bars on each bar, we might be less inclined
to think that there is a real (statistically significant) difference among groups. Of course, it’s
impossible to draw any conclusions without their including measures of variation or statistical
analyses.

As a final note, usually if there is no statistical difference among groups, there is no need to plot their
values separately. In the example of this plot, if there were no statistical difference among racial
groups, we could just report one median for the whole lot, with a five-number summary, or a single
box plot. There are exceptions to this; for example, in this case if salaries among racial groups were of
some particular interest, we might show statistics for each group even if there were no statistical
differences.

96
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Original caption: “Median academic-year salary for full-time, non-tenure-track

faculty by race or ethnicity, Fall 2010.”

Original source: American Association of University Professors. 2013. Annual Report on the Economic
Status of the Profession, 2012–2013. www.aaup.org/13EconomicStatus.

For bar charts, start the y-axis at zero

The following plots mimic plots I saw for the same data on Fox News and The Guardian. They compare
the enrollment for the Affordable Care Act (“Obamacare”) in March 2014 to the stated goal for
enrollment in March 2014.

The main deficiency of the plot on the left is that the bottom of the y-axis doesn’t represent zero, but
5.75 million, making the difference in the bars look large compared with the size of first bar. For good
measure, units on the y-axis are left off.

The plot on the right more honestly depicts the relationship between the difference in the bars and the
size of the bars by setting the bottom of the y-axis to zero. Also, units are included on the y-axis.

97
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Avoid disorienting the audience

The following plot mimics one released by Reuters in 2014. I didn’t try to mimic the aesthetics of the
plot, but you can see the original in the article cited in “Original source” at the end of the section.

The original plot had the confusing heading and sub-heading, “Gun deaths in Florida: number of
murders committed using firearms”. Of course, murders aren’t the same as gun deaths, so it’s not clear
what is being measured on the y-axis.

A quick look at the plot would suggest that gun murders dropped after 2005. But a more careful
inspection reveals that the y-axis is inverted, in that smaller numbers are on the top and larger
numbers are on the bottom. Therefore, gun murders went up after 2005!

At first I found it difficult to believe that this plot could have been made in good faith. Most commonly
we put larger numbers toward the top of the y-axis. I suspect, though, that the author was using a less-
common convention that positive outcomes are put toward the top of the y-axis. It makes sense if you
think of zeros murders as the goal, and increasing murders are further below that goal. An example of
this convention is the Hickey article in the “Reference” the end of the section. In that article, actors
with a low billing score are at the top of the y-axis, since first billing is the best.

Reuters did release a second plot with the y-axis in its normal orientation, which is mimicked in the
second plot.

As a final note, there are cases where the y-axis is traditionally inverted. For example when plotting
the depth to water table in groundwater hydrology, the surface (0 depth) is plotted at the top, so as the
depth increases, the value is vertically lower on the plot.

98
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

First plot, with y-axis reversed.

Second plot, with y-axis in its common orientation.

Original source: Engel, P. 2014. This chart shows an alarming rise in Florida gun deaths after 'Stand
Your Ground' was enacted. Business Insider. www.businessinsider.com/gun-deaths-in-florida-
increased-with-stand-your-ground-2014-2.

Reference: Hickey, W. 2015. Congratulations, You’ve Been Cast In Star Wars! Will You Ever Work
Again? Five Thirty Eight. fivethirtyeight.com/features/congratulations-youve-been-cast-in-star-wars-
will-you-ever-work-again/.

99
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Use the best model

The following plot mimics one published by the Organisation for Economic Co-operation and
Development. The model they fit to the data is shown as the blue curve. It’s subtle, but the model
makes assumptions that may not fit the data well. In particular, the author chose a simple logarithmic
model for which y continues to increase as x increases. [In reality, I think they chose a model from the
limited options in Excel, probably without thinking about it much.] One implication in this plot is that
life expectancy in the United States, USA, is notably lower than what the model predicts.

But I don’t think there’s any good reason to suspect that life expectancy would continue to increase as
health care spending increases, beyond some limit. Life expectancy is probably limited by human
biology, limits in medical science, and intractable human behaviors. Looking at just the data points
and ignoring the blue curve, the data suggest to me that life expectancy increases as spending
increases up to about 2500 dollars (about where Israel, ISR, is) and then plateaus. Spending beyond
this critical value results in no further increase in life expectancy.

I created the model in the second plot, which uses a quadratic plateau model. With this model, a
quadratic curve rises to a plateau. This model predicts a critical level of about 3500 dollars, about
where Japan, JPN, is. It also suggests that life expectancy in the United States is lower than the model
predicts, but not much more than Switzerland’s, CHE, or Japan’s is above it. But more importantly, this
model and other similar alternatives predict a critical value in healthcare spending, above which we
should not expect an increase in life expectancy. Of course, there are other factors that would affect
life expectancy, and maybe spending above this critical value is warranted in some cases.

The third plot uses a linear plateau model, which is composed of a linear increasing segment and a
plateau segment. It fits the data slightly better than the quadratic plateau model, and indicates a
critical level near 2500 dollars.

First plot with logarithmic model.

100
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Second plot with quadratic plateau model.

Third plot with linear plateau model.

Original source: OECD. 2013. Health at a Glance 2013. www.oecd.org/els/health-systems/Health-at-a-

Glance-2013.pdf.

101
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

References

Apple 2015. “Numbers for Mac.” www.apple.com/mac/numbers/.

Chang, W. “Graphs with ggplot2.” No date. Cookbook for R. www.cookbook-r.com/Graphs/index.html.

LibreOffice. 2015. “Calc.” www.libreoffice.org/discover/calc/.

Microsoft. 2015. “Excel.” products.office.com/en-us/excel.

Robbins, N.B. and J. Robbins. 2016. Effective graphs with Microsoft R Open. www.joyce-
robbins.com/wp-content/uploads/2016/04/effectivegraphsmro1.pdf.

[USGS] U.S. Geological Survey. 2015. National Water Information System: Web Interface.
waterdata.usgs.gov/nwis/.

Wickham, H. 2013. ggplot2 0.9.3.1. docs.ggplot2.org/current/.

Optional readings I

“General tips for all graphs” in “Guide to fairly good graphs” in McDonald, J.H. 2014. Handbook of
Biological Statistics. www.biostathandbook.com/graph.html.

“Exporting graphics” in “A Few Notes to Get Started with R” in Mangiafico, S.S. 2015. An R Companion
for the Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/a_06.html.

Optional readings II

[Video] “Understanding summary statistics: The box plot” from Statistics Learning Center (Dr. Nic).
2015. www.youtube.com/watch?v=bhkqq0w60Gc.

“Graphing Distributions”, Chapter 2 in Lane, D.M. (ed.). No date. Introduction to Statistics, v. 2.0.
onlinestatbook.com/. (Stem and Leaf Displays, Histograms, Frequency Polygons, Box Plots, Bar Charts,
Line Graphs, Dot Plots.)

“Stem-and-Leaf Graphs (Stemplots), Line Graphs, and

Bar Graphs”, Chapter 2.1 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

“Histograms, Frequency Polygons, and Time Series

Graphs”, Chapter 2.2 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

“Box Plots”, Chapter 2.4 in Openstax. 2013. Introductory Statistics.

openstaxcollege.org/textbooks/introductory-statistics.

102
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

“Scatterplots for paired data” , Chapter 1.6.1 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012.
OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Dot plots and the mean” , Chapter 1.6.2 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012.
OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Histograms and shape” , Chapter 1.6.3 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012.
OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Box plots, quartiles, and the median” , Chapter 1.6.5 in Diez, D.M., C.D. Barr , and M. Çetinkaya-
Rundel. 2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Contingency tables and bar plots” , Chapter 1.7.1 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel.
2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Segmented bar and mosaic plot” , Chapter 1.7.3 in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel.
2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.

Exercises E

1. Look at the histogram of the steps taken by male and female students. Would you describe the
distributions of each as symmetrical, skewed, bimodal, or uniform?

2. Look at the box plot of the steps taken by all students. Estimate the minimum, 25th percentile,
median, 75th percentile, and maximum.

3. In box plots including the mean, a symmetrical distribution will have the mean close to the median,
and the box will be symmetrical on either side of the median line. Looking at the box plot of the steps
taken by female and male students, would you describe the distributions of each as symmetrical or
skewed?

4. Looking at the plot of mean steps for female and male students, estimate the mean of each.

5. It is justified to say that group means with non-overlapping 95% confidence intervals are
statistically different. Looking at the plot of mean steps with confidence intervals for female and male
students, are the two means statistically different?

6. Looking at the interaction plot for mean steps for female and male students by each class, do the
error bars for standard error overlap or not for each teacher?

7. Look at the scatter plot. Does it look like there is a correlation or relationship between Steps and
Rating, or not? If so, how would you describe it?

8. Look at any of the plots for nominal data. Are there more females or males in these classes? In
which class is the greatest disparity?

103
 BASIC PLOTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

9–11. As part of a nutrition education program, extension educators had students keep diaries of what
they ate for a day and then calculated the calories students consumed.
Student Teacher Sex Calories Rating
a Tetsuo male 2300 3
b Tetsuo female 1800 3
c Tetsuo male 1900 4
d Tetsuo female 1700 5
e Tetsuo male 2200 4
f Tetsuo female 1600 3
g Tetsuo male 1800 3
h Tetsuo female 2000 3
i Kaneda male 2100 4
j Kaneda female 1900 5
k Kaneda male 1900 4
l Kaneda female 1600 4
m Kaneda male 2000 4
n Kaneda female 2000 5
o Kaneda male 2100 3
p Kaneda female 1800 4

For each of the following, answer the question, and show the plot or other output from the analyses
you used to answer the question.

9. Plot a histogram on the calories consumed for males and females.

a. Write a caption for this plot. Submit the plot and caption.

b. Would you describe the distribution of each as symmetrical, skewed, bimodal, or uniform?

10. Produce box plots of the calories consumed for males and females.

a. Write a caption for this plot. Submit the plot and caption.

b. Report the 25th percentile, median, and 75th percentile. (Estimate from the plot, or report
more precise statistics.)

11. Plot means of the calories consumed for males and females. Include error bars for either
confidence intervals or standard error of the means. You can use either qplot or ggplot, and you can
use either the bar chart style or the "plot of means" style.

a. Write a caption for this plot. Submit the plot and caption.

b. Add a sentence reporting the respective means in the caption.

c. Are the calories consumed likely to be statistically different?

104
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Understanding Statistics and Hypothesis Testing

Hypothesis Testing and p-values

Statistical inference

Most of what we’ve covered in this book so far is about producing descriptive statistics: calculating
means and medians, plotting data in various ways, and producing confidence intervals. The bulk of the
rest of this book will cover statistical inference: using statistical tests to draw some conclusion about
the data. We’ve already done this a little bit in earlier chapters by using confidence intervals to
conclude if means are different or not among groups.

As Dr. Nic mentions in her article in the “References and further reading” section, this is the part where
people sometimes get stumped. It is natural for most of us to use summary statistics or plots, but
jumping to statistical inference needs a little change in perspective. The idea of using some statistical
test to answer a question isn’t a difficult concept, but some of the following discussion gets a little
theoretical. The video from the Statistics Learning Center in the “References and further reading”
section does a good job of explaining the basis of statistical inference.

The most important thing to gain from this chapter is an understanding of how to use the p-value,
alpha, and decision rule to test the null hypothesis. But once you are comfortable with that, you will
want to return to this chapter to have a better understanding of the theory behind this process.

Hypothesis testing

The null and alternative hypotheses

The statistical tests in this book rely on testing a null hypothesis, which has a specific formulation for
each test. The null hypothesis always describes the case where e.g. two groups are not different or
there is no correlation between two variables, etc.

The alternative hypothesis is the contrary of the null hypothesis, and so describes the cases where
there is a difference among groups or a correlation between two variables, etc.

Notice that the definitions of null hypothesis and alternative hypothesis have nothing to do with what
you want to find or don't want to find, or what is interesting or not interesting, or what you expect to
find or what you don’t expect to find. If you were comparing the height of men and women, the null
hypothesis would be that the height of men and the height of women were not different. Yet, you
might find it surprising if you found this hypothesis to be true for some population you were studying.
Likewise, if you were studying the income of men and women, the null hypothesis would be that the
income of men and women are not different, in the population you are studying. In this case you might
be hoping the null hypothesis is true, though you might be unsurprised if the alternative hypothesis
were true. In any case, the null hypothesis will take the form that there is no difference between
groups, there is no correlation between two variables, or there is no effect of this variable in our
model.

105
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

p-value definition
Most of the tests in this book rely on using a statistic called the p-value to evaluate if we should reject,
or fail to reject, the null hypothesis.

Given the assumption that the null hypothesis is true, the p-value is defined as the probability of
obtaining a result equal to or more extreme than what was actually observed in the data.

We’ll unpack this definition in a little bit.

Decision rule
The p-value for the given data will be determined by conducting the statistical test.

This p-value is then compared to a pre-determined value alpha. Most commonly, an alpha value of
0.05 is used, but there is nothing magic about this value.

If the p-value for the test is less than alpha, we reject the null hypothesis.

If the p-value is greater than or equal to alpha, we fail to reject the null hypothesis.

Coin flipping example

For an example of using the p-value for hypothesis testing, imagine you have a coin you will toss 100
times. The null hypothesis is that the coin is fair—that is, that it is equally likely that the coin will land
on heads as land on tails. The alternative hypothesis is that the coin is not fair. Let’s say for this
experiment you throw the coin 100 times and it lands on heads 95 times out of those hundred. The p-
value in this case would be the probability of getting 95, 96, 97, 98, 99, or 100 heads, or 0, 1, 2, 3, 4, or
5 heads, assuming that the null hypothesis is true.

This is what we call a two-sided test, since we are testing both extremes suggested by our data:
getting 95 or greater heads or getting 95 or greater tails. In most cases we will use two sided tests.

You can imagine that the p-value for this data will be quite small. If the null hypothesis is true, and the
coin is fair, there would be a low probability of getting 95 or more heads or 95 or more tails.

Using a binomial test, the p-value is < 0.0001.

(Actually, R reports it as < 2.2e-16, which is shorthand for the number in scientific notation, 2.2 x 10-16,
which is 0.00000000000000022, with 15 zeros after the decimal point.)

If we assume an alpha of 0.05, since the p-value is less than alpha, we reject the null hypothesis. That
is, we conclude that the coin is not fair.

binom.test(5, 100, 0.5)

Exact binomial test

number of successes = 5, number of trials = 100, p-value < 2.2e-16

alternative hypothesis: true probability of success is not equal to 0.5

106
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Passing and failing example

As another example, imagine we are considering two classrooms, and we have counts of students who
passed a certain exam. We want to know if one classroom had statistically more passes or failures
than the other.

In our example each classroom will have 10 students. The data is arranged into a contingency table.
Classroom Passed Failed
A 8 2
B 3 7

We will use Fisher’s exact test to test if there is an association between Classroom and the counts of
passed and failed students. The null hypothesis is that there is no association between Classroom and
Passed/Failed, based on the relative counts in each cell of the contingency table.
Input =("
Classroom Passed Failed
A 8 2
B 3 7
")

Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix

Passed Failed
A 8 2
B 3 7

fisher.test(Matrix)

Fisher's Exact Test for Count Data

p-value = 0.06978

The reported p-value is 0.070. If we use an alpha of 0.05, then the p-value is greater than alpha, so we
fail to reject the null hypothesis. That is, we did not have sufficient evidence to say that there is an
association between Classroom and Passed/Failed.

More extreme data in this case would be if the counts in the upper left or lower right (or both!) were
greater.
Classroom Passed Failed
A 9 1
B 3 7

107
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Classroom Passed Failed

A 10 0
B 3 7

and so on, with Classroom B...

In most cases we would want to consider as "extreme" not only the results when Classroom A has a
high frequency of passing students, but also results when Classroom B has a high frequency of passing
students. This is called a two-sided or two-tailed test. If we were only concerned with one classroom
having a high frequency of passing students, relatively, we would instead perform a one-sided test.
The default for the fisher.test function is two-sided, and usually you will want to use two-sided tests.
Classroom Passed Failed
A 2 8
B 7 3

Classroom Passed Failed

A 1 9
B 7 3

Classroom Passed Failed

A 0 10
B 7 3

and so on, with Classroom B...

In both cases, "extreme" means there is a stronger association between Classroom and Passed/Failed.

Theory and practice of using p-values

Wait, does this make any sense?

Recall that the definition of the p-value is:

Given the assumption that the null hypothesis is true, the p-value is defined as the probability of
obtaining a result equal to or more extreme than what was actually observed in the data.

The astute reader might be asking herself, “If I’m trying to determine if the null hypothesis is true or
not, why would I start with the assumption that the null hypothesis is true? And why am I using a
probability of getting certain data given that a hypothesis is true? Don’t I want to instead determine
the probability of the hypothesis given my data?”

The answer is yes, we would like a method to determine the likelihood of our hypothesis being true
given our data, but we use the Null Hypothesis Significance Test approach since it is relatively
straightforward, and has wide acceptance historically and across fields.

108
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Of course, in practice we do use the results of the statistical tests to reach conclusions about the null
hypothesis. Technically, the p-value says nothing about the alternative hypothesis. But logically, if the
null hypothesis is rejected, then its logical complement, the alternative hypothesis, is supported.
Practically, this is how we handle significant p-values, though this practical approach generates
disapproval in some theoretical circles.

Statistics is like a jury?

Note the language used when testing the null hypothesis. Based on the results of our statistical tests,
we either reject the null hypothesis, or fail to reject the null hypothesis.

This is somewhat similar to the approach of a jury in a trial. The jury either finds sufficient evidence to
declare someone guilty, or fails to find sufficient evidence to declare someone guilty.

Failing to convict someone isn’t necessarily the same as declaring someone innocent. Likewise, if we
fail to reject the null hypothesis, we shouldn’t assume that the null hypothesis is true. It may be that
we didn’t have sufficient samples to get a result that would have allowed us to reject the null
hypothesis, or maybe there are some other factors affecting the results that we didn’t account for. This
is similar to an “innocent until proven guilty” stance.

Errors in inference
For the most part, the statistical tests we use are based on probability, and our data could always be
the result of chance. Considering the coin flipping example above, if we did flip a coin 100 times and
came up with 95 heads, we would be compelled to conclude that the coin was not fair. But 95 heads
could happen with a fair coin strictly by chance.

We can, therefore, make two kinds of errors in testing the null hypothesis:

• A Type I error occurs when the null hypothesis really is true, but based on our decision rule
we reject the null hypothesis. In this case, our result is a false positive; we think there is an
effect (unfair coin, association between variables, difference among groups) when really
there isn’t. The probability of making this kind error is alpha, the same alpha we used in our
decision rule.

• A Type II error occurs when the null hypothesis is really false, but based on our decision rule
we fail to reject the null hypothesis. In this case, our result is a false negative; we have failed
to find an effect that really does exist. The probability of making this kind of error is called
beta.

The following table summarizes these errors.

Reality
___________________________________

Decision of Test Null is true Null is false

Reject null hypothesis Type I error Correctly

109
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

(prob. = alpha) reject null

(prob. = 1 – beta)

Retain null hypothesis Correctly Type II error

retain null (prob. = beta)
(prob. = 1 – alpha)

Statistical power
The statistical power of a test is a measure of the ability of the test to detect a real effect. It is related
to the effect size, the sample size, and our chosen alpha level.

The effect size is a measure of how unfair a coin is, how strong the association between two variables,
or how large the difference among groups. As the effect size increases or as the number of
observations we collect increases, or as the alpha level decreases, the power of the test increases.

Statistical power in the table above is indicated by 1 – beta, which is the probability of correctly
rejecting the null hypothesis.

An example should make these relationship clear. Imagine we are sampling a class of 7th grade
students for their height. In reality, for this class, the girls are taller than the boys, but the difference is
small (that is the effect size is small), and there is a lot of variability in students’ heights. You can
imagine that in order to detect the difference between girls and boys that we would have to measure
many students. If we fail to sample enough students, we might make a Type II error. That is, we might
fail to detect the actual difference in heights between sexes.

If we had a different experiment with a larger effect size—for example the weight difference between
mature hamsters and mature hedgehogs—we might need fewer samples to detect the difference.

Note also, that our chosen alpha plays a role in the power of our test, too. All things being equal, across
many tests, if we decrease our alpha, that is to insist on a lower rate of Type I errors, we are more
likely to commit a Type II error, and so have a lower power. This is analogous to a case of a meticulous
jury that has a very high standard of proof to convict someone. In this case, the likelihood of a false
conviction is low, but the likelihood of a letting a guilty person go free is relatively high.

The 0.05 alpha value is not dogma

The level of alpha is traditionally set at 0.05 in some disciplines, though there is sometimes reason to
choose a different value.

One situation in which the alpha level is increased is in preliminary studies in which it is better to
include potentially significant effects even if there is not strong evidence for keeping them. In this
case, the researcher is accepting an inflated chance of Type I errors in order to decrease the chance of
Type II errors.

Imagine an experiment in which you wanted to see if various environmental treatments would
improve student learning. In a preliminary study, you might have many treatments, with few
observations each, and you want to retain any potentially successful treatments for future study. For
example, you might try playing classical music, improved lighting, complimenting students, and so on,

110
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

and see if there is any effect on student learning. You might relax your alpha value to 0.10 or 0.15 in
the preliminary study to see what treatments to include in future studies.

On the other hand, in situations where a Type I, false positive, error might be costly in terms of money
or people’s health, a lower alpha can be used, perhaps, 0.01 or 0.001. You can imagine a case in which
there is an established treatment say for cancer, and a new treatment is being tested. Because the new
treatment is likely to be expensive and to hold people’s lives in the balance, a researcher would want
to be very sure that the new treatment is more effective than the established treatment. In reality, the
researchers would not just lower the alpha level, but also look at the effect size, submit the research
for peer review, replicate the study, be sure there were no problems with the design of the study or
the data collection, and weigh the practical implications.

The 0.05 alpha value is almost dogma

In theory, as a researcher, you would determine the alpha level you feel is appropriate. That is, the
probability of making a Type I error when the null hypothesis is in fact true.

In reality, though, 0.05 is almost always used in most fields for readers of this book. Choosing a
different alpha value will rarely go without question. It is best to keep with the 0.05 level unless you
have good justification for another value, or are in a discipline where other values are routinely used.

Practical advice
One good practice is to report actual p-values from analyses. It is fine to also simply say, e.g. “The
dependent variable was significantly correlated with variable A (p < 0.05).” But I prefer when possible
to say, “The dependent variable was significantly correlated with variable A (p = 0.026).

It is probably best to avoid using terms like “marginally significant” or “borderline significant” for p-
values less than 0.10 but greater than 0.05, though you might encounter similar phrases. It is better to
simply report the p-values of tests or effects in straight-forward manner. If you had cause to include
certain model effects or results from other tests, they can be reported as e.g., “Variables correlated
with the dependent variable with p < 0.15 were A, B, and C.”

Philosophy of statistical analyses

Statistics is not like a trial

When analyzing data, the analyst should not approach the task as would a lawyer for the prosecution.
That is, the analyst should not be searching for significant effects and tests, but should instead be like
an independent investigator using lines of evidence to find out what is most likely to true given the
data, graphical analysis, and statistical analysis available.

Practical significance and statistical significance

It is important to remember to not let p-values be the only guide for drawing conclusions. It is equally
important to look at the size of the effects you are measuring, as well as take into account practical
considerations like the costs of choosing a certain path of action.

For example, imagine we want to compare the SAT scores of two SAT preparation classes with a t-test.

111
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Class.A = c(1500, 1505, 1505, 1510, 1510, 1510, 1515, 1515, 1520, 1520)

Class.B = c(1510, 1515, 1515, 1520, 1520, 1520, 1525, 1525, 1530, 1530)

t.test(Class.A, Class.B)

Welch Two Sample t-test

t = -3.3968, df = 18, p-value = 0.003214

mean of x mean of y
1511 1521

The p-value is reported as 0.003, so we would consider there to be a significant difference between the
two classes (p < 0.05).

But we have to ask ourselves the practical question, is a difference of 10 points on the SAT large
enough for us to care about? What if enrolling in one class costs significantly more than the other
class? Is it worth the extra money for a difference of 10 points on average?

p-values and effect sizes

It should be remembered that p-values do not indicate the size of the effect being studied. It shouldn’t
be assumed that a small p-value indicates a large difference between groups, or vice-versa.

For example, in the SAT example above, the p-value is fairly small, but the size of the effect (difference
between classes) in this case is relatively small (10 points, especially small relative to the range of
scores students receive on the SAT).

In converse, there could be a relatively large size of the effects, but if there is a lot of variability in the
data or the sample size is not large enough, the p-value could be relatively large.

In this example, the SAT scores differ by 100 points between classes, but because the variability is
greater than in the previous example, the p-value is not significant.

Class.C = c(1000, 1100, 1200, 1250, 1300, 1300, 1400, 1400, 1450, 1500)
Class.D = c(1100, 1200, 1300, 1350, 1400, 1400, 1500, 1500, 1550, 1600)

t.test(Class.A, Class.B)

Welch Two Sample t-test

t = -1.4174, df = 18, p-value = 0.1735

mean of x mean of y
1290 1390 )

boxplot(cbind(Class.C, Class.D))

112
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The problem of multiple p-values

One concept that will be in important in the following discussion is that when there are multiple tests
producing multiple p-values, that there is an inflation of the Type I error rate. That is, there is a higher
chance of making false-positive errors.

This simply follows mathematically from the definition of alpha. If we allow a probability of 0.05, or
5% chance, of making a Type I error for any one test, as we do more and more tests, the chances that at
least one of them having a false positive becomes greater and greater.

p-value adjustment
One way we deal with the problem of multiple p-values in statistical analyses is to adjust p-values
when we do a series of tests together (for example, if we are comparing the means of multiple groups).
In R, the software can adjust the p-values, instead of us changing in our minds our alpha value.

Don’t use Bonferroni adjustments

There are various p-value adjustments available in R. In certain types of tests, in this book we use
Tukey range adjustment. In other cases, we will use FDR, which stands for false discovery rate, and in
R is an alias for the Benjamini and Hochberg method.

Unfortunately, students in analysis of experiments courses often learn to use Bonferroni adjustment
for p-values. This method is simple to do with hand calculations, but is excessively conservative in
most situations, and, in my opinion, antiquated.

There are other p-value adjustment methods, and the choice of which one to use is dictated either by
which are common in your field of study, or by doing enough reading to understand which are
statistically most appropriate for your application.

Preplanned tests
The statistical tests covered in this book assume that tests are preplanned for their p-values to be
accurate. That is, in theory, you set out an experiment, collect the data as planned, and then say “I’m
going to analyze it with kind of model and do these post-hoc tests afterwards”, and report these
results, and that’s all you would do.

113
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Some authors emphasize this idea of preplanned tests, but in reality, of course, researchers will often
collect data, and then use an exploratory data analysis approach to see what the data suggests before
heading into statistical analyses.

Exploratory data analysis

Exploratory data analysis often relies upon examining the data with plotting the data in various ways
and using simple tests like correlation tests to suggest what statistical analysis makes sense.

Sometimes the preplanned approach is appropriate. If an experiment is set out in a specific design,
then usually it is appropriate to use the analysis suggested by this design.

However, in general I recommend an exploratory approach. There may be interesting correlations

hidden in the data. Or perhaps the distribution of the data didn’t allow for the originally planned
analysis.

p-value hacking
It is important when approaching data from an exploratory approach, to avoid committing p-value
hacking. Imagine the case in which the researcher collects many different measurements across a
range of subjects. The researcher might be tempted to simply try different tests and models to relate
one variable to another, for all the variables. He might continue to do this until he found a test with a
significant p-value.

But this would be a form of p-value hacking.

Because an alpha value of 0.05 allows us to make a false-positive error five percent of the time, finding
one p-value below 0.05 after several successive tests may simply be due to chance.

Some forms of p-value hacking are more egregious. For example, if one were to collect some data, run
a test, and then continue to collect data and run tests iteratively until a significant p-value is found.

Publication bias
A related issue in science is that there is a bias to publish, or to report, only significant results. This
can also lead to an inflation of the false-positive rate. As a hypothetical example, imagine if there are
currently 20 similar studies being conducted testing a similar effect—let’s say the effect of
glucosamine supplements on joint pain. If 19 of those studies found no effect and so were discarded,
but one study found an effect using an alpha of 0.05, and was published, is this really any support that
glucosamine supplements decrease joint pain?

A few of xkcd comics

Significant
xkcd.com/882/

Null hypothesis
xkcd.com/892

114
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

P-values
xkcd.com/1478/

Experiments, sampling, and causation

Types of experimental designs

Experimental designs
A true experimental design assigns treatments in a systematic manner. The experimenter must be
able to manipulate the experimental treatments and assign them to subjects. Since treatments are
randomly assigned to subjects, a casual inference can be made for significant results. That is, we can
say that the variation in the dependent variable is caused by the variation in the independent variable.

For interval/ratio data, traditional experimental designs can be analyzed with specific parametric
models, assuming other model assumptions are met. These traditional experimental designs include:

• Completely random design

• Randomized complete block design
• Factorial
• Split-plot
• Latin square

Quasi-experiment designs
Often a researcher cannot assign treatments to individual experimental units, but can assign
treatments to groups. For example, if students are in a specific grade or class, it would not be practical
to randomly assign students to grades or classes. But different classes could receive different
treatments (such as different curricula). Causality can be inferred cautiously if treatments are
randomly assigned and there is some understanding of the factors that affect the outcome.

Observational studies
In observational studies, the independent variables are not manipulated, and no treatments are
assigned. Surveys are often like this, as are studies of natural systems without experimental
manipulation. Statistical analysis can reveal the relationships among variables, but causality cannot be
inferred. This is because there may be other unstudied variables that affect the measured variables in
the study.

Sampling
Good sampling practices are critical for producing good data. In general, samples need to be collected
in a random fashion so that bias is avoided.

In survey data, bias is often introduced by a self-selection bias. For example, internet or telephone
surveys include only those who respond to these requests. Might there be some relevant difference in
the variables of interest between those who respond to such requests and the general population
being surveyed? Or bias could be introduced by the researcher selecting some subset of potential
subjects, for example only surveying a 4-H program with particularly cooperative students and
ignoring other clubs.

115
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In election forecasting, good pollsters need to account for selection bias and other biases in the survey
process. For example, if a survey is done by landline telephone, those being surveyed are more likely
to be older than the general population of voters, and so likely to have a bias in their voting patterns.

Plan ahead and be consistent

It is sometimes necessary to change experimental conditions during the course of an experiment.
Equipment might fail, or unusual weather may prevent making meaningful measurements.

But in general, it is much better to plan ahead and be consistent with measurements.

Consistency
People sometimes have the tendency to change measurement frequency or experimental treatments
during the course of a study. This inevitably causes headaches in trying to analyze data, and makes
writing up the results messy. Try to avoid this.

Controls and checks

If you are testing an experimental treatment, include a check treatment that almost certainly will have
an effect and a control treatment that almost certainly won’t. A control treatment will receive no
treatment and a check treatment will receive a treatment known to be successful. In an educational
setting, perhaps a control group receives no instruction on the topic but on another topic, and the
check group will receive standard instruction.

Including checks and controls helps with the analysis in a practical sense, since they serve as standard
treatments against which to compare the experimental treatments. In the case where the
experimental treatments have similar effects, controls and checks allow you say, for example, “Means
for the all experimental treatments were similar, but were higher than the mean for control, and lower
than the mean for check treatment.”

Include alternate measurements

It often happens that measuring equipment fails or that a certain measurement doesn’t produce the
expected results. It is therefore helpful to include measurements of several variables that can capture
the potential effects. Perhaps test scores of students won’t show an effect, but a self-assessment
question on how much students learned will.

Include covariates
Including additional independent variables that might affect the dependent variable is often helpful in
an analysis. In an educational setting, you might assess student age, grade, school, town, background
level in the subject, or how well they are feeling that day.

The effects of covariates on the dependent variable may be of interest in itself. But also, including co-
variates in an analysis can better model the data, sometimes making treatment effects more clear or
making a model better meet model assumptions.

Optional discussion: Alternative methods to the Null Hypothesis Significance Test

116
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The NHST controversy

Particularly in the fields of psychology and education, there has been much criticism of the null
hypothesis significance test approach. From my reading, the main complaints against NHST tend to
be:

• Students and researchers don’t really understand the meaning of p-values.

• p-values don’t include important information like confidence intervals or parameter

estimates.

• p-values have properties that may be misleading, for example that they do not represent
effect size, and that they change with sample size.

• We often treat an alpha of 0.05 as a magical cutoff value.

Personally, I don’t find these to be very convincing arguments against the NHST approach.

The first complaint is in some sense pedantic: Like so many things, students and researchers learn the
definition of p-values at some point and then eventually forget. This doesn’t seem to impact the
usefulness of the approach.

The second point has weight only if researchers use only p-values to draw conclusions from statistical
tests. As this book points out, one should always consider the size of the effects and practical
considerations of the effects, as well present finding in table or graphical form, including confidence
intervals or measures of dispersion. There is no reason why parameter estimates, goodness-of-fit
statistics, and confidence intervals can’t be included when a NHST approach is followed.

The properties in the third point also don’t count much as criticism if one is using p-values correctly.
One should understand that it is possible to have a small effect size and a small p-value, and vice-versa.
This is not a problem, because p-values and effect sizes are two different concepts. We shouldn’t
expect them to be the same. The fact that p-values change with sample size is also in no way
problematic to me. It makes sense that when there is a small effect size or a lot of variability in the
data that we need many samples to conclude the effect is likely to be real.

(One case where I think the considerations in the preceding point are commonly problematic is when
people use statistical tests to check for the normality or homogeneity of data or model residuals. As
sample size increases, these tests are better able to detect small deviations from normality or
homoscedasticity. Too many people use them and think their model is inappropriate because the test
can detect a small effect size, that is, a small deviation from normality or homoscedasticity).

The fourth point is a good one. It doesn’t make much sense to come to one conclusion if our p-value is
0.049 and the opposite conclusion if our p-value is 0.051. But I think this can be ameliorated by
reporting the actual p-values from analyses. And at some point, one does sometimes have to have a
cutoff value.

Overall it seems to me that these complaints condemn poor practices that the authors observe: not
reporting the size of effects in some manner; not including confidence intervals or measures of
117
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

dispersion; basing conclusions solely on p-values; and not including important results like parameter
estimates and goodness-of-fit statistics.

Alternatives to the NHST approach

Estimates and confidence intervals

One approach to determining statistical significance is to use estimates and confidence intervals.
Estimates could be statistics like means, medians, proportions, or other calculated statistics. To
compare among group medians, one might calculate the medians for each group and a 95% confidence
interval for each median. Then, medians with non-overlapping confidence intervals are considered
statistically significantly different.

This approach is used in a few spots in this book, including in the Confidence Intervals chapter and in
the Jerry Coyne example.

This approach can be very straightforward, easy for readers to understand, and easy to present
clearly.

Another example of when this approach can be used is in the case of estimates from an analysis like
multiple regression. Instead of reporting p-values for terms in the model, the estimates and
confidence intervals for each term in the model could be reported. Terms with confidence intervals
not including zero contribute statistically to the model, and the value of slope coefficient suggests the
size of the contribution of the term.

Bayesian approach
The most popular competitor to the NHST approach is Bayesian inference. Bayesian inference has the
advantage of calculating the probability of the hypothesis given the data, which is what we thought we
should be doing in the “Wait, does this make any sense?” section above. Essentially it takes prior
knowledge about the distribution of the parameters of interest for a population and adds the
information from the measured data to reassess some hypothesis related to the parameters of interest.
If the reader will excuse the vagueness of this description, it makes intuitive sense. We start with what
we suspect to be the case, and then use new data to assess our hypothesis.

One disadvantage of the Bayesian approach is that it is not obvious in most cases what could be used
for legitimate prior information. A second disadvantage is that conducting Bayesian analysis is not as
straightforward as the tests presented in this book.

References and further reading

[Video] “Understanding statistical inference” from Statistics Learning Center (Dr. Nic). 2015.
www.youtube.com/watch?v=tFRXsngz4UQ.

[Video] “Hypothesis tests, p-value” from Statistics Learning Center (Dr. Nic). 2011.
www.youtube.com/watch?v=0zZYBALbZgg.

[Video] “Understanding the p-value” from Statistics Learning Center (Dr. Nic). 2011.

118
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

www.youtube.com/watch?v=eyknGvncKLw.

[Video] “Important statistical concepts: significance, strength, association, causation” from

Statistics Learning Center (Dr. Nic). 2012. www.youtube.com/watch?v=FG7xnWmZlPE.

“Understanding statistical inference” from Dr. Nic. 2015. Learn and Teach Statistics & Operations
Research. learnandteachstatistics.wordpress.com/2015/11/09/understanding-statistical-
inference/.

“Basic concepts of hypothesis testing” in McDonald, J.H. 2014. Handbook of Biological Statistics.
www.biostathandbook.com/hypothesistesting.html.

“Hypothesis testing”, section 4.3, in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012. OpenIntro
Statistics, 2nd ed. www.openintro.org/.

“Hypothesis Testing with One Sample”, sections 9.1–9.2 in Openstax. 2013. Introductory Statistics.
openstaxcollege.org/textbooks/introductory-statistics.

"Proving causation" from Dr. Nic. 2013. Learn and Teach Statistics & Operations Research.
learnandteachstatistics.wordpress.com/2013/10/21/proving-causation/.

[Video] “Variation and Sampling Error” from Statistics Learning Center (Dr. Nic). 2014.
www.youtube.com/watch?v=y3A0lUkpAko.

[Video] “Sampling: Simple Random, Convenience, systematic, cluster, stratified” from Statistics
Learning Center (Dr. Nic). 2012. www.youtube.com/watch?v=be9e-Q-jC-0.

“Confounding variables” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/confounding.html.

“Overview of data collection principles”, section 1.3, in Diez, D.M., C.D. Barr , and M. Çetinkaya-
Rundel. 2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Observational studies and sampling strategies”, section 1.4, in Diez, D.M., C.D. Barr , and M.
Çetinkaya-Rundel. 2012. OpenIntro Statistics, 2nd ed. www.openintro.org/.

“Experiments”, section 1.5, in Diez, D.M., C.D. Barr , and M. Çetinkaya-Rundel. 2012. OpenIntro
Statistics, 2nd ed. www.openintro.org/.

Exercises F

1. Which of the following pair is the null hypothesis?

A) The number of heads from the coin is not different from the number of tails.

B) The number of heads from the coin is different from the number of tails.

119
 HYPOTHESIS TESTING AND P-VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

2. Which of the following pair is the null hypothesis?

A) The height of boys is different than the height of girls.

B) The height of boys is not different than the height of girls.

3. Which of the following pair is the null hypothesis?

A) There is an association between classroom and sex. That is, there is a difference in counts of
girls and boys between the classes.

B) There is no association between classroom and sex. That is, there is no difference in counts
of girls and boys between the classes.

4. We flip a coin 10 times and it lands on heads 7 times. We want to know if the coin is fair.

What is the null hypothesis?

Looking at the code below, and assuming an alpha of 0.05,

What do you decide (use the reject or fail to reject language)?

In practical terms, what do you conclude?

binom.test(7, 10, 0.5)

Exact binomial test

number of successes = 7, number of trials = 10, p-value = 0.3438

5. We measure the height of 9 boys and 9 girls in a class. We want to know if one group is taller than
the other.

What is the null hypothesis?

Looking at the code below, and assuming an alpha of 0.05,

What do you decide (use the reject or fail to reject language)?

In practical terms, what do you conclude?

Girls = c(152, 150, 140, 160, 145, 155, 150, 152, 147)

Boys = c(144, 142, 132, 152, 137, 147, 142, 144, 139)

t.test(Girls, Boys)

120
 REPORTING RESULTS OF DATA AND ANALYSES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Welch Two Sample t-test

t = 2.9382, df = 16, p-value = 0.009645

mean of x mean of y
150.1111 142.1111

boxplot(cbind(Girls, Boys))

6. We count the number of boys and girls in two classrooms. We are interested to know if there is an
association between the classrooms and the number of girls and boys. That is, we want to know if
there are more girls than boys in one of the classes.

What is the null hypothesis?

Looking at the code below, and assuming an alpha of 0.05,

What do you decide (use the reject or fail to reject language)?

In practical terms, what do you conclude?

Classroom Girls Boys
A 13 7
B 5 15

Input =("
Classroom Girls Boys
A 13 7
B 5 15
")

Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

fisher.test(Matrix)

Fisher's Exact Test for Count Data

p-value = 0.02484

Reporting Results of Data and Analyses

121
 REPORTING RESULTS OF DATA AND ANALYSES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Given the variety of experimental designs, potential types of data, and analytical approaches, it is
relatively impossible to develop a cookbook approach to reporting data summaries and analyses. That
being said, it is the intent of this chapter to give some broad and practical advice for this task.

Packages used in this chapter

The packages used in this chapter include:

• FSA

The following commands will install these packages if they are not already installed:
if(!require(FSA)){install.packages("FSA")}

Reporting analyses, results, and assumptions

The following bullets list some of the information you should include in reporting summaries and
analyses. These can be included in a combination of text, plots, tables, plot captions, and table
headings.

A variety of plots are shown in the Basic Plots chapter, as well as in the chapters for specific statistical
analyses. Simple tables for grouped data are shown in the chapters on Descriptive Statistics, Confidence
Intervals, Descriptive Statistics for Likert Data, and Confidence Intervals for Medians.

Procedures

• A description of the analysis. Give the test used, indicate the dependent variables and
independent variables, and describe the experimental design or model in as much detail as is
appropriate. Mention the experimental units, dates, and location.

• Model assumptions checks. Describe what assumptions about the data or residuals are made
by the test and how you assessed your data’s conformity with these assumptions.

• Cite the software and packages you used.

• Cite a reference for statistical procedures or code.

Results

• A measure of the central tendency or location of the data, such as the mean or median for
groups or populations

• A measure of the variation for the mean or median for each group, such standard deviation,
standard error, first and third quartile, or confidence intervals

• Size of the effect. Often this is conveyed by presenting means and medians in a plot or table
so that the reader can see the differences. Sometimes, specific statistics are reported to

122
 REPORTING RESULTS OF DATA AND ANALYSES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

indicate “effect size”. See the “Optional technical note on effect sizes” below for more
information.

• Number of observations, either for each group, or for total observations

• p-value from the analysis

• Goodness-of-fit statistics, if appropriate, such as r-squared, or pseudo R-squared.

• Any other relevant statistics, such as the predictive equation from linear regression or the
critical x value from a linear plateau model.

• Results from post-hoc analyses. Summarize the differences among groups with letters, in a
plot or table. Include the alpha value for group separations

Notes on different data and analyses

Interval/ratio or ordinal data analyzed by group

The bullet points above should be appropriate for this kind of analysis. Data can be summarized in
tables or with in a variety of plots including “plot of means”, “plot of medians”, “interaction plot”, box
plot, histogram, or bar plot. For some of these, error bars can indicate measures of variation. Mean
separations can be indicated with letters within plots.

Nominal data arranged in contingency tables

Nominal data arranged in contingency tables can be presented as frequencies in contingency tables, or
in plots. Bar plots of frequencies can be used. Confidence intervals can be added as error bars to plots.
Another alternative is using mosaic plots.

Bivariate data analyses

Bivariate relationships can be shown with a scatterplot of the two variables, often with the best fit
model drawn in. These models include those from linear regression or curvilinear regression. It is
usually important to present the p-value for the model and the r-squared or pseudo R-squared value.

Advice on tables and plots

Plots
Some advice for producing plots is given in the “Some advice on producing plots” section of the
Basic Plots chapter.

For additional advice on presenting data in plots, see McDonald (2014a) in the “References” section.

Tables
Table style and format will vary widely from publication to publication. As general advice, the style
and format should follow that of the journals or extension publications in your field or institution, or
where you hope to get published.

123
 REPORTING RESULTS OF DATA AND ANALYSES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For additional advice on presenting data in tables, see McDonald (2014b) in the “References” section.

Table headings and plot captions

Both table headings and plot captions should give as much information as is practical to make the table
or plot understandable if it were separated from the rest of the publication.

Example elements

Element Example
Description of data “Mean of reading scores for program curricula”
Experimental units “for 8th grade students”
Location “in Watkins Glen, NY”
Date “1999–2000”
Statistics “Error bars represent standard error of the
mean. The effect of curricula on mean reading
score was significant by one-way ANOVA (p =
0.027). Means sharing a letter not significantly
different by Tukey-adjusted mean separations
(alpha = 0.05). Total observations = 24.”

Example description of statistical analysis and results

The citation information for the R software can be found with:

citation()

Citation information for individual packages can be found with, e.g.:

library(FSA)

citation("FSA")

Procedures
A study of student learning was conducted in 1999–2000 in Watkins Glen, NY. Students were
randomly assigned to one of four curricula, which they studied for one-hour per week under
teacher-supervised conditions, and their scores on an assessment exam was recorded at the
end of the study. A one-way analysis of variance was conducted with student score as the
dependent variable and curriculum as the independent variable (Mangiafico, 2016).
Treatment means were separated by Tukey-adjusted comparisons. Model residuals were
checked for normality and homoscedasticity by visual inspection of residual plots. Analysis of
variance and post-hoc tests were conducted in R (R Core Team, 2016) with the car and lsmeans
packages. Data summary was conducted with the FSA package.

124
 REPORTING RESULTS OF DATA AND ANALYSES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Results

Figure 1. Mean of reading scores for program curricula for 8th grade students in Watkins Glen, NY,
1999–2000. Error bars represent standard error of the mean. The effect of curricula on mean
reading score was significant by one-way ANOVA (p < 0.0001). Means sharing a letter not
significantly different by Tukey-adjusted mean separations (alpha = 0.05). Total observations = 24.

Table 1. Mean of reading scores for program curricula for 8th grade students in
Watkins Glen, NY, 1999–2000. The effect of curricula on mean reading score was
significant by one-way ANOVA (p < 0.0001). Means sharing a letter not significantly
different by Tukey-adjusted mean separations (alpha = 0.05). n indicates number of
observations. Std. err. indicates the standard error of the mean.
Curriculum n Mean score Std. err. Tukey group
( points / 100)
A 6 90 2.89 a
B 6 60 2.89 b
C 6 55 1.29 b
D 6 80 2.89 a

Optional technical note on effect sizes

In this chapter, I use the term “size of the effect” in a general sense of the magnitude of differences
among groups or the degree of association of variables. “Effect size” usually describes specific
statistics for a given analysis, such as Cohen's d, eta-squared, or odds ratio.

For some examples of these statistics, see Sullivan and Feinn (2012) or IDRE (2015) in the
“References” section of this chapter.

125
 CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The pwr package can calculate the effect size for proportions, chi-square goodness-of-fit, and chi-
square test of association. For the effect size for one-way ANOVA (Cohen’s f), see Mangiafico (2015) or
IDRE (2015). For effect sizes for nonparametric tests, see Tomczak and Tomczak (2014), and King and
Rosopa (2010).

References

“One-way Anova” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/d_05.html.

“Guide to fairly good graphs” in McDonald, J.H. 2014a. Handbook of Biological Statistics.
www.biostathandbook.com/graph.html.

“Presenting data in tables” in McDonald, J.H. 2014b. Handbook of Biological Statistics.

www.biostathandbook.com/table.html.

Sullivan, G.M. and R. Feinn. 2012. Using Effect Size—or Why the P Value is Not Enough. Journal
of Graduate Medical Education 4(3): 279–282.
www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/.

Tomczak, M. and Tomczak, E. 2014. The need to report effect size estimates revisited. An
overview of some recommended measures of effect size. Trends in Sports Sciences 1(21):1–
25. www.tss.awf.poznan.pl/files/3_Trends_Vol21_2014__no1_20.pdf.

[IDRE] Institute for Digital Research and Education. 2015. How is effect size used in power
analysis? UCLA.
www.ats.ucla.edu/stat/mult_pkg/faq/general/effect_size_power/effect_size_power.htm/.

King, B.M. and P.J. Rosopa. 2010. Some (Almost) Assumption-Free Tests. In Statistical Reasoning
in the Behavioral Sciences, 6th ed. Wiley.

Choosing a Statistical Test

Choosing a statistical test can be a daunting task for those starting out in the analysis of experiments.
This chapter provides a table of tests and models covered in this book, as well as some general advice
for approaching the analysis of your data.

Plan your experimental design before you collect data

It is important to have an experimental design planned out before you start collecting data, and to
have some an idea of how you plan on analyzing the data. One of the most common mistakes people
make in doing research is collecting a bunch of data without having thought through what questions

126
 CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

they are trying to answer, what specific hypotheses they want to test, and what statistical tests they
can use to test these hypotheses.

What is the hypothesis?

The most important consideration in choosing a statistical test is determining what hypothesis you
want to test. Or, more generally, what question are you are trying to answer.

Often people have a notion about the purpose of the research they are conducting, but haven’t
formulated a specific hypothesis. It is possible to begin with exploratory data analysis, to see what
interesting secrets the data wish to say. But ultimately, choosing a statistical test relies on having in
mind a specific hypothesis to test.

For example, we may know that our goal is to determine if one curriculum works better than another.
But then we must be more specific in our hypothesis. Perhaps we wish to compare the mean of scores
that students get on an exam across the different curricula. Then a specific null hypothesis is, There is
no difference among the mean of student scores across curricula.

In this example, we identified the dependent variable as Student scores, and the independent variable
as Curriculum.

Of course, we might make things more complicated. For example, if the curricula were used in
different classrooms, we might want to include Classroom as an independent blocking variable.

What number and type of variables do you have?

To a large extent, the appropriate statistical test for your data will depend upon the number and types
of variables you wish to include in the analysis.

Consider the type of dependent variable you wish to include.

• If it is of interval/ratio type, you can consider parametric statistics or nonparametric

statistics.

• However, if it is an ordinal variable, you would look toward nonparametric and ordinal
regression models.

• Nominal variables arranged in contingency tables can be analyzed with chi-square and
similar tests. Nominal dependent variables can be related to independent variables with
logistic regression.

• Count data dependent variables can be related to independent variables with Poisson
regression and related models.

127
 CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The number and type of independent variables will also be taken into account. As will whether there
are paired observations or random blocking variables.

The table below lists the tests in this book according to their number and types of variables.

Note that each test has its own set of assumptions for appropriate data, which should be assessed
before proceeding with the analysis.

Also note that the tests in this book cover cases with a single dependent variable only. There are other
statistical tests, included under the umbrella of multivariate statistics that can analyze multiple
dependent variables simultaneously. These include multivariate analysis of variance (MANOVA),
canonical correlation, and discriminant function analysis.

The “References” and “Optional readings” sections of this chapter includes a few other guides to
choosing statistical tests.

Table of tests by variable type

Test DV type, or DV IV type Number of IV Levels in IV Test type
variable type
when there
is no DV
One-sample Ordinal or Independent Single default N/A N/A Nonparametric
Wilcoxon interval/ratio value
Sign test for one- Ordinal or Independent Single default N/A N/A Nonparametric
sample interval/ratio value
Two-sample Mann– Ordinal or Independent Nominal 1 2 Nonparametric
Whitney interval/ratio
Mood’s median test Ordinal or Independent Nominal 1 2 Nonparametric
for two-sample interval/ratio
Two-sample paired Ordinal or Paired Nominal 1, or 2 when 2 Nonparametric
rank-sum interval/ratio one is
blocking
Sign test for two- Ordinal or Paired Nominal 1, or 2 when 2 Nonparametric
sample paired interval/ratio one is
blocking
Kruskal–Wallis Ordinal or Independent Nominal 1 2 or more Nonparametric
interval/ratio
Mood’s median Ordinal or Independent Nominal 1 2 or more Nonparametric
interval/ratio
Friedman Ordinal or Independent Nominal 2 when one is 2 or more Nonparametric
interval/ratio blocked, or blocking,
paired in
unreplicated
complete
block design
Quade Ordinal or Independent Nominal 2 when one is 2 or more Nonparametric
interval/ratio blocked, or blocking, in
paired unreplicated
complete
block design

128
CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One-way Ordinal or Independent Nominal 1 2 or more Permutation

Permutation Test of interval/ratio
Independence
One-way Ordinal or Independent Nominal 2 when one is 2 or more Permutation
Permutation Test of interval/ratio blocked, or blocking
Symmetry paired
Two-sample CLM Ordinal Independent Nominal 1 2 Ordinal
regression
Two-sample paired Ordinal Paired Nominal 2 when one is 2 Ordinal
CLMM blocking regression
One-way ordinal Ordinal Independent Nominal 1 2 or more Ordinal
ANOVA CLM regression
One-way repeated Ordinal Independent Nominal 2 when one is 2 or more Ordinal
ordinal ANOVA blocking regression
CLMM
Two-way ordinal Ordinal Independent Nominal 2 2 or more Ordinal
ANOVA CLM regression
Two-way repeated Ordinal Independent Nominal 3 when one is 2 or more Ordinal
ordinal ANOVA blocking regression
CLMM
Goodness-of-fit tests Nominal Independent Expected N/A Overall: Nominal
for nominal counts vector of
variables counts and
• binomial test expected
• multinomial test proportions
• G-test goodness-
of-fit
• Chi-square test
goodness-of-fit
Association tests for Nominal Independent Nominal N/A Overall: 2- Nominal
nominal variables way
• Fisher exact test of contingency
association table
• G-test of
association
• Chi-square test of
association
Tests for paired Nominal Paired Nominal N/A Overall: 2- Nominal
nominal data way marginal
• McNemar contingency
• McNemar– table
Bowker
Cochran–Mantel– Nominal Independent Nominal N/A Overall: 3- Nominal
Haenszel way
contingency
table
Cochran’s Q Nominal (2 Paired Nominal 2 when one is 2 or more Nominal
levels only) blocking
Linear-by-linear Ordered Independent Ordered N/A Overall: 2- Nominal
nominal nominal way or 3-way
(ordinal) (ordinal) contingency
table

129
CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cochran–Armitage Ordered Independent Nominal N/A Overall: 2- Nominal

(extended) nominal way or 3-way
(ordinal) contingency
table
Log-linear model Nominal Independent Nominal N/A Overall: Generalized
(multiway contingency linear model
frequency analysis) table with 2
or dimensions
Logistic regression Nominal with Independent Interval/ratio 1 or more 2 or more Generalized
(standard) 2 levels or nominal linear model
Multinomial logistic Nominal with Independent Interval/ratio 1 or more 2 or more Generalized
regression 2 or more or nominal linear model
levels
Mixed-effects Nominal with Independent Interval/ratio 1 or more 2 or more Generalized
logistic regression 2 levels or paired or nominal when one is linear model
blocking or
random
One-sample t-test Interval/ratio Independent Single default N/A N/A Parametric
value
Two-sample t-test Interval/ratio Independent Nominal 1 2 Parametric
Paired t-test Interval/ratio Paired Nominal 1, or 2 when 2 Parametric
one is
blocking
One-way ANOVA Interval/ratio Independent Nominal 1 2 or more Parametric
One-way ANOVA Interval/ratio Independent Nominal 2 when one is 2 or more Parametric
with blocks blocking
One-way ANOVA Interval/ratio Independent Nominal 2 when one is 2 or more Parametric
with random blocks blocking
Two-way ANOVA Interval/ratio Independent Nominal 2 2 or more Parametric
Repeated measures Interval/ratio Paired across Nominal 2 or more 2 or more Parametric
ANOVA time when one is
time effect
Multiple correlation Interval/ratio Independent Interval/ratio 1 or more Overall: Parametric or
or ordinal, or ordinal, multiple nonparametric
depending on depending on vectors of depending on
type selected type selected interval/ratio type selected
or ordinal
data
Pearson correlation Interval/ratio Independent Interval/ratio 1 Overall: two Parametric
vectors of
interval/ratio
data
Kendall correlation Interval/ratio Independent Interval/ratio 1 Overall: two Nonparametric
or ordinal or ordinal vectors of
interval/ratio
or ordinal
data
Spearman Interval/ratio Independent Interval/ratio 1 Overall: two Nonparametric
correlation or ordinal or ordinal vectors of
interval/ratio
or ordinal
data
Linear regression Interval/ratio Independent Interval/ratio 1 N/A Parametric

130
 CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Polynomial Interval/ratio Independent Interval/ratio 2 or more N/A Parametric

regression that are
polynomial
terms
Nonlinear Interval/ratio Independent Interval/ratio 1 N/A Parametric
regression and
curvilinear
regression
Multiple regression Interval/ratio Independent Interval/ratio 2 or more N/A Parametric
Robust linear Interval/ratio Independent Interval/ratio 1 N/A Robust
regression parametric
Kendall–Theil Interval/ratio Independent Interval/ratio 1 N/A Nonparametric
regression
Linear plateau and Interval/ratio Independent Interval/ratio 1 N/A Parametric
quadratic plateau
models
Cate–Nelson Interval/ratio Independent Interval/ratio 1 N/A Mostly
analysis nonparametric
Hermite and Count Independent Interval/ratio 1 or more 2 or more Generalized
Poisson regression or nominal linear model
• Hermite
regression
• Poisson regression
• Negative binomial
regression
• Zero-inflated
regression
Beta regression Proportion or Independent Interval/ratio 1 or more 2 or more Generalized
percentage or nominal linear model

Optional discussion: Sometimes it’s all about the hypothesis

Tests that may have analogous purposes, like comparing a measurement variable across two groups,
may test very different hypotheses.

For example, imagine you are investigating the income of two towns. Let’s say the income of Town A
is normally distributed about a mean and median of $48,000. The income of Town B has a similar
median, but has right skew, with some observations close to $1 million.

What test or statistic would you use to compare the income of these two towns?

You might be tempted to compare the means of the two towns with a t-test. In this case, however,
means may not be the best statistic for skewed data, and this data may not meet the assumptions of
the t-test.

You might be interested in comparing the median of the income of the two towns, for example with
Mood’s median test. This might make sense for some regulatory purpose that is concerned with
medians.

131
 CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

On the other hand, looking for a systemic change in the income across the two towns may make more
sense. For example, the higher incomes in Town B may give the town a different character, for
example, some streets with larger homes or upscale stores. For this, you might use the Mann–Whitney
test.

Another approach is to use a permutation test.

Or you might compare the overall distributions of incomes for the two towns using the Kolmogorov–
Smirnov test.

Finally, we might want to compare at the 75th percentile of income for the two towns. This could be
done using quantile regression.

Example
The following code compares some of these results for a hypothetical data set of income in two towns.

Note that the assumptions and pitfalls of these tests are not discussed here, but should be considered
in real situations.
### load required packages

if(!require(FSA)){install.packages("FSA")}
if(!require(psych)){install.packages("psych")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(coin)){install.packages("coin")}
if(!require(quantreg)){install.packages("quantreg")}

### Read the data frame

TwoTowns = read.table("http://rcompanion.org/documents/TwoTowns.csv",
header=TRUE, sep=",")

### Check the data frame

library(psych)

headTail(TwoTowns)

summary(TwoTowns)

### Summarize the data

library(FSA)

Summarize(Income ~ Town,
data=TwoTowns,
digits=3)

Town n mean sd min Q1 median Q3 max

132
CHOOSING A STATISTICAL TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

1 Town.A 101 48146.43 10851.67 23560 40970 48010 56420 77770

2 Town.B 101 115275.22 163878.17 29050 34140 47220 108200 880000

boxplot(Income ~ Town,
data=TwoTowns)

### Mood’s median test

library(RVAideMemoire)

mood.medtest(Income ~ Town,
data = TwoTowns)

Mood's median test

X-squared = 0, df = 1, p-value = 1

### Mann–Whitney test

wilcox.test(Income ~ Town,
data=TwoTowns)

Wilcoxon rank sum test with continuity correction

W = 4672, p-value = 0.3029

alternative hypothesis: true location shift is not equal to 0

### Permutation test

library(coin)

independence_test(Income ~ Town,
data = TwoTowns)

Asymptotic General Independence Test

Z = -3.9545, p-value = 7.669e-05

### Kolmogorov–Smirnov test

library(FSA)

ksTest(Income ~ Town,
data = TwoTowns)

Two-sample Kolmogorov-Smirnov test

D = 0.35644, p-value = 5.349e-06

133
 INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### quantile regression considering the 75th percentile

library(quantreg)

model.q = rq(Income ~ Town,

data = TwoTowns,
tau = 0.75)

model.null = rq(Income ~ 1,
data = TwoTowns,
tau = 0.75)

anova(model.q, model.null)

Quantile Regression Analysis of Deviance Table

Df Resid Df F value Pr(>F)

1 1 200 5.7342 0.01756 *

References

[IDRE] Institute for Digital Research and Education. 2015. What statistical analysis should I use?
UCLA. www.ats.ucla.edu/stat/stata/whatstat/.

“Choosing a statistical test” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/testchoice.html.

Optional readings

[Video] “Choosing which statistical test to use” from Statistics Learning Center (Dr. Nic). 2014.
www.youtube.com/watch?v=rulIUAN0U3w.

Independent and Paired Values

An assumption of most statistical tests is that observations are independent of one another. This
means that the value for one observation is unlikely to be influenced by the value of another
observation. If we pick students at random from a class and measure their height, we can assume the
height of the first student will not affect the height of the next student. These observations would be
independent. If, however, we measured the height of the same students across years, we would expect
that a student who is tall this year would likely be tall the next, and so on. These observations would
not be independent. We might call this second set of observations non-independent, paired, dependent,
or correlated.

134
 INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Dependent samples commonly arise in a few situations. One is repeated measures, in which the same
subject is measured on multiple dates. This is like the student height example described above.

A second is when we are taking multiple measurements of the same individual. An example of this
might be if we are testing students on multiple concepts; we might suspect that if a student scores well
in one section, that she is likely to score well in the other sections. Another example would be
measuring the length of people’s hands. We would suspect that someone with a large left hand is
likely to have a large right hand. A final example would be if student raters were measuring multiple
instructors. We might suspect that a rater who scores one instructor low might be likely to score
another instructor low.

A related concept is that of blocks. If observations can be broken into meaningful groups where values
are likely to be different, this should be taken into account. For example, if we are measuring students’
scores from two classes, and we suspect scores would be lower for one class than the other. If we
were testing instructional methods, we may care about the effect of the instructional methods, and not
care at all about the classes per se, but we want to take differences due to the different classes into
account.

Packages used in this chapter

The packages used in this chapter include:

• FSA
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(FSA)){install.packages("FSA")}
if(!require(rcompanion)){install.packages("rcompanion")}

An example of paired and unpaired data

Input = ("
Individual Hand Length
A Left 17.5
B Left 18.4
C Left 16.2
D Left 14.5
E Left 13.5
F Left 18.9
G Left 19.5
H Left 21.1
I Left 17.8
J Left 16.8
K Left 18.4
L Left 17.3
M Left 18.9
N Left 16.4
O Left 17.5
P Left 15.0
A Right 17.6

135
 INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

B Right 18.5
C Right 15.9
D Right 14.9
E Right 13.7
F Right 18.9
G Right 19.5
H Right 21.5
I Right 18.5
J Right 17.1
K Right 18.9
L Right 17.5
M Right 19.5
N Right 16.5
O Right 17.4
P Right 15.6
")

Data = read.table(textConnection(Input),header=TRUE)

### Note: for the paired test, data must be ordered so that
### the first observation of Group 1
### is the same subject as the first observation of Group 2

### The following will order the data frame by Hand, and then Individual

Data = Data[order(Data$Hand, Data$Individual),]

### Check the data frame

Data

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Box plot and summary statistics by group

Consider an example where left hands and right hands of the same individuals were measured. Below,
the descriptive statistics suggest that left hands and right hands had similar means, medians, and
standard deviations for Length.
library(FSA)

Summarize(Length ~ Hand,
data=Data,
digits=3)

136
 INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

boxplot(Length ~ Hand,
data=Data,
ylab="Length, cm")

Hand n mean sd min Q1 median Q3 max percZero

1 Left 16 17.356 1.948 13.5 16.35 17.50 18.52 21.1 0
2 Right 16 17.594 1.972 13.7 16.35 17.55 18.90 21.5 0

Bar plot to show paired differences

The previous summary statistics, however, do not capture the paired nature of the data. Instead, we
want to investigate the difference between left hand and right hand for each individual. We can
calculate this difference, and use a bar plot to visualize the difference. For most observations, the right
hand was larger, with Right – Left being greater than zero.
Left_hand = Data$Length[Data$Hand=="Left"]

Right_hand = Data$Length[Data$Hand=="Right"]

Difference = Right_hand - Left_hand

barplot(Difference,
col="dark gray",
xlab="Observation",
ylab="Difference (Right – Left)")

137
 INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Paired t-test and unpaired test

t-tests are discussed later in this book. It isn’t important that you understand the test fully at this
point. In this example, a t-test that ignores the pairing of observations found no difference between
the mean length for left hand and right hand, whereas the t-test that accounts for the paired
observations found a significant difference. On average the right hands were about 0.2 cm longer than
their paired left hands.

t.test(Length ~ Hand,
data = Data,
paired = FALSE)

Welch Two Sample t-test

t = -0.3427, df = 29.996, p-value = 0.7342

### No difference between left hand and right if length treated as not paired

t.test(Length ~ Hand,
data = Data,
paired = TRUE)

Paired t-test

t = -3.3907, df = 15, p-value = 0.004034

mean of the differences

-0.2375

### Significant difference between left hand and right

### if length treated as paired

Histogram of differences with normal curve

To be sure our t-test was valid, we’ll plot the differences in hands to be sure their distribution is
approximately normal.
138
INDEPENDENT AND PAIRED VALUES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Left_hand = Data$Length[Data$Hand=="Left"]

Right_hand = Data$Length[Data$Hand=="Right"]

Difference = Right_hand - Left_hand

library(rcompanion)

plotNormalHistogram(Difference,
xlab = "Difference")

### Distribution of differences is probably close enough to normal

### for paired t-test

139
 INTRODUCTION TO LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Likert Data

Introduction to Likert Data

Likert data—properly pronounced like “LICK-ert”—are ordered responses to questions or ratings.
Responses could be descriptive words, such as “agree”, “neutral”, or “disagree,” or numerical, such as
“On a scale of 1 to 5, where 1 is ‘not interested’ and 5 is ‘very interested’…” Likert data is commonly
collected from surveys evaluating education programs, as well in a variety of opinion surveys and
social science surveys.

Likert data

Numbers of responses
Most commonly, a 5- or 7- point scales is used for Likert items. It is believed that most people can
think about or visualize 5 or 7 ordered options easily. Younger children, however, may do better with
a 3-point scale or a simple dichotomous question. On the other hand, if the audience is educated about
a subject and trained in the evaluation, a 10-point scale could be used.

Symmetry
Responses to Likert items are usually symmetrical. That is, if there are options for “agree” and
“strongly agree”, there should be options for “disagree” and “strongly disagree”.

Neutral responses
Responses to Likert items also tend to have a neutral option, such as “neutral”, “neither agree nor
disagree”. But neutral responses may also be terms like “sometimes” or “occasionally” if “never” and
“rarely” on one side are balanced with “often” and “always”.

Form of responses
Numbered responses are typically described with descriptive terms, either for every number, for just
the end points, or for the end points and the middle points, for example:
Strongly Agree Neutral Disagree Strongly
agree disagree

1 2 3 4 5

———————————————————————————————

Strongly Strongly
agree disagree

1 2 3 4 5

———————————————————————————————

Strongly Neutral Strongly

agree disagree

1 2 3 4 5

140
 INTRODUCTION TO LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Other options for Likert responses include faces (smiley face, neutral face, frowny face), and a line on
which respondents mark their response.

Opt-out answers
Questions may also include opt-out responses, like “Don’t know” or “Not applicable”. These are
included outside the Likert responses.
Strongly Neutral Strongly
agree disagree

1 2 3 4 5 Not applicable

I am in favor of including opt-out responses as it tends to encourage more honest responses. It seems
to me it is better to allow a respondent to opt out of answering a question rather than force an
inauthentic response. It is possible that a respondent has no opinion or doesn’t understand a question,
or that a question is not applicable for them.

That being said, the opt-out answer “Don’t know” may not be a great choice, simply because
respondents and researchers may interpret “Don’t know” as a “Neutral” answer. It may be better to
choose less ambiguous opt-out answers like “Not applicable”.

Examples of Likert item responses

See:
Vagias, W.M. (2006). Likert-type scale response anchors. Clemson International Institute for
Tourism & Research Development, Department of Parks, Recreation and Tourism
Management. Clemson University. www.clemson.edu/centers-
institutes/tourism/documents/sample-scales.pdf.

Brown, S. Likert Scale Examples for Surveys. Iowa State University Extension.
www.extension.iastate.edu/ag/staff/info/likertscaleexamples.pdf.

Likert items and Likert scales

Technically, a Likert item is a single question with Likert responses, whereas a Likert scale is a group
of items viewed together as a single measure. For example, one could have several Likert items with
various questions about religious attitudes or behaviors, and then combine those items to a single
Likert scale on religiosity.

When presenting methods and results, it is important to be clear if data were handled as Likert item
data or Likert scale data.

This book will treat Likert data as individual Likert items, and will not create Likert scales.

Analysis of Likert data

141
 INTRODUCTION TO LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Likert data should be treated as ordinal data

There is some agreement that Likert item data should generally be treated as ordinal and not treated
as interval/ratio data.

One consideration is that values in interval/ratio data need to be equally spaced. That is, 2 is equally
between 1 and 3, and you could average 1 and 3 and the response would be 2. But it is not clear that
“agree” is equally spaced between “strongly agree” and “neutral”. Nor is it clear that “strongly agree”
and “neutral” could be averaged for a result of “agree”. Simply numbering the response levels does not
make the responses interval/ratio data.

This book will treat Likert data as ordinal data. It will avoid using parametric tests, such as t-test and
ANOVA, with Likert data. Likert data typically do not meet the assumptions of those parametric tests.

Instead, we will use nonparametric tests, permutation tests, and ordinal regression.

A couple other properties suggest that Likert data should not be treated as interval/ratio data. Likert
data are not continuous; that is, there typically aren’t any decimal points in Likert responses. Also, the
responses in Likert data are constrained at their ends; that is, on a five-point scale, the responses
cannot be below 1 or above 5.

Where it is useful, this book will treat Likert data as nominal data for certain types of summaries. In
general it is better to not treat ordinal data as nominal data in statistical analyses. One reason is that
when treating the data as nominal data, the information about the ordered nature of the response
categories is lost. However, sometimes it is useful to collapse Likert responses into categories; for
example, grouping “strongly agree” and “agree” together as one category and reporting its frequency
as a percentage of responses.

Some people treat Likert data as interval/ratio data

Not everyone agrees that Likert item data should not be treated as interval/ratio data. A quick search
of the internet will produce plenty of examples of people defending treating Likert data as
interval/ratio data.

Cases in which treating Likert responses as interval/ratio data may be reasonable include:
• When there are a high number of response options per question (say 10)
• When only the endpoints of the responses are indicated with text descriptors
• When response options are assumed to be equally spaced
• When respondents mark their answer on a line so that the precise location of the mark can be
measured

Analysis of Likert scale data

When several Likert items are combined into a scale, so that there are many possible numeric
outcomes, the results are often treated as interval/ratio data.

This is not entirely permissible from a theoretical point of view since Likert scales are made up of
Likert items, and so have the same properties. But it is often a reasonable approach if the data meet
the assumptions of the analysis. This is particularly the case if the scale data take on many values.

142
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Analysis of Likert data

Ordinal regression
Probably the best tool for the analysis of experiments with Likert item data as the dependent variable
is ordinal regression. The ordinal package in R provides a powerful and flexible framework for ordinal
regression. It can handle a wide variety of experimental designs, including those with paired or
repeated observations. Ordinal regression is relatively easy to perform in R, but might be somewhat
challenging for the novice in statistical analyses. Occasionally there are problems with fitting models
or checking model assumptions. These cases may be frustrating for the novice user.

Permutation tests
Another tool appropriate for the analysis of Likert item data are permutation tests. The coin package
in R provides a relatively powerful and flexible framework for permutation tests with ordinal
dependent variables. It can handle models analogous to a one-way analysis of variance with blocks,
including paired or repeated observations in a complete block design. This covers more than all the
designs that can be handled with the common traditional nonparametric tests.

Traditional nonparametric tests

Traditional nonparametric tests are generally considered appropriate for analyses with ordinal
dependent variables. They have the advantages of being widely used and likely to be familiar for
readers. One disadvantage of these tests is that the variety of designs they can handle is limited. The
Kruskal–Wallis test can analyze a model analogous to a one-way analysis of variance. The Friedman
and Quade tests can analyze data in an unreplicated complete block design with paired or repeated
observations.

As a technical note, some authors have questioned using these tests with Likert item data. One
consideration is that the underlying statistics for some tests are based on the dependent variable being
continuous in nature. Another consideration is that, while these tests have provisions to handle tied
values, that they may not behave well when there are many ties, as is likely for Likert data.

Optional reading

“Oh Ordinal data, what do we do with you?” from Dr. Nic. 2013. Learn and Teach Statistics &
Operations Research. learnandteachstatistics.wordpress.com/2013/07/08/ordinal/.

“Can Likert Scale Data ever be Continuous?” from Grace-Martin, K. 2008. The Analysis Factor.
www.theanalysisfactor.com/can-likert-scale-data-ever-be-continuous/.

Descriptive Statistics for Likert Data

Median
For Likert data, the primary measure of location we will use is the median. For a set of numbers, the
median is the middle value when the data are arranged in numerical order. For example, for the set of

143
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

values 1, 2, 3, 4, 5, 5, 5, the median value is 4. If there are an even number of values, the two middle
values are averaged.

In concept, 50% of observations are less than the median, and 50% are greater than the median
(ignoring the median itself in cases of an odd number of observations). You could also find the median
of ordinal data consisting just of text descriptors. For example, for the set of values:

“high school”, “associate degree”, “bachelor’s degree”, “master’s degree”, “doctorate”,

the median is “bachelor’s degree”.

Using medians makes sense for ordinal data, whereas using means would in general not be
appropriate.

Count of responses and bar plot

In order to visualize the location and variation of Likert data, we will look at the distribution of the
responses. The distribution of responses can be enumerated by counting the number of responses for
each response level, and can be visualized with a bar plot of these counts. This is similar to a
histogram of the responses.

Quartiles and percentiles

As a measure of the variation of data, it is also useful to look at the quartiles of the data set. The first
quartile indicates the value below which 25% of the values fall. It is equivalent to the 25th percentile.
The second quartile is equivalent to the median. The third quartile indicates the value below which
75% of the values fall. It is equivalent to the 75th percentile.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ggplot2
• plyr
• boot
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(plyr)){install.packages("plyr")}
if(!require(boot)){install.packages("boot")}
if(!require(rcompanion)){install.packages("rcompanion")}

144
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Descriptive statistics for one-sample data

Imagine a scenario in which 10 respondents are evaluating a single speaker on a single Likert item.
The following example will create a data frame called Data consisting of three variables: Speaker,
Rater, and Likert. The speaker is the same for all observations, and Likert represents the response on a
five-point Likert item.

One-sample data
One-sample data refers to a single set of values that is not broken into groups. In this example, we
have only one speaker with 10 ratings, hence one set of ratings values.

Variables and functions in this example

A new variable, Likert.f, will be created in the data frame. It will have the same values as the Likert
variable, but we will tell R to treat it as an ordered factor variable, whereas Likert is a numeric
variable. When summarizing data as a nominal variable, Likert.f will be used. But when summarizing
data as a numeric variable, Likert will be used.

Note that the str function reports that Speaker is a factor variable, Rater and Likert are integer
variables, and Likert.f is an ordered factor variable. The levels of the factor variables are shown with
the level function.

Recall that Data$Likert tells R to use the Likert variable in the data frame Data.
Input =("
Speaker Rater Likert
'Maggie Simpson' 1 3
'Maggie Simpson' 2 4
'Maggie Simpson' 3 5
'Maggie Simpson' 4 4
'Maggie Simpson' 5 4
'Maggie Simpson' 6 4
'Maggie Simpson' 7 4
'Maggie Simpson' 8 3
'Maggie Simpson' 9 2
'Maggie Simpson' 10 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Create a new variable which is the Likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE,
levels = c("1", "2", "3", "4", "5")
)

### Double check our data frame

library(psych)

145
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

headTail(Data)

Speaker Rater Likert Likert.f

1 Maggie Simpson 1 3 3
2 Maggie Simpson 2 4 4
3 Maggie Simpson 3 5 5
4 Maggie Simpson 4 4 4
... <NA> ... ... <NA>
7 Maggie Simpson 7 4 4
8 Maggie Simpson 8 3 3
9 Maggie Simpson 9 2 2
10 Maggie Simpson 10 5 5

str(Data)

'data.frame': 10 obs. of 4 variables:

$ Speaker : Factor w/ 1 level "Maggie Simpson": 1 1 1 1 1 1 1 1 1 1
$ Rater : int 1 2 3 4 5 6 7 8 9 10
$ Likert : int 3 4 5 4 4 4 4 3 2 5
$ Likert.f: Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 3 4 5 4 4 4 4 3 2 5

levels(Data$Speaker)

[1] "Maggie Simpson"

levels(Data$Likert.f)

[1] "1" "2" "3" "4" "5"

summary(Data)

### Remove unnecessary objects

rm(Input)

Summary treating Likert data as nominal data

The number of responses for each response level of Likert.f can be shown with the summary function,
either on the Likert.f variable itself or as part of the whole data frame Data.

This same summary is then visualized with a bar plot.

summary(Data$Likert.f)

1 2 3 4 5
0 1 2 5 2

### The top row is the value of Likert.f,

146
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### and the next row is the counts for each value.

summary(Data)

Speaker Rater Likert Likert.f

Maggie Simpson:10 Min. : 1.00 Min. :2.00 1:0
1st Qu.: 3.25 1st Qu.:3.25 2:1
Median : 5.50 Median :4.00 3:2
Mean : 5.50 Mean :3.80 4:5
3rd Qu.: 7.75 3rd Qu.:4.00 5:2
Max. :10.00 Max. :5.00

xtabs and bar plot

XT = xtabs(~ Likert.f,
data=Data)

XT

Likert.f
1 2 3 4 5
0 1 2 5 2

barplot(XT,
col="dark gray",
xlab="Maggie's Likert",
ylab="Frequency")

Summary treating Likert data as numeric data

In this code the Likert response data is treated as numeric data. The summary function produces the
median, as well as the minimum, maximum, first quartile, and third quartile for a variable. The
summary function can be used for either a single variable or a whole data frame. The Summarize
function in the FSA package produces those statistics as well as the number of observations (n).

147
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

These statistics are visualized with a box plot. Note that the heavy line in the box is the median, and
the ends of the box are the first quartile and third quartile. The extent of the whiskers in the box plot
defaults to being the most extreme values, but no more than 1.5 times the length of the box itself.
Values beyond the extent of the whiskers will be indicated with circles. Note that in the box plot for
this example that the whiskers extend to the most extreme values in the data (2 and 5), and that there
are no circles in the plot.
summary(Data$Likert)

Min. 1st Qu. Median Mean 3rd Qu. Max.

2.00 3.25 4.00 3.80 4.00 5.00

summary(Data)

Speaker Rater Likert Likert.f

Maggie Simpson:10 Min. : 1.00 Min. :2.00 1:0
1st Qu.: 3.25 1st Qu.:3.25 2:1
Median : 5.50 Median :4.00 3:2
Mean : 5.50 Mean :3.80 4:5
3rd Qu.: 7.75 3rd Qu.:4.00 5:2
Max. :10.00 Max. :5.00

library(FSA)

Summarize(Data$Likert,
digits=3)

n mean sd min Q1 median Q3 max percZero

10.000 3.800 0.919 2.000 3.250 4.000 4.000 5.000 0.000

Box plot
boxplot(Data$Likert,
ylab="Likert scores",
xlab="Maggie's results")

148
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Descriptive statistics for one-way or multi-way data

For this example, imagine that there are two speakers, each of which is being evaluated on each of
three questions on 10-point Likert items. There are six raters, each of whom evaluates each speaker
on each of these items.

One-way data
One-way data refers to a data set with a single measured variable, but divided into groups. In this
example looking at values of Likert for Spongebob and for Patrick would be an example of one-way
data. Note that for one-way data there is a single dependent variable (Likert) and a single independent
factor variable (Speaker).

Looking at Likert for the three levels of Question (Information, Presentation, and Questions) would also
be an example of one-way data.

Multi-way data
Multi-way data refers to looking at a single measured variable divided into groups, where the groups
are defined by at least two factor variables. If we look at values of Likert for each of the two speakers
for each of the three questions, we are looking at two-way data.

Variables and functions in this example

The following example will create a data frame called Data consisting of four variables: Speaker,
Question, Rater, and Likert. An additional variable named Likert.f will be created as an ordered factor
variable with the same value as Likert for each observation.

Note that the str function reports that Speaker, Question, and Rater are factor variables, Likert is an
integer variable, and Likert.f is an ordered factor variable. The levels of the factor variables are shown
with the level function.

Recall that Data$Likert tells R to use the Likert variable in the data frame Data.
Input =("
Speaker Question Rater Likert
Spongebob Information a 5
Spongebob Information b 6
Spongebob Information c 7
Spongebob Information d 8
Spongebob Information e 6
Spongebob Information f 5
Spongebob Presentation a 8
Spongebob Presentation b 7
Spongebob Presentation c 8
Spongebob Presentation d 9
Spongebob Presentation e 10
Spongebob Presentation f 7
Spongebob Questions a 3
Spongebob Questions b 4

149
DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Spongebob Questions c 5
Spongebob Questions d 4
Spongebob Questions e 6
Spongebob Questions f 5
Patrick Information a 6
Patrick Information b 7
Patrick Information c 8
Patrick Information d 9
Patrick Information e 7
Patrick Information f 6
Patrick Presentation a 9
Patrick Presentation b 9
Patrick Presentation c 10
Patrick Presentation d 10
Patrick Presentation e 8
Patrick Presentation f 8
Patrick Questions a 5
Patrick Questions b 6
Patrick Questions c 6
Patrick Questions d 7
Patrick Questions e 5
Patrick Questions f 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factor levels, otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

Data$Question = factor(Data$Question,
levels=unique(Data$Question))

### Create an ordered factor of Likert data

Data$Likert.f = factor(Data$Likert,
ordered=TRUE,
levels = c("1", "2", "3", "4", "5",
"6", "7", "8", "9", "10")
)

### Examine data frame

library(psych)

headTail(Data)

Speaker Question Rater Likert Likert.f

1 Spongebob Information a 5 5
2 Spongebob Information b 6 6
3 Spongebob Information c 7 7

150
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

4 Spongebob Information d 8 8
... <NA> <NA> <NA> ... <NA>
33 Patrick Questions c 6 6
34 Patrick Questions d 7 7
35 Patrick Questions e 5 5
36 Patrick Questions f 7 7

str(Data)

'data.frame': 36 obs. of 5 variables:

$ Speaker : Factor w/ 2 levels "Spongebob","Patrick": 1 1 1 1 1 1 1 1 1 1 ...
$ Question: Factor w/ 3 levels "Information",..: 1 1 1 1 1 1 2 2 2 2 ...
$ Rater : Factor w/ 6 levels "a","b","c","d",..: 1 2 3 4 5 6 1 2 3 4 ...
$ Likert : int 5 6 7 8 6 5 8 7 8 9 ...
$ Likert.f: Ord.factor w/ 10 levels "1"<"2"<"3"<"4"<..: 5 6 7 8 6 5 8 7 8 9 ...

levels(Data$Speaker)

[1] "Spongebob" "Patrick"

levels(Data$Question)

[1] "Information" "Presentation" "Questions"

levels(Data$Rater)

[1] "a" "b" "c" "d" "e" "f"

levels(Data$Likert.f)

[1] "1" "2" "3" "4" "5" "6" "7" "8" "9" "10"

summary(Data)

### Remove unnecessary objects

rm(Input)

Summary treating Likert data as nominal data

The number of responses for each response level of Likert.f is shown with the summary function, but
note that these counts are for the entire data frame.

The xtabs function can be used to generate counts across groups. This process is known as cross-
tabulation, or cross-tabs. The prop.table function, with the margin=1 option produces the proportions
of observations for each row.

151
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

This same summary is then visualized with a bar plot.

summary(Data)

Speaker Question Rater Likert Likert.f

Spongebob:18 Information :12 a:6 Min. : 3.000 6 :7
Patrick :18 Presentation:12 b:6 1st Qu.: 5.750 7 :7
Questions :12 c:6 Median : 7.000 5 :6
d:6 Mean : 6.833 8 :6
e:6 3rd Qu.: 8.000 9 :4
f:6 Max. :10.000 10 :3
(Other):3

XT = xtabs(~ Speaker + Likert.f, data=Data)

XT

Likert.f
Speaker 1 2 3 4 5 6 7 8 9 10
Spongebob 0 0 1 2 4 3 3 3 1 1
Patrick 0 0 0 0 2 4 4 3 3 2

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5 6 7 8 9 10
Spongebob 0.0000 0.0000 0.0556 0.1111 0.2222 0.1667 0.1667 0.1667 0.0556 0.0556
Patrick 0.0000 0.0000 0.0000 0.0000 0.1111 0.2222 0.2222 0.1667 0.1667 0.1111

sum(XT)

[1] 36

### Sum of counts in the table

rowSums(XT)

Spongebob Patrick
18 18

### Sum of counts for each row

colSums(XT)

1 2 3 4 5 6 7 8 9 10
0 0 1 2 6 7 7 6 4 3

### Sum of counts for each column

152
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plot by group

XT = xtabs(~ Speaker + Likert.f, data=Data)

XT

barplot(XT,
beside=TRUE,
legend=TRUE,
ylim=c(0, 5), # adjust to remove legend overlap
xlab="Likert score",
ylab="Frequency"
)

xtabs to produce counts for two-way data

### Note that the dependent variable (Likert.f)
### is the middle variable in the formula

XT2 = xtabs(~ Speaker + Likert.f + Question,

data=Data)

XT2

, , Question = Information

Likert.f
Speaker 1 2 3 4 5 6 7 8 9 10
Spongebob 0 0 0 0 2 2 1 1 0 0
Patrick 0 0 0 0 0 2 2 1 1 0

, , Question = Presentation

Likert.f
Speaker 1 2 3 4 5 6 7 8 9 10
Spongebob 0 0 0 0 0 0 2 2 1 1

153
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Patrick 0 0 0 0 0 0 0 2 2 2

, , Question = Questions

Likert.f
Speaker 1 2 3 4 5 6 7 8 9 10
Spongebob 0 0 1 2 2 1 0 0 0 0

Summary treating Likert data as numeric data

In this code the Likert response data is treated as numeric data. The summary function produces the
median, quartiles, and extremes. But note that these statistics are for all values for the whole data
frame.

The Summary function in the FSA package can produce these summary statistics across groups for
one-way data or multi-way data.

Histograms are then produced to visualize the distribution of responses across groups. Histograms for
both one-way and two-way data can be produced with the histogram function in the lattice package.

Finally, box plots of the two-way data are produced.

summary(Data)

Speaker Question Rater Likert Likert.f

Spongebob:18 Information :12 a:6 Min. : 3.000 6 :7
Patrick :18 Presentation:12 b:6 1st Qu.: 5.750 7 :7
Questions :12 c:6 Median : 7.000 5 :6
d:6 Mean : 6.833 8 :6
e:6 3rd Qu.: 8.000 9 :4
f:6 Max. :10.000 10 :3
(Other):3

library(FSA)

Summarize(Likert ~ Question,
data=Data,
digits=3)

Question n mean sd min Q1 median Q3 max percZero

1 Information 12 6.667 1.231 5 6.00 6.5 7.25 9 0
2 Presentation 12 8.583 1.084 7 8.00 8.5 9.25 10 0
3 Questions 12 5.250 1.215 3 4.75 5.0 6.00 7 0

library(FSA)

Summarize(Likert ~ Speaker + Question,

data=Data,
digits=3)

Speaker Question n mean sd min Q1 median Q3 max percZero

154
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

1 Spongebob Information 6 6.167 1.169 5 5.25 6.0 6.75 8 0

2 Patrick Information 6 7.167 1.169 6 6.25 7.0 7.75 9 0
3 Spongebob Presentation 6 8.167 1.169 7 7.25 8.0 8.75 10 0
4 Patrick Presentation 6 9.000 0.894 8 8.25 9.0 9.75 10 0
5 Spongebob Questions 6 4.500 1.049 3 4.00 4.5 5.00 6 0
6 Patrick Questions 6 6.000 0.894 5 5.25 6.0 6.75 7 0

Bar plots by group

The histogram function in the lattice package will produce bar plots if the counted variable is ordinal
in nature, as Likert.f is.
library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,2) # columns and rows of individual plots
)

library(lattice)

histogram(~ Likert.f | Question,

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

155
DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histograms for two-way data

library(lattice)

histogram(~ Likert.f | Speaker + Question,

data=Data,
layout=c(2,3) # columns and rows of individual plots
)

156
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Box plots for two-way data

In general, using the names option is not necessary, but here it is useful because the default group
names are long.
boxplot(Likert ~ Speaker + Question,
data=Data,
names=c("SB.Inf","P.Inf","SB.Pres", "P.Pres", "SB.Quest", "P.Quest"),
ylab="Value")

### Note in this example, SB is Spongebob, P is Patrick,

### Inf is Information, Pres is Presentation,
### and Quest is Question.

Interaction plot using medians and quartiles

Interaction plots are useful to visualize two-way data by plotting the value of a measured variable
across the interaction of two factor variables.

In this example, the median Likert is calculated for each level of the interaction of two factors, and the
first and third quartiles are used to indicate the spread of data about each median.

The Summarize function in the FSA package creates a data frame called Sum. The variables median, Q1,
and Q3 in this data frame will then be used in the plot.

The interaction plot is produced with the ggplot2 package. This package is very versatile and powerful
for creating plots, but the code can be intimidating at first. Note that the code indicates that the data
frame to use is Sum, and the use of the variables Speaker, median, Question, Q1, Q3 from this data
frame. Also note that the y axis label defined as “Median Likert score”.
### Create a data frame called Sum with median and quartiles

library(FSA)

Sum = Summarize(Likert ~ Speaker + Question,

data=Data,
digits=3)

157
DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Sum

Speaker Question n mean sd min Q1 median Q3 max percZero

1 Spongebob Information 6 6.167 1.169 5 5.25 6.0 6.75 8 0
2 Patrick Information 6 7.167 1.169 6 6.25 7.0 7.75 9 0
3 Spongebob Presentation 6 8.167 1.169 7 7.25 8.0 8.75 10 0
4 Patrick Presentation 6 9.000 0.894 8 8.25 9.0 9.75 10 0
5 Spongebob Questions 6 4.500 1.049 3 4.00 4.5 5.00 6 0
6 Patrick Questions 6 6.000 0.894 5 5.25 6.0 6.75 7 0

### Produce interaction plot

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x=Speaker,
y=median,
color=Question)) +
geom_errorbar(aes(ymin=Q1,
ymax=Q3),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold"))+
ylab("Median Likert score")

158
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interaction plot using medians and confidence intervals

An interaction plot can also use confidence intervals of the medians for its error bars. Confidence
intervals are discussed in the Confidence Intervals chapter, and confidence intervals for medians are
discussed in the Confidence Intervals for Medians chapter.

The function groupwiseMedian in the rcompanion package creates a data frame, called Sum, here that
includes medians and confidence intervals of the medians for one-way or multi-way data. By default
the function uses confidence intervals by the BCa method, but can produce confidence intervals by
other methods.

Note that the order of speakers has changed from the previous interaction plot.
library(rcompanion)

Sum = groupwiseMedian(data = Data,

group = c("Speaker", "Question"),
var = "Likert",
conf=0.95,
R=5000)

Sum

Speaker Question n Median Conf.level Bca.lower Bca.upper

1 Spongebob Information 6 6.0 0.95 5 6.5
2 Spongebob Presentation 6 8.0 0.95 7 8.5
3 Spongebob Questions 6 4.5 0.95 3 5.0
4 Patrick Information 6 7.0 0.95 6 7.5
5 Patrick Presentation 6 9.0 0.95 8 9.5
6 Patrick Questions 6 6.0 0.95 5 6.5

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x=Speaker,
y=Median,
color=Question)) +
geom_errorbar(aes(ymin=Bca.lower,
ymax=Bca.upper),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold"))+
ylab("Median Likert score")

159
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Descriptive statistics with opt-out responses

Opt-out responses include options like “don’t know” or “not applicable”. A typical approach to
handling opt-out responses in Likert data is to separate them from the numeric or ordered factor
responses of the question and report them separately.

For example, if responses to a question were 5, 5, 5, 4, 4, 2, ‘Don’t know’, ‘N/A’, you might report that
you had 8 total responses, with 12.5% “Don’t know”, 12.5% “N/A”, and a median response of 4.5.

Analysis for vector data

The Summarize function in the FSA will calculate the median for numeric data. Note that the numeric
summary includes the number of observations and the number of “valid” observations, which excludes
the non-numeric values.
Q1 = c("5", "5", "5", "4", "4", "2", "Don't know", "N/A")

Q1.f = as.factor(Q1)

summary(Q1.f)

2 4 5 Don't know N/A

1 2 3 1 1

160
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Q1.n = as.numeric(Q1)

library(FSA)

Summarize(Q1.n,
digits = 2)

n nvalid mean sd min Q1 median Q3 max

8.00 6.00 4.17 1.17 2.00 4.00 4.50 5.00 5.00

Analysis for long-form data frame

Similar summaries can be obtained for data in a long-form data frame.

Note that the read.table function by default reads in the values of Likert as a factor variable, since it
contains character values, not as a character variable. Also note that you shouldn’t use NA for a value
of “not applicable” since NA is reserved by R for a missing value.
Input =("
Rater Question Likert
'1' Q1 5
'2' Q1 5
'3' Q1 5
'4' Q1 4
'5' Q1 4
'6' Q1 2
'7' Q1 DontKnow
'8' Q1 NotApp
'1' Q2 3
'2' Q2 3
'3' Q2 3
'4' Q2 2
'5' Q2 DontKnow
'6' Q2 2
'7' Q2 DontKnow
'8' Q2 DontKnow
'1' Q3 1
'2' Q3 2
'3' Q3 3
'4' Q3 2
'5' Q3 3
'6' Q3 1
'7' Q3 2
'8' Q3 3
")

Data = read.table(textConnection(Input),header=TRUE)

XT = xtabs(~ Question + Likert,

data = Data)

XT

161
DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Likert
Question 1 2 3 4 5 DontKnow NotApp
Q1 0 1 0 2 3 1 1
Q2 0 2 3 0 0 3 0
Q3 2 3 3 0 0 0 0

### Counts for each question

Data$Likert.n = as.numeric(as.character(Data$Likert))

library(FSA)

Summarize(Likert.n ~ Question,
data = Data)

Question n nvalid mean sd min Q1 median Q3 max percZero

1 Q1 8 6 4.166667 1.1690452 2 4.00 4.5 5 5 0
2 Q2 8 5 2.600000 0.5477226 2 2.00 3.0 3 3 0
3 Q3 8 8 2.125000 0.8345230 1 1.75 2.0 3 3 0

### Numerical summary

Summary for whole data frame

XT = xtabs( ~ Likert,
data = Data)

XT

Likert
1 2 3 4 5 DontKnow NotApp
2 6 6 2 3 4 1

sum(XT)

[1] 24

### Sum of counts in table

Summarize( ~ Likert.n,
data = Data,
digits = 2)

n nvalid mean sd min Q1 median Q3 max

24.00 19.00 2.89 1.24 1.00 2.00 3.00 3.50 5.00

162
 DESCRIPTIVE STATISTICS FOR LIKERT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Exercises G

1. Considering Maggie Simpson's data,

• How many Likert responses of "4" were there?

• What percentage of responses was that?

• What was the median Likert response?

• What was the minimum response?

• How would you summarize the bar plot of Maggie's responses?

2. Considering Spongebob and Patrick's workshops,

• How many Likert responses of "5" did Spongebob and Patrick get, respectively?

• What percentage of total responses for each of them were these?

• How would you summarize the bar plot of their responses?

• What were the median scores for Patrick for each of Information, Presentation, and
Questions?

• What do the histograms for the two-way data (Spongebob and Patrick for each of
Information, Presentation, and Questions) suggest to you?

• Are your impressions borne out by the interaction plots?

5. Sandy Cheeks and Squidward each delivered presentations, and were each evaluated on each of the
two courses (A and B).
Speaker Course Likert
Sandy A 5
Sandy A 5
Sandy A 5
Sandy A 4
Sandy A 3
Sandy A 3
Sandy B 5
Sandy B 4
Sandy B 3
Sandy B 3
Sandy B 2
Sandy B 2
Squidward A 4

163
 DESCRIPTIVE STATISTICS WITH THE LIKERT PACKAGE SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Squidward A 4
Squidward A 3
Squidward A 3
Squidward A 2
Squidward A 2
Squidward B 3
Squidward B 3
Squidward B 2
Squidward B 2
Squidward B 1
Squidward B 1

a. Summarize the data as one-way or two-way data, using at least one plot, and at least one
numerical summary. Interpret your results, and describe your conclusions. Remember to
use statistics that are appropriate for Likert data.

Descriptive Statistics with the likert Package

The likert package can be used to produce attractive summaries and plots of one-sample or one-way
Likert data. The package is somewhat finicky with the form the data it accepts, however. Data must be
in “wide” format. Long-format and wide-format data are discussed in the section “Long-format and
wide-format data” in the Types of Variables chapter. In addition, sample sizes for each level of the
factor variable must be equal, but you can use NA values for missing observations.

Packages used in this chapter

The packages used in this chapter include:

• psych
• likert

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(likert)){install.packages("likert")}

The likert package

Input =("
Pooh Piglet Tigger
3 2 4
5 4 4
4 2 4
4 2 4
4 1 5
4 2 3
4 3 5

164
 DESCRIPTIVE STATISTICS WITH THE LIKERT PACKAGE SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

4 2 4
5 2 4
5 3 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Change Likert scores to factor and specify levels

Data$Pooh = factor(Data$Pooh,
levels = c("1", "2", "3", "4", "5"),
ordered = TRUE)

Data$Piglet = factor(Data$Piglet,
levels = c("1", "2", "3", "4", "5"),
ordered = TRUE)

Data$Tigger = factor(Data$Tigger,
levels = c("1", "2", "3", "4", "5"),
ordered = TRUE)

### Double check the data frame

library(psych)

headTail(Data)

str(Data)

'data.frame': 10 obs. of 3 variables:

$ Pooh : Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 3 5 4 4 4 4 4 4 5 5
$ Piglet: Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 2 4 2 2 1 2 3 2 2 3
$ Tigger: Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 4 4 4 4 5 3 5 4 4 3

summary(Data)

### Remove unnecessary objects

rm(Input)

Summary statistics and plots with the likert package

Percent responses in each group

library(likert)

likert(Data)

165
 DESCRIPTIVE STATISTICS WITH THE LIKERT PACKAGE SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Item 1 2 3 4 5
1 Pooh 0 0 10 60 30
2 Piglet 10 60 20 10 0
3 Tigger 0 0 20 60 20

### Note: if there are NA’s, this summary doesn’t tell you!

### Count responses with summary function in native stats package

summary(Data)

Pooh Piglet Tigger

1:0 1:1 1:0
2:0 2:6 2:0
3:1 3:2 3:2
4:6 4:1 4:6
5:3 5:0 5:2

### Note: if there are NA’s, this summary will tell you

library(likert)

Result = likert(Data)

summary(Result)

Item low neutral high mean sd

1 Pooh 0 10 90 4.2 0.6324555
3 Tigger 0 20 80 4.0 0.6666667
2 Piglet 70 20 10 2.3 0.8232726

### Note: responses are grouped into "low", "neutral", and "high"

### Note: if there are NA’s, this summary doesn’t tell you!

Plots that treat Likert data like factor data

Bar plot
library(likert)

Result = likert(Data)

plot(Result,
type="bar")

166
 DESCRIPTIVE STATISTICS WITH THE LIKERT PACKAGE SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Note: for the percent numbers,

### responses are grouped into "low", "neutral", and "high"

Heat plot
library(likert)

Result = likert(Data)

plot(Result,
type="heat",
low.color = "white",
high.color = "blue",
text.color = "black",
text.size = 4,
wrap = 50)

167
 DESCRIPTIVE STATISTICS WITH THE LIKERT PACKAGE SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plots that treat Likert data like numeric data

Density plot
library(likert)

Result = likert(Data)

plot(Result,
type="density",
facet = TRUE,
bw = 0.5)

168
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Note: Vertical lines are means for each group.

### Note: Curves are density plots, which show the distribution of values
### similar to a histogram.

References

Jason Bryer, J. and Speerschneider, K. likert: An R package analyzing and visualizing Likert items.
jason.bryer.org/likert/.

Jason Bryer, J. and Speerschneider, K. Package ‘likert’. cran.r-

project.org/web/packages/likert/likert.pdf.

Confidence Intervals for Medians

Confidence intervals are discussed in the Confidence Intervals chapter.

The methods for determining confidence intervals for medians are distinct from the traditional
method for means. There are different methods for calculating confidence intervals for the median,
and there are few different methods are presented in this chapter.

When data distributions are normal or uniform in distribution, and the number of observations is
large (say, n ≥ 100), any of the methods should work reasonably well. When distributions are heavily
skewed or the number of observations is relatively small (say, n < 20), results from some methods can
differ notably from others.

For routine use, I probably recommend against the basic, normal, exact, and wilcox methods. The BCa
(bias corrected, accelerated) is often cited as the best for theoretical reasons. The percentile method is
also cited as typically good. However, if you get the “extreme order statistics used as endpoints”
warning message, use a different test. For small data sets the interval from BCa may be wider than for
some other methods.

For a description of the bootstrap confidence interval methods, see Carpenter and Bithell (2000) in the
“References” section below.

There are also some variants in the calculation of the median, such as the “(pseudo)median” calculated
by the wilcox.test function.

Packages used in this chapter

The packages used in this chapter include:

• psych

169
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• FSA
• boot
• DescTools
• plyr
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(boot)){install.packages("boot")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(plyr)){install.packages("plyr")}
if(!require(rcompanion)){install.packages("rcompanion")}

Medians, quantiles, and confidence intervals for one-sample data

For one-sample data, the median can be calculated with the median function, the summary function,
and the Summarize function from the FSA package.

Confidence intervals for the median can be calculated with the wilcox.test function, or by a process
called bootstrapping with the boot function in the boot package, or with the MedianCI function in the
DescTools package.

The function groupwiseMedian in the rcompanion package produces medians, and allows for the
calculation “basic”, “normal”, “percentile”, and “bca” bootstrap confidence intervals for medians, as
well as the confidence intervals for medians from the wilcox.test function. It can also calculate these
statistics for grouped data (one-way or multi-way).

This example will use some theoretical data for Lisa Simpson, rated on a 10-point Likert item.
Input =("
Speaker Rater Likert
'Lisa Simpson' 1 8
'Lisa Simpson' 2 10
'Lisa Simpson' 3 9
'Lisa Simpson' 4 10
'Lisa Simpson' 5 6
'Lisa Simpson' 6 5
'Lisa Simpson' 7 3
'Lisa Simpson' 8 7
'Lisa Simpson' 9 8
'Lisa Simpson' 10 5
'Lisa Simpson' 11 10
'Lisa Simpson' 12 4
'Lisa Simpson' 13 8
'Lisa Simpson' 14 6
'Lisa Simpson' 15 9
'Lisa Simpson' 16 8
'Lisa Simpson' 17 7

170
CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Lisa Simpson' 18 5
'Lisa Simpson' 19 8
'Lisa Simpson' 20 7
'Lisa Simpson' 21 9
'Lisa Simpson' 22 8
'Lisa Simpson' 23 7
'Lisa Simpson' 24 5
'Lisa Simpson' 25 10
'Lisa Simpson' 26 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Produce median with median, summary, and Summarize functions

median(Data$Likert)

[1] 7.5

summary(Data)

Speaker Rater Likert

Lisa Simpson:26 Min. : 1.00 Min. : 3.000
1st Qu.: 7.25 1st Qu.: 6.000
Median :13.50 Median : 7.500
Mean :13.50 Mean : 7.269
3rd Qu.:19.75 3rd Qu.: 8.750
Max. :26.00 Max. :10.000

The Summarize function in the FSA package

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

171
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Speaker n nvalid mean sd min Q1 median Q3 max percZero

1 Lisa Simpson 26 26 7.269 1.951 3 6 7.5 8.75 10 0

Confidence interval for medians with the wilcox.test function

For one-sample data, the wilcox.test function will produce a confidence interval for the median. It will
also produce a “(pseudo)median”. For details on the calculations of these statistics, see ?wilcox.test.

Note that the conf.int=TRUE option must be used to produce the confidence interval, and the
conf.level=0.95 option indicates that a 95% confidence interval should be calculated.

wilcox.test(Data$Likert,
alternative="two.sided",
correct=TRUE,
conf.int=TRUE,
conf.level=0.95)

95 percent confidence interval:

6.499961 8.000009

Confidence interval for median by bootstrap

Bootstrapping is a method by which a statistic is calculated by repeated sampling the given data to
better estimate the distribution of values in the population from which the sample was taken.

The boot package in R can derive various statistics with a bootstrap process. Note that the grammar of
the function is somewhat complicated, but here Data$Likert is the variable we wish to get the statistics
for, and R=10000 indicates the number of bootstrap replicates to use. Mboot here is defined as the
result of the bootstrap. The boot.ci function produces four types of confidence intervals for the
median. Note also that the displaying the result Mboot gives a standard error for the estimated
median. See ?boot.ci for more details on these methods.

If the function takes too long to complete, you can decrease the R= value.

Note that results for any statistic derived from an iterative process like bootstrapping may be slightly
different if the process is re-run.

Optional technical note: Bootstrapped confidence intervals may not be reliable for discreet data, such
as the ordinal Likert data used in these examples, especially for small samples. My understanding is
that the determination of the confidence intervals assumes a continuous and bell-shaped distribution
of the statistic (the median, in this case) from the bootstrapped samples. That is, hist(Mboot$t[,1], col =
"darkgray") below should produce a continuous and bell-shaped distribution. This concern is not
considered in the examples in this book.
library(boot)

Mboot = boot(Data$Likert,
function(x,i) median(x[i]),
R=10000)

172
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

boot.ci(Mboot,
conf = 0.95,
type = c("norm", "basic" ,"perc", "bca")
)

Intervals :
Level Normal Basic
95% ( 6.510, 8.535 ) ( 7.000, 8.500 )

Level Percentile BCa

95% ( 6.5, 8.0 ) ( 6.0, 8.0 )

### Other information

Mboot

hist(Mboot$t[,1],
col = "darkgray")

Median and confidence interval with the DescTools package

With the MedianCI function in the DescTools package, the method=exact option uses the confidence
interval from the SignTest function in DescTools. The method=boot option uses the basic method from
the boot package.
library(DescTools)

MedianCI(Data$Likert,
conf.level = 0.95,
na.rm = FALSE,
method = "exact",
R = 10000)

median lwr.ci upr.ci

7.5 6.0 8.0

groupwiseMedian function to produce medians and confidence intervals

The function groupwiseMedian in the rcompanion package produces medians, and calculates the
“basic”, “normal”, “percentile”, and “bca” bootstrap confidence intervals for medians, as well as the
confidence intervals for medians from the wilcox.test function.

It can calculate these statistics for grouped data (one-way or multi-way), with the factors for two-way
data indicated with, for example, group=c("Factor.A", "Factor.B"). An example of using this function
with two-way data is in the “Interaction plot using medians and confidence intervals” section in the
Descriptive Statistics for Likert Data chapter.

The basic, normal, percentile, bca, wilcox, and exact options determine which types of confidence
intervals will be calculated.

173
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that results for any statistic derived from an iterative process like bootstrapping may be slightly
different.

Also note that Speaker is being treated a grouping variable, even though it has only one level.

See library(rcompanion); ?groupwiseMedian for more information on this function.

library(rcompanion)

groupwiseMedian(data=Data,
group="Speaker",
var="Likert",
conf=0.95,
R=5000,
percentile=TRUE,
bca=TRUE,
digits=3)

Speaker n Median Conf.level Percentile.lower Percentile.upper Bca.lower Bca.upper

1 Lisa Simpson 26 7.5 0.95 6.5 8 6 8

Medians, quantiles, and confidence intervals for grouped data

The following example revisits the Pooh, Piglet and Tigger data from the Descriptive Statistics with the
likert Package chapter.

The Summarize function in the FSA package will produce medians and quantiles for grouped data
including one-way and multi-way data.

The function groupwiseMedian will produce medians and confidence intervals for grouped data,
including one-way and multi-way data.
Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3

174
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Piglet 2
Piglet 2
Piglet 3
Tigger 4
Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

Summarize function in FSA package for grouped data

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n nvalid mean sd min Q1 median Q3 max percZero

1 Piglet 10 10 2.3 0.823 1 2 2 2.75 4 0
2 Pooh 10 10 4.2 0.632 3 4 4 4.75 5 0
3 Tigger 10 10 4.0 0.667 3 4 4 4.00 5 0

groupwiseMedian function to produce medians and confidence intervals for grouped data
The groupwiseMedian function produces medians, and calculates confidence intervals for medians
using the boot.ci function in the boot package and the wilcox.test function.

It can calculate these statistics for grouped data, with the factors for two-way data indicated with, for
example, group=c("Factor.A", "Factor.B"). An example of using this function with two-way data is in
the “Interaction plot using medians and confidence intervals” section in the Descriptive Statistics for
Likert Data chapter.

The basic, normal, percentile, bca, wilcox, and exact options determine which types of confidence
intervals will be calculated. For details on the calculations of these statistics, see ?wilcox.test and
?boot.ci.

Note that results for any statistic derived from an iterative process like bootstrapping may be slightly
different. Conducting bootstraps across several groups may take a long time if the R option is set to a
high value; however setting R to too low a value may produce errors. R=1000 should be fine for most
applications.

See library(rcompanion); ?groupwiseMedian for more information on this function.

175
 CONFIDENCE INTERVALS FOR MEDIANS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

groupwiseMedian(data=Data,
group="Speaker",
var="Likert",
conf=0.95,
R=5000,
percentile=TRUE,
bca=FALSE,
digits=3)

Speaker n Median Conf.level Percentile.lower Percentile.upper

1 Piglet 10 2 0.95 2.0 3.0
2 Pooh 10 4 0.95 4.0 5.0
3 Tigger 10 4 0.95 3.5 4.5

References

Carpenter, J. and J. Bithel. 2000. “Bootstrap confidence intervals: when, which, what? A practical guide
for medical statisticians”. Statistics in Medicine 19:1141–1164.
www.tau.ac.il/~saharon/Boot/10.1.1.133.8405.pdf.

Exercises H

1. Considering Lisa Simpson's data,

• What was her median Likert score?

• Minimum? Maximum? 25th percentile? 75th percentile?

2. For Lisa Simpson's data, what is the 95% confidence interval for the median using the following
methods?

• wilcox.test
• normal
• basic
• percentile
• BCa

3. Considering the Pooh, Piglet, and Tigger data,

• For Piglet, what was his median score and 95% confidence interval using the percentile
method?

5. Bart Simpson and Milhouse Van Houten were each evaluated on a Likert scale. Determine the
median score for each, and determine the 95% confidence interval for the median score.

176
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Be sure to state which method for the confidence interval you used.

• How can you interpret the results?

Speaker Likert
Bart 7
Bart 7
Bart 8
Bart 9
Bart 5
Bart 6
Bart 6
Bart 7
Bart 8
Bart 7
Bart 7
Bart 7
Milhouse 8
Milhouse 8
Milhouse 7
Milhouse 7
Milhouse 9
Milhouse 9
Milhouse 10
Milhouse 6
Milhouse 7
Milhouse 8
Milhouse 7
Milhouse 8

Converting Numeric Data to Categories

There are occasions when it is useful to categorize Likert scores, Likert scales, or continuous data into
groups or categories. In general, there are no universal rules for converting numeric data to
categories. A few methods are presented here.

Categorizing data by a range of values

One approach is to create categories according to logical cut-off values in the scores or measured
values. An example of this is the common grading system in the U.S. in which a 90% grade or better is
an “A”, 80–89% is “B”, etc. It is common in this approach to make the categories with equal spread in
values. For example, there is a 10 point spread in a “B” grade and a 10 point spread in a “C” grade. But
this equality is not required. Below in the “Categorize data by range of values” example, 5-point Likert
data are converted into categories with 4 and 5 being “High”, 3 being “Medium”, and 1 and 2 being
“Low”.

This approach relies on the chosen cut-off points being meaningful. For example, for a grade of 70–79
to be considered “sufficient”, the evaluation instruments (e.g. the tests, quizzes, and assignments) need

177
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

to be calibrated so that a grade of 75 really is “sufficient”, and not “excellent” or “good”, etc. You can
imagine a case where a 4 or 5 on a 5-point Likert item is considered “high”, but if all respondents
scored 4 or 5 on the item, it might not be clear that these values are “high”, but may just be the typical
response. Likewise, in this case, the decision to group 4 and 5 as “high” needs be compared with a
decision to group 4 with 2 and 3 as “medium”. This breakdown may be closer to how people interpret
a 5-point Likert scale. Either grouping could be meaningful depending on the purpose of the
categorization and the interpretation of each level in the Likert item.

Categorizing data by percentiles

A second approach is to use percentiles to categorize data. Remember, the value for the 90th percentile
is the score or measurement below which 90% of respondents scored.

The advantage to this approach is that it does not rely on the scoring system being meaningful in its
absolute values. That is, students scoring above the 90th percentile are scoring higher than 90% of
students. It doesn’t matter if the score for the 90th percentile is 90 out of 100 or 50 out of 100. If the
groups use equally-spaced breakpoints, for example 0th, 25th, 50th, 75th, and 100th percentiles, there
should be approximately an equal number of respondents in each category.

Categorizing data with clustering

A third approach is to use a clustering algorithm to divide data into groups with similar
measurements. This is useful when there are multiple measurements for an individual. For example,
if students receive scores on reading, math, and analytical thinking, an algorithm can determine if, for
example, there is a group of students who do well on all three measurements, and a group of students
who do well in math and analysis but who do poorly on reading. Most clustering algorithms assume
that measurement data is continuous in nature. Below, the partitioning around medoids (PAM)
method is used with the manhattan metric. This may be relatively more suitable for ordinal data than
some other methods

Packages used in this chapter

The packages used in this chapter include:

• psych
• cluster
• fpc

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(cluster)){install.packages("cluster")}
if(!require(fpc)){install.packages("fpc")}

Examples for converting numeric data to categories

Input =("
Instructor Student Likert
Homer a 3
Homer b 4

178
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Homer c 4
Homer d 4
Homer e 4
Homer f 5
Homer g 5
Homer h 5
Homer i 3
Homer j 2
Homer k 3
Homer l 4
Homer m 5
Homer n 5
Homer o 5
Homer p 4
Homer q 4
Homer r 3
Homer s 2
Homer t 5
Homer u 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Categorize data by range of values

The following example will categorize responses on a single 5-point Likert item. The algorithm will be
a score of 1 or 2 will be called “low”, a score of 3 “medium”, and a score of 4 or 5 “high”.

Categorize data
Data$Category[Data$Likert == 1 | Data$Likert == 2] = "Low"
Data$Category[Data$Likert == 3 ] = "Medium"
Data$Category[Data$Likert == 4 | Data$Likert == 5] = "High"

Data

179
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Order factor levels to make output easier to read

Data$Category = factor(Data$Category,
levels=c("Low", "Medium", "High"))

Summarize counts of categories

XT = xtabs(~ Category + Instructor,
data = Data)

XT

Instructor
Category Homer
Low 2
Medium 5
High 14

Report students in each category

Data$Student[Data$Category == "Low"]

[1] j s

Data$Student[Data$Category == "Medium"]

[1] a i k r u

Data$Student[Data$Category == "High"]

[1] b c d e f g h l m n o p q t

Alternate method with tapply

tapply(X = Data$Student,
INDEX = Data$Category,
FUN = print)

$Low
[1] j s

$Medium
[1] a i k r u

$High
[1] b c d e f g h l m n o p q t

180
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summary table
Category Range Count Students
Low 1 or 2 2 j, s
Medium 3 5 a, i, k, r, u
High 4 or 5 14 b, c, d, e, f, g, h, l, m, n, o, p, q, t

Categorize data by percentile

The following example will categorize responses on a single 5-point Likert item. Respondents scoring
below the 33rd percentile will be labeled “Lower third”; those between the 33rd and 67th percentile
“Middle third”; and those above the 67th percentile “Upper third”.

Categorize data
Percentile_00 = min(Data$Likert)
Percentile_33 = quantile(Data$Likert, 0.33333)
Percentile_67 = quantile(Data$Likert, 0.66667)
Percentile_100 = max(Data$Likert)

RB = rbind(Percentile_00, Percentile_33, Percentile_67, Percentile_100)

dimnames(RB)[[2]] = "Value"

RB

Value
Percentile_00 2.0000
Percentile_33 3.6666
Percentile_67 4.3334
Percentile_100 5.0000

Data$Group[Data$Likert >= Percentile_00 & Data$Likert < Percentile_33] = "Lower_third"

Data$Group[Data$Likert >= Percentile_33 & Data$Likert < Percentile_67] = "Middle_third"
Data$Group[Data$Likert >= Percentile_67 & Data$Likert <= Percentile_100] = "Upper_third"

Data

Order factor levels to make output easier to read

Data$Group = factor(Data$Group,
levels=c("Lower_third", "Middle_third", "Upper_third"))

Summarize counts of groups

XT = xtabs(~ Group + Instructor,
data = Data)

XT

181
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instructor
Group Homer
Lower_third 7
Middle_third 7
Upper_third 7

Report students in each group

tapply(X = Data$Student,
INDEX = Data$Group,
FUN = print)

$Lower_third
[1] a i j k r s u

$Middle_third
[1] b c d e l p q

$Upper_third
[1] f g h m n o t

Summary table
Group Range Count Students
Lower third 1, 2, or 3 7 a, i, j, k, r, s, u
Middle third 4 7 b, c, d, e, l, p, q
Upper third 5 7 f, g, h, m, n, o, t

Categorize data by clustering

In the following example, each student has a score for Happy and for Tired. The students will be
divided into clusters based on the similarities of their scores across both measures.
Input =("
Instructor Student Happy Tired
Marge a 5 5
Marge b 5 5
Marge c 2 5
Marge d 2 5
Marge e 5 5
Marge f 5 5
Marge g 1 5
Marge h 1 5
Marge i 1 5
Marge j 5 5
Marge k 3 3
Marge l 3 3
Marge m 3 3
Marge n 5 2
Marge o 5 2
Marge p 5 1

182
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Marge q 5 1
Marge r 5 1
Marge s 4 1
Marge t 4 1
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Plot data
In the following plot, each letter represents a Student from the data frame. To me, this plot suggests
that the data could be reasonably clustered into 4 or perhaps 7 or 8 clusters.
plot(jitter(Happy) ~ jitter(Tired),
data = Data,
pch=as.character(Data$Student))

Use only numeric data

Data.num = Data[c("Happy", "Tired")]

183
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Determine the optimal number of clusters

The pamk function in the fpc package can determine the optimum number of clusters for the
partitioning around medoids (PAM) method. It will also complete the PAM analysis, but we’ll do that
separately below.

Practical considerations may override the results of the pamk function. In the example below, if we
include 1 in the possible range of cluster numbers, the function will determine that 1 is the optimum
number. Also, if we extend the range to, say, 10, the function will choose 7 as the optimum number,
but this may be too many for our purposes.
library(fpc)

PAMK = pamk(Data.num,
krange = 2:5,
metric="manhattan")

PAMK$nc

[1] 4

### This is the optimum number of clusters in the range

plot(PAMK$crit)
lines(PAMK$crit)

For the range of cluster numbers shown on the x-axis, the crit value is maximized
at 4, suggesting 4 is optimum number of clusters for this range.

Categorize data
We will use the pam function in the cluster package to divide our data into four clusters.
library(cluster)

PAM = pam(x = Data.num,

184
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

k = 4, ### Number of clusters to find

metric="manhattan")

PAM

Medoids:
ID Happy Tired
[1,] 10 5 5
[2,] 9 1 5
[3,] 13 3 3
[4,] 16 5 1

Clustering vector:
[1] 1 1 2 2 1 1 2 2 2 1 3 3 3 4 4 4 4 4 4 4

### Add clusters to data frame

PAMClust = rep("NA", length(Data$Likert))

PAMClust[PAM$clustering == 1] = "Cluster 1"

PAMClust[PAM$clustering == 2] = "Cluster 2"
PAMClust[PAM$clustering == 3] = "Cluster 3"
PAMClust[PAM$clustering == 4] = "Cluster 4"

Data$Cluster = PAMClust

Data

Order factor levels to make output easier to read

Data$Cluster = factor(Data$Cluster,
levels=c("Cluster 1", "Cluster 2",
"Cluster 3", "Cluster 4"))

Summarize counts of groups

XT = xtabs(~ Cluster + Instructor,
data = Data)

XT

Instructor
Cluster Marge
Cluster 1 5
Cluster 2 5
Cluster 3 3
Cluster 4 7

185
 CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Report students in each group

tapply(X = Data$Student,
INDEX = Data$Cluster,
FUN = print)

$`Cluster 1`
[1] a b e f j

$`Cluster 2`
[1] c d g h i

$`Cluster 3`
[1] k l m

$`Cluster 4`
[1] n o p q r s t

Summary table
Cluster Interpretation Count Students
Cluster 1 Tired and happy 5 a, b, e, f, j
Cluster 2 Tired and not happy 5 c, d, d, h, i
Cluster 3 Middle of the road 3 k, l, m
Cluster 4 Not tired and happy 7 n, o, p, q, r, s, t

Final plot
ggplot(Data,
aes(x = Tired,
y = Happy,
color = Cluster)) +
geom_point(size=3) +
geom_jitter(width = 0.4, height = 0.4) +
theme_bw()

186
CONVERTING NUMERIC DATA TO CATEGORIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

187
 INTRODUCTION TO TRADITIONAL NONPARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Traditional Nonparametric Tests

Introduction to Traditional Nonparametric

Tests
The traditional nonparametric tests presented in this book are primarily rank-based tests that use the
ranks of data instead of their numeric values. This makes them appropriate for data sets where the
dependent variable is interval/ratio or ordinal. Some authors urge caution when using these tests
with data where there are likely to be many ties, including Likert data.

Nonparametric test assumptions

Nonparametric tests do not assume that the underlying data have any specific distribution. However,
it is important to understand the assumptions of each specific test before using it.

Advantages of nonparametric tests

• The tests presented in this section are relatively common, and your audience is relatively
likely to be familiar with them.

• They are appropriate for interval/ratio or ordinal dependent variables.

• Their nonparametric nature makes them appropriate for interval/ratio data that don’t meet
the assumptions of parametric analyses. These include data that are skewed, non-normal,
contain outliers, or possibly are censored. Censored data is data where there is an upper or
lower limit to values. For example, if ages under 5 are reported as “under 5”.

Interpretation of nonparametric tests

In general, these tests determine if there is a systematic difference between two groups. This may be
due to a difference in location (e.g. median) or in the shape or spread of the distribution of the data. It
is therefore appropriate to report significant results as, e.g., “There is a significant difference between
Likert scores from the pre-test and the post-test." Or, "The significant Mann–Whitney test indicates
that Likert scores from the two classes come from different populations."

For the Mann-Whitney, Kruskal-Wallis, and Friedman tests, if the distributions of the groups have the
same shape and spread, then it can be assumed that the difference between groups is a difference in
medians. Otherwise, the difference is a difference in distributions.

In reality, you should look at the distributions of each group in these tests, with histograms or box
plots, so that your conclusions can accurately reflect the data. You don't want to imply that differences
between two treatments are differences in medians when they are really differences in the shape or
spread of the distributions. On the other hand, if it really does look like the difference is a difference in
location, you want to be clear about this.

188
 INTRODUCTION TO TRADITIONAL NONPARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

As a point of interest, Mangiafico (2015) and McDonald (2014) in the “References” section provide an
example of a significant Kruskal–Wallis test where the groups have identical medians.

Using traditional nonparametric tests with ordinal data

Some authors caution against using traditional nonparametric tests with ordinal dependent variables,
since many of them were developed for use with continuous (interval/ratio) data. Other authors
argue that, since these tests rank-transform data before analysis and have adjustments for tied ranks,
that they are appropriate for ordinal data. Some authors have further concerns about situations
where are likely to be many ties in ranks, such as Likert data.

Using permutation tests and ordinal regression for ordinal data

The following sections of this book, Ordinal Tests with Cumulative Link Models and Permutation Tests
describe different approach to handling ordinal data that may be a better approach than using the
traditional nonparametric tests described in this section, at least in some cases.

Using traditional nonparametric tests with interval/ratio data

These nonparametric tests are commonly used for interval/ratio data when the data fail to meet the
assumptions of parametric analysis.

Some authors discourage using common nonparametric tests for interval/ratio data.

• One issue is the interpretation of the results mentioned above. That is, often results are
incorrectly interpreted as a difference in medians when they are really describing a
difference in distributions.

• Another problem is the lack of flexibility in designs these test can handle. For example, there
is no common equivalent for a parametric two-way analysis of variance.

• Finally, these tests may lack power relative to their parametric equivalents.

Given these considerations and the fact that that parametric statistics are often relatively robust to
minor deviations in their assumptions, some authors argue that it is often better to stick with
parametric analyses for interval/ratio data if it’s possible to make them work.

References

“Kruskal–Wallis Test” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/d_06.html.

“Kruskal–Wallis Test” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/kruskalwallis.html.

189
 ONE-SAMPLE WILCOXON SIGNED-RANK TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One-sample Wilcoxon Signed-rank Test

One-sample tests are not used too often, but are useful to compare a set of values to a given default
value. For example, one might ask if a set of five-point Likert scores are significantly different from a
“default” or “neutral” score of 3. Another use might be to compare a current set of values to a
previously published value.

Appropriate data
• One-sample data
• Data are ordinal, interval, or ratio
• Data are relatively symmetrical about their median

Hypotheses
• Null hypothesis: The distribution of the data set is symmetric about the default value.
• Alternative hypothesis (two-sided): The distribution of the data set is not symmetric about
the default value.

Interpretation
Reporting significant results as “Likert scores were significantly different from a neutral value of 3” is
acceptable.

Notes on name of test

The names used for the one-sample Wilcoxon signed-rank test and similar tests can be confusing.
Occasionally the name “signed-rank test” will be used. “Sign test” may also be used, although properly
the sign test is a different test. Both “signed-rank test” and “sign test” are sometimes used to refer to
either one-sample or two-sample tests.

The best advice is to use a name specific to the test being used.

Other notes and alternative tests

Caution should be used when using this test if data contain outliers or unsymmetrical data.

If data are not symmetrical, the sign test can be used as an alternative. The sign test is described in the
next chapter.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• BSDA

The following commands will install these packages if they are not already installed:

190
 ONE-SAMPLE WILCOXON SIGNED-RANK TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(BSDA)){install.packages("BSDA")}

One-sample Wilcoxon signed-rank test example

This example will re-visit the Maggie Simpson data from the Descriptive Statistics for Likert Data
chapter.

The example answers the question, “Are Maggie’s scores significantly different from a ‘neutral’ score of
3?”

The test will be conducted with the wilcox.test function, which produces a p-value for the hypothesis,
as well a pseudo-median and confidence interval.

Note that the bar plot shows that the data are relatively symmetrical in distribution, suggesting that
the one-sample Wilcoxon test will be appropriate.

Input =("
Speaker Rater Likert
'Maggie Simpson' 1 3
'Maggie Simpson' 2 4
'Maggie Simpson' 3 5
'Maggie Simpson' 4 4
'Maggie Simpson' 5 4
'Maggie Simpson' 6 4
'Maggie Simpson' 7 4
'Maggie Simpson' 8 3
'Maggie Simpson' 9 2
'Maggie Simpson' 10 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

191
 ONE-SAMPLE WILCOXON SIGNED-RANK TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

rm(Input)

Summarize data treating Likert scores as factors

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
cross tabulated over values of Speaker. The prop.table function translates a table into proportions.
The margin=1 option indicates that the proportions are calculated for each row.

xtabs( ~ Speaker + Likert.f,

data = Data)

Likert.f
Speaker 2 3 4 5
Maggie Simpson 1 2 5 2

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 2 3 4 5
Maggie Simpson 0.1 0.2 0.5 0.2

Bar plot
XT = xtabs(~ Likert.f,
data=Data)

barplot(XT,
col="dark gray",
xlab="Maggie's Likert",
ylab="Frequency")

192
 ONE-SAMPLE WILCOXON SIGNED-RANK TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n mean sd min Q1 median Q3 max percZero

1 Maggie Simpson 10 3.8 0.919 2 3.25 4 4 5 0

One-sample Wilcoxon signed-rank test

In the wilcox.test function, the mu option indicates the value of the default value to compare to. In this
example Data$Likert is the one-sample set of values on which to conduct the test. For the meaning of
other options, see ?wilcox.test.
wilcox.test(Data$Likert,
mu=3,
conf.int=TRUE,
conf.level=0.95)

Wilcoxon signed rank test with continuity correction

V = 32.5, p-value = 0.04007

alternative hypothesis: true location is not equal to 3

### Note p-value in the output above

### You will get the "cannot compute exact p-value with ties" error
### You can ignore this, or use the exact=FALSE option.

95 percent confidence interval:

3.000044 4.500083

sample estimates:

(pseudo)median
4.000032

### Note that the output will also produce a pseudo-median value
### and a confidence interval if the conf.int=TRUE option is used.

Exercises I

1. Considering Maggie Simpson’s data,

What was her median score?

What were the first and third quartiles for her scores?

193
 ONE-SAMPLE WILCOXON SIGNED-RANK TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Are the data reasonably symmetric about their median, or should another test be used?

According to the one-sample Wilcoxon signed-rank test, is her median score significantly
different from a neutral score of 3?

Is the confidence interval output from the test useful in answering the previous question?

Overall, how would you summarize her results?

Do these results reflect what you would expect from looking at the bar plot?

2. Brian Griffin wants to assess the education level of students in his course on creative writing for
adults. He wants to know the median education level of his class, and if the education level of his class
is different from the typical Bachelor’s level.

Brian used the following table to code his data.

Code Abbreviation Level

1 < HS Less than high school

2 HS High school
3 BA Bachelor’s
4 MA Master’s
5 PhD Doctorate

The following are his course data.

Instructor Student Education

'Brian Griffin' a 3
'Brian Griffin' b 2
'Brian Griffin' c 3
'Brian Griffin' d 3
'Brian Griffin' e 3
'Brian Griffin' f 3
'Brian Griffin' g 4
'Brian Griffin' h 5
'Brian Griffin' i 3
'Brian Griffin' j 4
'Brian Griffin' k 3
'Brian Griffin' l 2

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the median education level? (Be sure to report the education level, not just the
numeric code!)

194
 SIGN TEST FOR ONE-SAMPLE DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Are the data reasonably symmetric about their median, or should another test be used?

According to the one-sample Wilcoxon signed-rank test, is the median education level
significantly different from a typical level of Bachelor’s?

Is the confidence interval output from the test useful in answering the previous question?

Plot Brian’s data in a way that helps you visualize the data. Do the results reflect what you
would expect from looking at the plot?

Sign Test for One-sample Data

The sign test is similar in purpose to the one-sample Wilcoxon signed-rank test. It is often less
powerful than the signed-rank test, but has the advantage of not requiring the data to be symmetric in
distribution. The test compares the number of observations greater or less than the default value
without accounting for magnitude of the difference between each observation and the default value.

The test is conducted with the SIGN.test function in the BSDA package or the SignTest function in the
DescTools package. These functions produce a p-value for the hypothesis, as well as the median and
confidence interval of the median for the data.

Appropriate data
• One-sample data
• Data are ordinal, interval, or ratio

Hypotheses
• Null hypothesis: The median of the data set is equal to the default value.
• Alternative hypothesis (two-sided): The median of the data set is not equal to the default
value.

Interpretation
Reporting significant results as “Likert scores were significantly different from a default value of 3” is
acceptable. As is “Median Likert scores were significantly different from a default value of 3”

Packages used in this chapter

The packages used in this chapter include:

• BSDA
• DescTools

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("BSDA")}

195
 SIGN TEST FOR ONE-SAMPLE DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(BSDA)){install.packages("DescTools")}

One-sample sign test example

For appropriate plots and data frame checking, see the One-sample Wilcoxon Signed-rank Test chapter.
Input =("
Speaker Rater Likert
'Maggie Simpson' 1 3
'Maggie Simpson' 2 4
'Maggie Simpson' 3 5
'Maggie Simpson' 4 4
'Maggie Simpson' 5 4
'Maggie Simpson' 6 4
'Maggie Simpson' 7 4
'Maggie Simpson' 8 3
'Maggie Simpson' 9 2
'Maggie Simpson' 10 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Sign test with the BSDA package

Note that Data$Likert is the one-sample data, and md=3 indicates the default value to compare to.
library(BSDA)

SIGN.test(Data$Likert,
md = 3)

One-sample Sign-Test

s = 7, p-value = 0.07031

alternative hypothesis: true median is not equal to 3

196
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Note the p-value in the output above

95 percent confidence interval:

3.000000 4.675556

sample estimates:
median of x
4

### Median value and confidence interval

Sign test with the DescTools package

Note that Data$Likert is the one-sample data, and mu=3 indicates the default value to compare to.
library(DescTools)

SignTest(Data$Likert,
mu = 3)

One-sample Sign-Test

S = 7, number of differences = 8, p-value = 0.07031

### Note the p-value in the output above

alternative hypothesis: true median is not equal to 3

97.9 percent confidence interval:

3 5

sample estimates:
median of the differences
4

### Median value and confidence interval

Two-sample Mann–Whitney U Test

When to use this test

The two-sample Mann–Whitney U test compares values for two groups. A significant result suggests
that the values for the two groups are different. It is equivalent to a two-sample Wilcoxon rank-sum
test.

In the context of this book, the test is useful to compare the scores or ratings from two speakers, two
different presentations, or two groups of audiences.

197
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

If the shape and spread of the distributions of values of each group is similar, then the test compares
the medians of the two groups. Otherwise, the test is really testing if there is a systematic difference in
the values of the two groups.

The test assumes that the observations are independent. That is, it is not appropriate for paired
observations or repeated measures data.

The test is performed with the wilcox.test function.

If the distributions of values of each group are similar in shape, but have outliers, then Mood’s median
test is an appropriate alternative.

Appropriate data
• Two-sample data. That is, one-way data with two groups only
• Dependent variable is ordinal, interval, or ratio
• Independent variable is a factor with two levels. That is, two groups
• Observations between groups are independent. That is, not paired or repeated measures
data
• In order to be a test of medians, the distributions of values for each group need to be of
similar shape and spread; outliers affect the spread. Otherwise the test is a test of
distributions.

Hypotheses
If the distributions of the two groups are similar in shape and spread:
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

If the distributions of the two groups are not similar in shape and spread:
• Null hypothesis: The distribution of values for each group are equal.
• Alternative hypothesis (two-sided): There is systematic difference in the distribution of
values for the groups.

Interpretation
If the distributions of the two groups are similar in shape:
Significant results can be reported as “The median value of group A was significantly different
from that of group B.”

If the distributions of the two groups are not similar in shape:

Significant results can be reported as “Values for group A were significantly different from
those for group B.”

Other notes and alternative tests

The Mann–Whitney U test can be considered equivalent to the Kruskal–Wallis test with only two
groups.

Mood’s median test compares the medians of two groups. It is described in the next chapter.
198
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Another alternative is to use cumulative link models for ordinal data, which are described later in this
book.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}

Two-sample Mann–Whitney U test example

This example re-visits the Pooh and Piglet data from the Descriptive Statistics with the likert Package
chapter.

It answers the question, “Are Pooh's scores significantly different from those of Piglet?”

The Mann–Whitney U test is conducted with the wilcox.test function, which produces a p-value for the
hypothesis. First the data are summarized and examined using bar plots for each group.

Because the bar plots show that the distributions of scores for Pooh and Piglet are relatively similar in
shape, the Mann–Whitney U test can be interpreted as a test of medians.

Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2

199
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Piglet 2
Piglet 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Create a new variable which is the Likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
cross tabulated over values of Speaker. The prop.table function translates a table into proportions.
The margin=1 option indicates that the proportions are calculated for each row.

xtabs( ~ Speaker + Likert.f,

data = Data)

Likert.f
Speaker 1 2 3 4 5
Piglet 1 6 2 1 0
Pooh 0 0 1 6 3

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5
Piglet 0.1 0.6 0.2 0.1 0.0
Pooh 0.0 0.0 0.1 0.6 0.3

200
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plots of data by group

library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,2) # columns and rows of individual plots
)

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n mean sd min Q1 median Q3 max percZero

1 Piglet 10 2.3 0.823 1 2 2 2.75 4 0
2 Pooh 10 4.2 0.632 3 4 4 4.75 5 0

Two-sample Mann–Whitney U test example

This example uses the formula notation indicating that Likert is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?wilcox.test.

wilcox.test(Likert ~ Speaker,
data=Data)

201
 TWO-SAMPLE MANN–WHITNEY U TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Wilcoxon rank sum test with continuity correction

W = 5, p-value = 0.0004713
alternative hypothesis: true location shift is not equal to 0

### You may get a "cannot compute exact p-value with ties" error.
### You can ignore this or use the exact=FALSE option.

Exercises J

1. Considering Pooh and Piglet’s data,

What was the median score for each instructor?

What were the first and third quartiles for each instructor’s scores?

Are the data for both instructors reasonably symmetric about their medians?

Based on your previous answer, what is the null hypothesis for the Mann–Whitney test?

According to the Mann–Whitney test, is there a difference in scores between the instructors?

How would you summarize the results of the descriptive statistics and tests?

2. Brian and Stewie Griffin want to assess the education level of students in their courses on creative
writing for adults. They want to know the median education level for each class, and if the education
level of the classes were different between instructors.

They used the following table to code his data.

Code Abbreviation Level

1 < HS Less than high school

2 HS High school
3 BA Bachelor’s
4 MA Master’s
5 PhD Doctorate

The following are the course data.

Instructor Student Education
'Brian Griffin' a 3
'Brian Griffin' b 2
'Brian Griffin' c 3
'Brian Griffin' d 3
'Brian Griffin' e 3
'Brian Griffin' f 3
'Brian Griffin' g 4
'Brian Griffin' h 5
'Brian Griffin' i 3

202
 MOOD’S MEDIAN TEST FOR TWO-SAMPLE DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Brian Griffin' j 4
'Brian Griffin' k 3
'Brian Griffin' l 2
'Stewie Griffin' m 4
'Stewie Griffin' n 5
'Stewie Griffin' o 4
'Stewie Griffin' p 4
'Stewie Griffin' q 4
'Stewie Griffin' r 4
'Stewie Griffin' s 3
'Stewie Griffin' t 5
'Stewie Griffin' u 4
'Stewie Griffin' v 4
'Stewie Griffin' w 3
'Stewie Griffin' x 2

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the median score for each instructor? (Be sure to report the education level, not just
the numeric code!)

What were the first and third quartiles for each instructor’s scores?

Are the data for both instructors reasonably symmetric about their medians?

Based on your previous answer, what is the null hypothesis for the Mann–Whitney test?

According to the Mann–Whitney test, is there a difference in scores between the instructors?

Plot Brian and Stewie’s data in a way that helps you visualize the data. Do the results reflect
what you would expect from looking at the plot? How would you summarize the results of
the descriptive statistics and tests?

Mood’s Median Test for Two-sample Data

Mood’s median test compares the medians of two or more groups. It is often less powerful than the
Mann–Whitney U test, but can be used when the spread of values in the groups are different due to
outliers.

The test can be conducted with the mood.medtest function in the RVAideMemoire package.

Appropriate data
• One-way data with two or more groups
• Dependent variable is ordinal, interval, or ratio

203
 MOOD’S MEDIAN TEST FOR TWO-SAMPLE DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Independent variable is a factor with levels indicating groups

• Observations between groups are independent. That is, not paired or repeated measures
data
• Distributions of values for each group are similar in shape; however, the test is not sensitive
to outliers

Hypotheses
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

Interpretation
Significant results can be reported as “The median value of group A was significantly different
from group B.”

Packages used in this chapter

The packages used in this chapter include:

• RVAideMemoire

The following commands will install these packages if they are not already installed:
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}

Example using the RVAideMemoire package

This example uses the formula notation indicating that Likert is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?mood.medtest.

For appropriate plots and data frame checking, see the Two-sample Mann–Whitney U Test chapter.
Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2

204
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Piglet 3
Piglet 2
Piglet 2
Piglet 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Mood’s Median Test

library(RVAideMemoire)

mood.medtest(Likert ~ Speaker,
data = Data)

Mood's median test

p-value = 0.001093

Two-sample Paired Rank-sum Test

When to use this test

The two-sample rank-sum test for paired data is used to compare values for two groups where each
observation in one group is paired with one observation in the other group. The distribution of
differences in the paired samples should be symmetric in shape.

The test is useful to compare scores on a pre-test vs. scores on a post-test, or scores or ratings from
two speakers, two different presentations, or two groups of audiences when there is a reason to pair
observations, such as being done by the same rater.

205
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

A discussion of paired data can be found in the Independent and Paired Values chapter of this book.

The test is performed with the wilcox.test function with the paired=TRUE option.

Appropriate data
• Two-sample paired data. That is, one-way data with two groups only, where the
observations are paired between groups.
• Dependent variable is ordinal, interval, or ratio
• Independent variable is a factor with two levels. That is, two groups
• The distribution of differences in paired samples is symmetric

Hypotheses
• Null hypothesis: The distribution of the differences in paired values is symmetric around
zero.
• Alternative hypothesis (two-sided): The distribution of the differences in paired values is not
symmetric around zero.

Interpretation
Significant results can be reported as “Values for group A were significantly different from
those of group B.”

Other notes and alternative tests

If the distribution of differences between paired samples is not symmetrical, the two-sample sign test
for paired data can be used. This test is described in the next chapter.

Another alternative is to use cumulative link models for ordinal data, which are described later in this
book.

Packages used in this chapter

The packages used in this chapter include:

• psych
• BSDA

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(BSDA)){install.packages("BSDA")}

Two-sample paired rank-sum test example

For this example, imagine we want to compare scores for Pooh between Time 1 and Time 2. Here,
we’ve recorded the identity of the student raters, and Pooh’s score for each rater. This allows us to
focus on the changes for each rater between Time 1 and Time 2. This makes for a more-powerful test
than would the Mann–Whitney U test in cases like this where one rater might tend to rate high and

206
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

another rater might tend to rate low, but there is an overall trend in how raters change their scores
between Time 1 and Time 2.

Note in this example we needed to record the identity of the student rater so that a rater’s score from
Time 1 can be paired with their score from Time 2. If we cannot pair data in this way—for example, if
we did not record the identity of the raters—the data would have to be treated as unpaired,
independent samples, for example like those in the Two-sample Mann–Whitney U Test chapter.

Also note that the data is arranged in long form. In this form, the data must be ordered so that the first
observation where Time = 1 is paired to the first observation where Time = 2, and so on.
Input =("
Speaker Time Student Likert
Pooh 1 a 1
Pooh 1 b 4
Pooh 1 c 3
Pooh 1 d 3
Pooh 1 e 3
Pooh 1 f 3
Pooh 1 g 4
Pooh 1 h 3
Pooh 1 i 3
Pooh 1 j 3
Pooh 2 a 4
Pooh 2 b 5
Pooh 2 c 4
Pooh 2 d 5
Pooh 2 e 4
Pooh 2 f 5
Pooh 2 g 3
Pooh 2 h 4
Pooh 2 i 3
Pooh 2 j 4
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

207
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot the paired data

Scatter plot with one-to-one line

Paired data can be visualized with a scatter plot of the paired cases. In the plot below, points that fall
above and to the left of the blue line indicate cases for which the value for Time 2 was greater than for
Time 1.

Note that the points in the plot are jittered slightly so that points which would fall directly on top of
one another can be seen.

First, two new variables, Time.1 and Time.2, are created by extracting the values of Likert for
observations with the Time variable equal to 1 or 2, respectively, and then the plot is produced.
Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]

plot(Time.1, jitter(Time.2), # jitter offsets points so you can see them all
pch = 16, # shape of points
cex = 1.0, # size of points
xlim=c(1, 5.5), # limits of x axis
ylim=c(1, 5.5), # limits of y axis
xlab="Time 1",
ylab="Time 2"
)
abline(0,1, col="blue", lwd=2) # line with intercept of 0 and slope of 1

Bar plot of differences

Paired data can also be visualized with a bar chart of differences. In the plot below, bars with a value
greater than zero indicate cases for which values for Time 2 are greater than for Time 1.

New variables are first created for Time.1, Time.2, and their Difference. And then the plot is produced.

208
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]

Difference = Time.2 - Time.1

barplot(Difference, # variable to plot

col="dark gray", # color of bars
xlab="Observation", # x-axis label
ylab="Difference (Time 2 – Time 1)") # y-axis label

Bar plot of differences

A bar plot of differences in paired data can be used to see if the differences in paired observations are
symmetrical. It is an assumption of the two-sample paired rank-sum test that the distribution of the
differences is symmetrical. For this example, the distribution is relatively symmetrical, suggesting the
rank-sum test is appropriate.

Here, new variables are created: Time.1, Time.2, Difference, and Diff.f, which has the same values as
Difference but as a factor variable. The xtabs function is used to create a count of values of Diff.f. The
barplot function then uses these counts.
Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]

Difference = Time.2 - Time.1

Diff.f = factor(Difference,
ordered = TRUE)

X = xtabs(~ Diff.f)

barplot(X,
col="dark gray",
xlab="Difference in Likert",
ylab="Frequency")

209
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Two-sample paired sign-rank test

Note that if data are in long format, the data must be ordered so that the first observation of Time 1 is
paired to the first observation of Time 2, and so on, because the wilcox.test function will take the
observations in order.

This example uses the formula notation indicating that Likert is the dependent variable and Time is the
independent variable. The data= option indicates the data frame that contains the variables, and
paired=TRUE indicates that the test for paired data should be used. For the meaning of other options,
see ?wilcox.test.
wilcox.test(Likert ~ Time,
data = Data,
paired = TRUE,
conf.int = TRUE,
conf.level = 0.95)

Wilcoxon signed rank test with continuity correction

V = 3.5, p-value = 0.02355
alternative hypothesis: true location shift is not equal to 0

### Note the p-value given in the above results

95 percent confidence interval:

-2.000051e+00 -1.458002e-05

### Confidence interval of median or location of differences

### You may get a "cannot compute exact p-value with ties" error.
### You can ignore this or use the exact=FALSE option.

210
 TWO-SAMPLE PAIRED RANK-SUM TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Exercises K

1. Considering Pooh’s data for Time 1 and Time 2,

What do the plots suggest about the relative value of the scores for Time 1 and Time 2? Do they
suggest that scores increased, decreased, or stayed the same between Time 1 and Time 2?

Is the distribution of the differences between paired samples relatively symmetrical? Is the
two-sample paired ranked-sum test appropriate in this case?

Does the two-sample paired ranked-sum test indicate that there is a significant difference
between Time 1 and Time 2?

2. Lois Griffin gave proficiency scores to her students in her course on piano playing for adults. She
gave a score for each student for their left hand playing and right hand playing. She wants to know if
students in her class are more proficient in the right hand, left hand, or if there is no difference in
hands.
Instructor Student Hand Score
'Lois Griffin' a left 8
'Lois Griffin' a right 9
'Lois Griffin' b left 6
'Lois Griffin' b right 5
'Lois Griffin' c left 7
'Lois Griffin' c right 9
'Lois Griffin' d left 6
'Lois Griffin' d right 7
'Lois Griffin' e left 7
'Lois Griffin' e right 7
'Lois Griffin' f left 9
'Lois Griffin' f right 9
'Lois Griffin' g left 4
'Lois Griffin' g right 6
'Lois Griffin' h left 5
'Lois Griffin' h right 8
'Lois Griffin' i left 5
'Lois Griffin' i right 6
'Lois Griffin' j left 7
'Lois Griffin' j right 8

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

Is the distribution of the differences between paired samples relatively symmetrical? Is the
two-sample paired ranked-sum test appropriate in this case?

211
 SIGN TEST FOR TWO-SAMPLE PAIRED DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Does the two-sample paired ranked-sum test indicate that there is a difference between hands?
If so, which hand received higher scores?

What can you conclude about the results of the plots, summary statistics, and statistical test?

What if Lois wanted to change the design of the experiment so that she could determine if each
student were more proficient in one hand or the other? That is, is student a more proficient
in left hand or right? Is student b more proficient in left hand or right? How should she
change what data she’s collecting to determine this?

Sign Test for Two-sample Paired Data

The sign test for two-sample paired data can be used for a similar purpose as the paired rank-sum test
in cases where the distribution of the differences is not symmetrical.

The SIGN.test function in the BSDA package requires the data to be separated into two variables, each
of which is ordered so that the first observation of each are paired, and so on. Information on options
for the function can be viewed with ?SIGN.test. The SignTest function in the DescTools package is
similar.

For appropriate plots see the Two-sample Paired Rank-sum Test chapter.

Appropriate data
• Two-sample paired data. That is, one-way data with two groups only, where the
observations are paired between groups.
• Dependent variable is ordinal, interval, or ratio
• Independent variable is a factor with two levels. That is, two groups

Hypotheses
• Null hypothesis: The median of the differences between pairs in group A and group B is zero.
• Alternative hypothesis (two-sided): The median of the differences between pairs in group A
and group B is not zero.

Interpretation
Significant results can be reported as “There was a significant difference in values between
group A and group B.”

Packages used in this chapter

The packages used in this chapter include:

• psych
• BSDA
• DescTools

212
 SIGN TEST FOR TWO-SAMPLE PAIRED DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(BSDA)){install.packages("BSDA")}
if(!require(DescTools)){install.packages("DescTools")}

Sign test for paired two-sample data example

Input =("
Speaker Time Student Likert
Pooh 1 a 1
Pooh 1 b 4
Pooh 1 c 3
Pooh 1 d 3
Pooh 1 e 3
Pooh 1 f 3
Pooh 1 g 4
Pooh 1 h 3
Pooh 1 i 3
Pooh 1 j 3
Pooh 2 a 4
Pooh 2 b 5
Pooh 2 c 4
Pooh 2 d 5
Pooh 2 e 4
Pooh 2 f 5
Pooh 2 g 3
Pooh 2 h 4
Pooh 2 i 3
Pooh 2 j 4
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

213
SIGN TEST FOR TWO-SAMPLE PAIRED DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Two-sample sign test with BSDA package

Time.1 = Data$Likert [Data$Time == 1]
Time.2 = Data$Likert [Data$Time == 2]

library(BSDA)

SIGN.test(x = Time.1,
y = Time.2,
alternative = "two.sided",
conf.level = 0.95)

Dependent-samples Sign-Test

S = 1, p-value = 0.03906

### p-value reported above

95 percent confidence interval:

-2.0000000 -0.3244444

sample estimates:
median of x-y
-1

### median of differences and confidence interval of differences

Two-sample sign test with DescTools package

Time.1 = Data$Likert [Data$Time == 1]
Time.2 = Data$Likert [Data$Time == 2]

library(DescTools)

SignTest(x = Time.1,
y = Time.2)

Dependent-samples Sign-Test

S = 1, number of differences = 9, p-value = 0.03906

### p-value reported above

alternative hypothesis: true median difference is not equal to 0

97.9 percent confidence interval:

-2 0

sample estimates:
median of the differences
-1

### median of differences and confidence interval of differences

214
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Kruskal–Wallis Test
The Kruskal–Wallis test is a rank-based test that is similar to the Mann–Whitney U test, but can be
applied to one-way data with more than two groups. The test is useful to compare the scores or
ratings from multiple speakers, presentations, or groups of audiences.

If the shape and spread of the distributions of values of each group is similar, then the test compares
the medians of the two groups. Otherwise, the test is really testing if there is a systematic difference in
the values among the groups.

The test is performed with the kruskal.test function.

If the distributions of values of each group are similar in shape, but have outliers, then Mood’s median
test is an appropriate alternative. Mood’s median test is described in the next chapter.

Post-hoc tests
The outcome of the Kruskal–Wallis test tells you if there are differences among the groups, but doesn’t
tell you which groups are different from other groups. In order to determine which groups are
different from others, post-hoc testing can be conducted. Probably the most common post-hoc test for
the Kruskal–Wallis test is the Dunn test, here conducted with the dunnTest function in the FSA
package. An alternative to this is to conduct Mann–Whitney tests on each pair of groups. This is
accomplished with pairwise.wilcox.test function.

Appropriate data
• One-way data
• Dependent variable is ordinal, interval, or ratio
• Independent variable is a factor with two or more levels. That is, two or more groups
• Observations between groups are independent. That is, not paired or repeated measures
data
• In order to be a test of medians, the distributions of values for each group need to be of
similar shape and spread. Otherwise the test is a test of distributions.

Hypotheses
If the distributions of the two groups are similar in shape and spread:
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

If the distributions of the two groups are not similar in shape and spread:
• Null hypothesis: The distribution of values for each group are equal.
• Alternative hypothesis (two-sided): There is systematic difference in the distribution of
values for the groups.

215
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interpretation
If the distributions of the two groups are similar in shape and spread:
Significant results can be reported as “There was a significant difference in median values
across groups.”
Post-hoc analysis allows you to say “The median for group A was higher than the median for
group B”, and so on.

If the distributions of the two groups are not similar in shape and spread:
Significant results can be reported as “There was a significant difference in values among
groups.”

Other notes and alternative tests

Mood’s median test compares the medians of groups. It is described in the next chapter.

Another alternative is to use cumulative link models for ordinal data, which are described later in this
book.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• multcompView
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(rcompanion)){install.packages("rcompanion")}

Kruskal–Wallis test example

This example re-visits the Pooh, Piglet, and Tigger data from the Descriptive Statistics with the likert
Package chapter.

It answers the question, “Are the scores significantly different among the three speakers?”

The Kruskal–Wallis test is conducted with the kruskal.test function, which produces a p-value for the
hypothesis. First the data are summarized and examined using bar plots for each group.

Note that because the bar plot show that the distributions of scores for each of the speakers are
relatively similar in shape and spread, the Kruskal–Wallis test can be interpreted as a test of medians.

216
KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2
Piglet 3
Tigger 4
Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

217
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Speaker + Likert.f,
data = Data)

Likert.f
Speaker 1 2 3 4 5
Pooh 0 0 1 6 3
Piglet 1 6 2 1 0
Tigger 0 0 2 6 2

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5
Pooh 0.0 0.0 0.1 0.6 0.3
Piglet 0.1 0.6 0.2 0.1 0.0
Tigger 0.0 0.0 0.2 0.6 0.2

Bar plots of data by group

library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

218
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n mean sd min Q1 median Q3 max percZero

1 Pooh 10 4.2 0.632 3 4 4 4.75 5 0
2 Piglet 10 2.3 0.823 1 2 2 2.75 4 0
3 Tigger 10 4.0 0.667 3 4 4 4.00 5 0

Kruskal–Wallis test example

This example uses the formula notation indicating that Likert is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?kruskal.test.
kruskal.test(Likert ~ Speaker,
data = Data)

Kruskal-Wallis rank sum test

Kruskal-Wallis chi-squared = 16.842, df = 2, p-value = 0.0002202

219
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Post-hoc test: Dunn test for multiple comparisons of groups

If the Kruskal–Wallis test is significant, a post-hoc analysis can be performed to determine which
groups differ from each other group.

Probably the most popular host-hoc test for the Kruskal–Wallis test is the Dunn test. The Dunn test
can be conducted with the dunnTest function in the FSA package.

Because the post-hoc test will produce multiple p-values, adjustments to the p-values can be made to
avoid inflating the possibility of making a type-I error. There are a variety of methods for controlling
the familywise error rate or for controlling the false discovery rate. See ?p.adjust for details on these
methods.

When there are many p-values to evaluate, it is useful to condense a table of p-values to a compact
letter display format. In the output, groups are separated by letters. Groups sharing the same letter
are not significantly different. Compact letter displays are a clear and succinct way to present results
of multiple comparisons.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

levels(Data$Speaker)

### Dunn test

library(FSA)

DT = dunnTest(Likert ~ Speaker,
data=Data,
method="bh") # Adjusts p-values for multiple comparisons;
# See ?dunnTest for options

DT

Dunn (1964) Kruskal-Wallis multiple comparison

p-values adjusted with the Benjamini-Hochberg method.

Comparison Z P.unadj P.adj

1 Pooh - Tigger 0.4813074 0.6302980448 0.6302980448
2 Pooh - Piglet 3.7702412 0.0001630898 0.0004892695
3 Tigger - Piglet 3.2889338 0.0010056766 0.0015085149

### Compact letter display

PT = DT$res

PT

library(rcompanion)

220
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

cldList(comparison = PT$Comparison,
p.value = PT$P.adj,
threshold = 0.05)

Group Letter MonoLetter

1 Pooh a a
2 Tigger a a
3 Piglet b b

Groups sharing a letter not signficantly different (alpha = 0.05).

Post-hoc test: pairwise Mann–Whitney U-tests for multiple comparisons

Another approach to post-hoc testing for the Kruskal–Wallis test is to use Mann–Whitney U-tests for
each pair of groups.

This can be conducted with the pairwise.wilcox.test function. This produces a table of p-values
comparing each pair of groups.

To prevent the inflation of type I error rates, adjustments to the p-values can be made using the
p.adjust.method option. Here the fdr method is used. See ?p.adjust for details on available p-value
adjustment methods.

When there are many p-values to evaluate, it is useful to condense a table of p-values to a compact
letter display format. This can be accomplished with a combination of the fullPTable function in the
rcompanion package and the multcompLetters function in the multcompView package.

A compact letter display condenses a table of p-values into a simpler format. In the output, groups are
separated by letters. Groups sharing the same letter are not significantly different. Compact letter
displays are a clear and succinct way to present results of multiple comparisons.

Here the fdr p-value adjustment method is used. See ?p.adjust for details on available methods.

The code creates a matrix of p-values called PT, then converts this to a fuller matrix called PT1. PT1 is
then passed to the multcompLetters function to be converted to a compact letter display.

Note that the p-value results of the pairwise Mann–Whitney U-tests differ somewhat from those of the
Dunn test.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

Data

### Pairwise Mann–Whitney

221
KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

PT = pairwise.wilcox.test(Data$Likert,
Data$Speaker,
p.adjust.method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Pairwise comparisons using Wilcoxon rank sum test

Pooh Tigger
Tigger 0.5174 -
Piglet 0.0012 0.0012

P value adjustment method: fdr

### Note that the values in the table are p-values comparing each
### pair of groups.

### Convert PT to a full table and call it PT1

PT = PT$p.value ### Extract p-value table

library(rcompanion)

PT1 = fullPTable(PT)

PT1

Pooh Tigger Piglet

Pooh 1.000000000 0.517377650 0.001241095
Tigger 0.517377650 1.000000000 0.001241095
Piglet 0.001241095 0.001241095 1.000000000

### Produce compact letter display

library(multcompView)

multcompLetters(PT1,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Pooh Tigger Piglet

"a" "a" "b"

### Values sharing a letter are not significantly different

222
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of medians and confidence intervals

The following code uses the groupwiseMedian function to produce a data frame of medians for each
speaker along with the 95% confidence intervals for each median with the percentile method.

These medians are then plotted, with their confidence intervals shown as error bars. The grouping
letters from the multiple comparisons (Dunn test or pairwise Mann–Whitney U-tests) are added.
library(rcompanion)

Sum = groupwiseMedian(data=Data,
group="Speaker",
var="Likert",
conf=0.95,
R=5000,
percentile=TRUE,
bca=FALSE,
digits=3)

Sum

Speaker n Median Conf.level Percentile.lower Percentile.upper

1 Pooh 10 4 0.95 4.0 5.0
2 Piglet 10 2 0.95 2.0 3.0
3 Tigger 10 4 0.95 3.5 4.5

X = 1:3
Y = Sum$Percentile.upper + 0.2
Label = c("a", "b", "a")

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Speaker,
y = Median)) +
geom_errorbar(aes(ymin = Percentile.lower,
ymax = Percentile.upper),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Median Likert score")+

annotate("text",
x = X,
y = Y,
label = Label)

223
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of median Likert versus Speaker. Error bars indicate the 95% confidence
intervals for the median with the percentile method.

Exercises L

1. Considering Pooh, Piglet, and Tigger’s data,

What was the median score for each instructor?

Are the data for all instructors reasonably similar shape and spread?

Based on your previous answer, what is the null hypothesis for the Kruskal–Wallis test?

According to the Kruskal–Wallis test, is there a statistical difference in scores among the
instructors?

Looking at the post-hoc analysis, which speakers’ scores are statistically different from which
others? Who had the statistically highest scores?

2. Brian, Stewie, and Meg want to assess the education level of students in their courses on creative
writing for adults. They want to know the median education level for each class, and if the education
level of the classes were different among instructors.

They used the following table to code his data.

224
 KRUSKAL–WALLIS TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Code Abbreviation Level

1 < HS Less than high school

2 HS High school
3 BA Bachelor’s
4 MA Master’s
5 PhD Doctorate

The following are the course data.

Instructor Student Education

'Brian Griffin' a 3
'Brian Griffin' b 2
'Brian Griffin' c 3
'Brian Griffin' d 3
'Brian Griffin' e 3
'Brian Griffin' f 3
'Brian Griffin' g 4
'Brian Griffin' h 5
'Brian Griffin' i 3
'Brian Griffin' j 4
'Brian Griffin' k 3
'Brian Griffin' l 2
'Stewie Griffin' m 4
'Stewie Griffin' n 5
'Stewie Griffin' o 4
'Stewie Griffin' p 4
'Stewie Griffin' q 4
'Stewie Griffin' r 4
'Stewie Griffin' s 3
'Stewie Griffin' t 5
'Stewie Griffin' u 4
'Stewie Griffin' v 4
'Stewie Griffin' w 3
'Stewie Griffin' x 2
'Meg Griffin' y 3
'Meg Griffin' z 4
'Meg Griffin' aa 3
'Meg Griffin' ab 3
'Meg Griffin' ac 3
'Meg Griffin' ad 2
'Meg Griffin' ae 3
'Meg Griffin' af 4
'Meg Griffin' ag 2
'Meg Griffin' ah 3
'Meg Griffin' ai 2
'Meg Griffin' aj 1

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

225
 MOOD’S MEDIAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

What was the median education level for each instructor’s class? (Be sure to report the
education level, not just the numeric code!)

Are the distributions of education levels for all instructors reasonably similar shape and
spread?

Based on your previous answer, what is the null hypothesis for the Kruskal–Wallis test?

According to the Kruskal–Wallis test, is there a difference in the education level of students
among the instructors?

Looking at the post-hoc analysis, which classes education levels are different from which
others? Who had the statistically highest education level?

Plot Brian, Stewie, and Meg’s data in a way that helps you visualize the data. Do the results
reflect what you would expect from looking at the plot? How would you summarize the
results of the descriptive statistics and tests?

Mood’s Median Test

Mood’s median test compares the medians of two or more groups. It is often less powerful than the
Kruskal–Wallis test, but can be used when the spread of values in the groups are different.

The test can be conducted with the mood.medtest function in the RVAideMemoire package.

Post-hoc tests
The outcome of Mood’s median test tells you if there are differences among the groups, but doesn’t tell
you which groups are different from other groups. In order to determine which groups are different
from others, post-hoc testing can be conducted. The function pairwiseMedianTest in the rcompanion
package can perform the post-hoc tests.

Appropriate data
• One-way data with two or more groups
• Dependent variable is ordinal, interval, or ratio
• Independent variable is a factor with levels indicating groups
• Observations between groups are independent. That is, not paired or repeated measures
data
• Distributions of values for each group are similar in shape; however, the test is not sensitive
to outliers

Hypotheses
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

226
 MOOD’S MEDIAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interpretation
Significant results can be reported as “There was a significant difference in the median values
among groups.”
Post-hoc analysis allows you to say “The median for group A was higher than the median for
group B”, and so on.

Packages used in this chapter

The packages used in this chapter include:

• RVAideMemoire
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(rcompanion)){install.packages("rcompanion")}

Example using the RVAideMemoire package

This example uses the formula notation indicating that Likert is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?mood.medtest.

A significant p-value for Mood’s median test indicates that not all medians among groups are equal.

For appropriate plots and data frame checking, see the Kruskal–Wallis Test chapter.
Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2

227
 MOOD’S MEDIAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Piglet 3
Tigger 4
Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Mood’s median test

library(RVAideMemoire)

mood.medtest(Likert ~ Speaker,
data = Data)

Mood's median test

p-value = 0.3211

An interesting thing happened with the result here. The test counts how many observations in each
group are greater than the global median for all groups together, in this case 4. It then tests if there is
a significant difference in this proportion among groups. For this data set, however, both Pooh and
Tigger have a majority of observations equal to the global median. Because they are equal to the global
median, they are not greater than the global median, and so aren’t much different than Piglet’s scores
on this count. The result in this case is a non-significant p-value.

But the test would come out differently if we were counting observation less than the global median,
because Pooh and Tigger have few of these, and Piglet has relatively many.

228
 MOOD’S MEDIAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

This is a quirk with Mood’s median test, and isn’t common in statistical tests.

One solution would be to re-code the function to count observations less than the global median. This
would be possible.

But it is easier to simply invert the scale we are using. This is really an arbitrary change, but for this
test, it can make a difference. Imagine if our original scale interpreted 5 to be the best, and 1 to be the
worst. When we designed the survey tool, we could just as easily have made 1 the best and 5 the
worst. And then instead of ranking “good” with a 4, the respondents would have marked it 2, and so
on. By the way the calculations are done, this arbitrary change in scale will change the results of
Mood’s median test.

For a 5-point scale, we do this inversion by simply by making a new variable equal to 6 minus the
original score.

With Mood’s median test, I recommend making this kind of inversion in cases where many values are
equal to the global median. Then use whichever result has a lower p-value.
Data$Likert.inv = 6 - Data$Likert

library(psych)

headTail(Data)

Speaker Likert Likert.inv

1 Pooh 3 3
2 Pooh 5 1
3 Pooh 4 2
4 Pooh 4 2
... <NA> ... ...
27 Tigger 5 1
28 Tigger 4 2
29 Tigger 4 2
30 Tigger 3 3

library(RVAideMemoire)

mood.medtest(Likert.inv ~ Speaker,
data = Data)

Mood's median test

p-value = 0.000402

Post-hoc test: pairwiseMedianTest function

If Mood’s median test is significant, a post-hoc analysis can be performed to determine which groups
differ from each other group.

229
 MOOD’S MEDIAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For this we will use the pairwiseMedianTest function in the rcompanion package, which conducts
Mood’s median test on all pairs of groups from one-way data.

Because the post-hoc test will produce multiple p-values, adjustments to the p-values can be made to
avoid inflating the possibility of making a type-I error. There are a variety of methods for controlling
the familywise error rate or for controlling the false discovery rate. See ?p.adjust for details on these
methods.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Pairwise median tests

library(rcompanion)

PT = pairwiseMedianTest(x = Data$Likert,
g = Data$Speaker,
exact = NULL,
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Comparison p.value p.adjust

1 Pooh - Tigger = 0 1 1.000000
2 Pooh - Piglet = 0 0.001093 0.003279
3 Tigger - Piglet = 0 0.005477 0.008216

### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 Pooh a a
2 Tigger a a
3 Piglet b b

Groups sharing a letter are not significantly different (alpha = 0.05).

Post-hoc test: pairwiseMedianMatrix function

### Order groups by median

230
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Pairwise median tests

library(rcompanion)

PT = pairwiseMedianMatrix(x = Data$Likert,
g = Data$Speaker,
exact = NULL,
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

library(multcompView)

multcompLetters(PT$Adjusted,
compare="<",
threshold=0.05,
Letters=letters)

Pooh Tigger Piglet

"a" "a" "b"

Friedman Test
The Friedman test determines if there are differences among groups for two-way data structured in a
specific way, namely in an unreplicated complete block design. In this design, one variable serves as the
treatment or group variable, and another variable serves as the blocking variable. It is the differences
among treatments or groups that we are interested in. We aren’t necessarily interested in differences
among blocks, but we want our statistics to take into account differences in the blocks. In the
unreplicated complete block design, each block has one and only one observation of each treatment.

For an example of this structure, look at the Belcher family data below. Rater is considered the
blocking variable, and each rater has one observation for each Instructor. The test will determine if
there are differences among values for Instructor, taking into account any consistent effect of a Rater.
For example if Rater a rated consistently low and Rater g rated consistently high, the Friedman test
can account for this statistically.

In other cases, the blocking variable might be the class where the ratings were done or the school
where the ratings were done. If you were testing differences among curricula or other teaching
treatments with different instructors, different instructors might be used as blocks.

231
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

If the distribution of the differences in scores between each pair of groups are all symmetrical, or if the
distribution of values for each group is similar in shape and spread, the Friedman test determines if
there is a difference in medians among groups. If not, the test determines if there is a systematic
difference in the values among the groups.

Some people critique the Friedman test for having low power in detecting differences among groups.
It has been suggested, however, that Friedman test may be powerful when there are five or more
groups.

Post-hoc tests
The outcome of the Friedman test tells you if there are differences among the groups, but doesn’t tell
you which groups are different from other groups. In order to determine which groups are different
from others, post-hoc testing can be conducted.

For a post-hoc analysis, the pairwiseSignTest function can be used. It performs a two-sample paired
sign test on each pair of groups.

Appropriate data
• Two-way data arranged in an unreplicated complete block design
• Dependent variable is ordinal, interval, or ratio
• Treatment or group independent variable is a factor with two or more levels. That is, two or
more groups
• Blocking variable is a factor with two or more levels
• Blocks are independent of each other and have no interaction with treatments
• In order to be a test of medians, the distribution of the differences in scores between each
pair of groups are all symmetrical, or the distributions of values for each group have similar
shape and spread. Otherwise the test is a test of distributions.

Hypotheses
If the distribution of the differences in scores between each pair of groups are all symmetrical, or the
distributions of values for each group have similar shape and spread:
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

If the above conditions are not met:

• Null hypothesis: The distributions of values for each group are equal.
• Alternative hypothesis (two-sided): There is systematic difference in the distribution of
values for the groups.

Interpretation
If the distribution of the differences in scores between each pair of groups are all symmetrical, or the
distributions of values for each group have similar shape and spread:
Significant results can be reported as “There was a significant difference in median values
across groups.”
Post-hoc analysis allows you to say “The median for group A was higher than the median for
group B”, and so on.

232
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

If the above conditions are not met:

Significant results can be reported as “There was a significant difference in values among
groups.”

Other notes and alternative tests

The Quade test is used for the same kinds of data and hypotheses, but can be more powerful in some
cases. It has been suggested that Friedman test may be preferable when there are a larger number of
groups (five or more), while the Quade is preferable for fewer groups. The Quade test is described in
the next chapter.

Another alternative is to use cumulative link models for ordinal data, which are described later in this
book.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• BSDA
• multcompView
• PMCMR
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(BSDA)){install.packages("BSDA")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(PMCMR)){install.packages("PMCMR")}
if(!require(rcompanion)){install.packages("rcompanion")}

Friedman test example

Input =("
Instructor Rater Likert
'Bob Belcher' a 4
'Bob Belcher' b 5
'Bob Belcher' c 4
'Bob Belcher' d 6
'Bob Belcher' e 6
'Bob Belcher' f 6
'Bob Belcher' g 10
'Bob Belcher' h 6
'Linda Belcher' a 8
'Linda Belcher' b 6
'Linda Belcher' c 8

233
FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Linda Belcher' d 8
'Linda Belcher' e 8
'Linda Belcher' f 7
'Linda Belcher' g 10
'Linda Belcher' h 9
'Tina Belcher' a 7
'Tina Belcher' b 5
'Tina Belcher' c 7
'Tina Belcher' d 8
'Tina Belcher' e 8
'Tina Belcher' f 9
'Tina Belcher' g 10
'Tina Belcher' h 9
'Gene Belcher' a 6
'Gene Belcher' b 4
'Gene Belcher' c 5
'Gene Belcher' d 5
'Gene Belcher' e 6
'Gene Belcher' f 6
'Gene Belcher' g 5
'Gene Belcher' h 5
'Louise Belcher' a 8
'Louise Belcher' b 7
'Louise Belcher' c 8
'Louise Belcher' d 8
'Louise Belcher' e 9
'Louise Belcher' f 9
'Louise Belcher' g 8
'Louise Belcher' h 10
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

234
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Instructor + Likert.f,
data = Data)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 2 1 4 0 0 0 1
Linda Belcher 0 0 1 1 4 1 1
Tina Belcher 0 1 0 2 2 2 1
Gene Belcher 1 4 3 0 0 0 0
Louise Belcher 0 0 0 1 4 2 1

XT = xtabs( ~ Instructor + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 0.250 0.125 0.500 0.000 0.000 0.000 0.125
Linda Belcher 0.000 0.000 0.125 0.125 0.500 0.125 0.125
Tina Belcher 0.000 0.125 0.000 0.250 0.250 0.250 0.125
Gene Belcher 0.125 0.500 0.375 0.000 0.000 0.000 0.000
Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

Bar plots by group

Note that the bar plots don’t show the effect of the blocking variable.
library(lattice)

histogram(~ Likert.f | Instructor,

data=Data,
layout=c(1,5) # columns and rows of individual plots
)

235
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plots of differences between groups

We can make a bar plot of the differences in values for two groups, just as we did for the sign test
previously. Values for Bob and Louisa were chosen based on their bar plots having very different
shapes.

The resulting plot shows a distribution that is clearly not symmetrical.

Note that the data must be ordered by the blocking variable so that the first observation for Louisa will
be paired with the first observation for Bob, and so on.

Also note that we had to specify the levels in the factor function defining Diff.f. This is so that the
values with zero counts will be displayed on the plot.
Bob = Data$Likert [Data$Instructor == "Bob Belcher"]
Louisa = Data$Likert [Data$Instructor == "Louise Belcher"]

Difference = Bob - Louisa

Diff.f = factor(Difference,
ordered = TRUE,
levels = c("-4", "-3", "-2", "-1", "0", "1", "2", "3", "4")

236
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

X = xtabs(~ Diff.f)

barplot(X,
col="dark gray",
xlab="Difference in Likert",
ylab="Frequency")

pairwiseDifferences to produce bar plots of differences between all groups

The pairwiseDifferences function in the rcompanion package will create a new data frame of differences
for all pairs of differences. The plotit=TRUE option will produce bar plots of the counts for each pair.
Otherwise the lattice package can be used to show these plots in one large trellis plot.

Note that the data must be ordered by the blocking variable so that the first observation for Bob will be
paired with the first observation for Linda, and so on.
library(rcompanion)

Data.diff=pairwiseDifferences(Data$Likert,
Data$Instructor,
factorize = TRUE,
plotit = TRUE)

library(psych)

headTail(Data.diff)

Comparison Difference Difference.f

1 Bob Belcher - Linda Belcher -4 -4
2 Bob Belcher - Linda Belcher -1 -1
3 Bob Belcher - Linda Belcher -4 -4
4 Bob Belcher - Linda Belcher -2 -2
... <NA> ... <NA>

237
FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

77 Gene Belcher - Louise Belcher -3 -3

78 Gene Belcher - Louise Belcher -3 -3
79 Gene Belcher - Louise Belcher -3 -3
80 Gene Belcher - Louise Belcher -5 -5

library(lattice)

histogram(~ Difference | Comparison,

data=Data.diff,
type = "count",
layout=c(2,5) # columns and rows of individual plots
)

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n mean sd min Q1 median Q3 max percZero

1 Bob Belcher 8 5.875 1.885 4 4.75 6 6.00 10 0
2 Linda Belcher 8 8.000 1.195 6 7.75 8 8.25 10 0
3 Tina Belcher 8 7.875 1.553 5 7.00 8 9.00 10 0
4 Gene Belcher 8 5.250 0.707 4 5.00 5 6.00 6 0
5 Louise Belcher 8 8.375 0.916 7 8.00 8 9.00 10 0

238
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Friedman test example

This example uses the formula notation indicating that Likert is the dependent variable, Instructor is
the independent variable, and Rater is the blocking variable. The data= option indicates the data
frame that contains the variables. For the meaning of other options, see ?friedman.test
friedman.test(Likert ~ Instructor | Rater,
data = Data)

Friedman rank sum test

Friedman chi-squared = 23.139, df = 4, p-value = 0.0001188

Post-hoc test: pairwise sign test for multiple comparisons of groups

Post-hoc testing can be conducted with the functions pairwiseSignTest and pairwiseSignMatrix. These
functions conduct a paired two-sample sign test on each pair of groups, and output the results as
either a table or a matrix, respectively. The matrix output can be converted to a compact letter display
using the multcompLetters function in the multcompView package.

To prevent the inflation of type I error rates, adjustments to the p-values can be made using the
p.adjust.method option. See ?p.adjust for details on available p-value adjustment methods.

It has been suggested that the sign test may lack power in detecting differences in paired data sets.
But is useful because it has few assumptions about the distributions of the data to compare, and is the
test analogous to the Friedman test with two groups.

Table format and compact letter display

Note that the data must be ordered by the blocking variable so that the first observation for Bob will be
paired with the first observation for Linda, and so on.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise sign tests

library(rcompanion)

PT = pairwiseSignTest(Data$Likert,
Data$Instructor,
method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

239
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Comparison S p.value p.adjust

1 Linda Belcher - Louise Belcher = 0 1 0.375 0.46880
2 Linda Belcher - Tina Belcher = 0 3 0.625 0.68750
3 Linda Belcher - Bob Belcher = 0 7 0.01563 0.03908
4 Linda Belcher - Gene Belcher = 0 8 0.007812 0.02604
5 Louise Belcher - Tina Belcher = 0 5 0.2187 0.31240
6 Louise Belcher - Bob Belcher = 0 7 0.07031 0.11720
7 Louise Belcher - Gene Belcher = 0 8 0.007812 0.02604
8 Tina Belcher - Bob Belcher = 0 6 0.03125 0.06250
9 Tina Belcher - Gene Belcher = 0 8 0.007812 0.02604
10 Bob Belcher - Gene Belcher = 0 4 0.6875 0.68750

### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 LindaBelcher a a
2 LouiseBelcher ab ab
3 TinaBelcher ab ab
4 BobBelcher bc bc
5 GeneBelcher c c

Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix format and compact letter display

Note that the data must be ordered by the blocking variable so that the first observation for Bob will be
paired with the first observation for Linda, and so on.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise sign tests

library(rcompanion)

PM = pairwiseSignMatrix(Data$Likert,
Data$Instructor,
method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

240
 FRIEDMAN TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

PM

$Adjusted
Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
Linda Belcher 1.00000 0.46880 0.68750 0.03908 0.02604
Louise Belcher 0.46880 1.00000 0.31240 0.11720 0.02604
Tina Belcher 0.68750 0.31240 1.00000 0.06250 0.02604
Bob Belcher 0.03908 0.11720 0.06250 1.00000 0.68750
Gene Belcher 0.02604 0.02604 0.02604 0.68750 1.00000

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "ab" "ab" "bc" "c"

### Groups sharing a letter are not significantly different.

Post-hoc Conover test

A different post-hoc test for the Friedman test can be conducted with the posthoc.friedman.conover.test
function in the PMCMR package. The output can be converted to a compact letter display using the
multcompLetters function in the multcompView package.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Conover test

library(PMCMR)

PT = posthoc.friedman.conover.test(y = Data$Likert,
groups = Data$Instructor,
blocks = Data$Rater,
p.adjust.method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Pairwise comparisons using Conover's test for a two-way

balanced complete block design

241
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Linda Belcher Louise Belcher Tina Belcher Bob Belcher

Louise Belcher 0.27303 - - -
Tina Belcher 0.31821 0.05154 - -
Bob Belcher 2.8e-05 1.9e-06 0.00037 -
Gene Belcher 1.2e-06 1.1e-07 8.8e-06 0.17328

P value adjustment method: fdr

### Compact letter display

PT0 = as.matrix(PT$p.value)

library(rcompanion)

PT1 = fullPTable(PT0)

library(multcompView)

multcompLetters(PT1,
compare="<",
threshold=0.05,
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "a" "a" "b" "b"

Quade Test
The Quade test is used for similar data and hypotheses as the Friedman test, namely for unreplicated
complete block designs.

While the Friedman test is a generalization of the paired sign test, the Quade test is a generalization of
the two-sample signed-rank test.

Some authors indicate that the Quade test is appropriate for ordinal data, while others suggest that it
is appropriate only for interval or ratio data. It is included in this section as an alternative to the
Friedman test.

Post-hoc tests
The outcome of the Quade test tells you if there are differences among the groups, but doesn’t tell you
which groups are different from other groups. In order to determine which groups are different from
others, post-hoc testing can be conducted with the pairwise.wilcox.test function.

242
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Appropriate data
• Two-way data arranged in an unreplicated complete block design
• Dependent variable is ordinal, interval, or ratio, although some authors say data must be
interval or ratio only.
• Treatment or group independent variable is a factor with two or more levels. That is, two or
more groups
• Blocking variable is a factor with two or more levels
• Blocks are independent of each other and have no interaction with treatments
• In order to be a test of medians, the distributions of values for each group should have similar
shape and spread. If not, the test determines if there is a systematic difference in the values
among the groups.

Hypotheses
If the distribution of the differences in scores between each pair of groups are all symmetrical, or the
distributions of values for each group have similar shape and spread:
• Null hypothesis: The medians of values for each group are equal.
• Alternative hypothesis (two-sided): The medians of values for each group are not equal.

If the above conditions are not met:

• Null hypothesis: The distributions of values for each group are equal.
• Alternative hypothesis (two-sided): There is systematic difference in the distribution of
values for the groups.

Interpretation
If the distribution of the differences in scores between each pair of groups are all symmetrical, or the
distributions of values for each group have similar shape and spread:
Significant results can be reported as “There was a significant difference in median values
across groups.”
Post-hoc analysis allows you to say “The median for group A was higher than the median for
group B”, and so on.

If the above conditions are not met:

Significant results can be reported as “There was a significant difference in values among
groups.”

Other notes and alternative tests

The Friedman test is used for the same kinds of data and hypotheses, The Friedman test is described in
the previous chapter.

Another alternative is to use cumulative link models for ordinal data, which are described later in this
book.

Packages used in this chapter

The packages used in this chapter include:

• psych

243
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• FSA
• lattice
• multcompView
• PMCMR
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(PMCMR)){install.packages("PMCMR")}
if(!require(rcompanion)){install.packages("rcompanion")}

Quade test example

Input =("
Instructor Rater Likert
'Bob Belcher' a 4
'Bob Belcher' b 5
'Bob Belcher' c 4
'Bob Belcher' d 6
'Bob Belcher' e 6
'Bob Belcher' f 6
'Bob Belcher' g 10
'Bob Belcher' h 6
'Linda Belcher' a 8
'Linda Belcher' b 6
'Linda Belcher' c 8
'Linda Belcher' d 8
'Linda Belcher' e 8
'Linda Belcher' f 7
'Linda Belcher' g 10
'Linda Belcher' h 9
'Tina Belcher' a 7
'Tina Belcher' b 5
'Tina Belcher' c 7
'Tina Belcher' d 8
'Tina Belcher' e 8
'Tina Belcher' f 9
'Tina Belcher' g 10
'Tina Belcher' h 9
'Gene Belcher' a 6
'Gene Belcher' b 4
'Gene Belcher' c 5
'Gene Belcher' d 5
'Gene Belcher' e 6
'Gene Belcher' f 6
'Gene Belcher' g 5
'Gene Belcher' h 5
'Louise Belcher' a 8

244
QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Louise Belcher' b 7
'Louise Belcher' c 8
'Louise Belcher' d 8
'Louise Belcher' e 9
'Louise Belcher' f 9
'Louise Belcher' g 8
'Louise Belcher' h 10
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Instructor + Likert.f,
data = Data)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 2 1 4 0 0 0 1
Linda Belcher 0 0 1 1 4 1 1
Tina Belcher 0 1 0 2 2 2 1
Gene Belcher 1 4 3 0 0 0 0
Louise Belcher 0 0 0 1 4 2 1

XT = xtabs( ~ Instructor + Likert.f,

data = Data)

245
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

prop.table(XT,
margin = 1)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 0.250 0.125 0.500 0.000 0.000 0.000 0.125
Linda Belcher 0.000 0.000 0.125 0.125 0.500 0.125 0.125
Tina Belcher 0.000 0.125 0.000 0.250 0.250 0.250 0.125
Gene Belcher 0.125 0.500 0.375 0.000 0.000 0.000 0.000
Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

Bar plots by group

Note that the bar plots don’t show the effect of the blocking variable.
library(lattice)

histogram(~ Likert.f | Instructor,

data=Data,
layout=c(1,5) # columns and rows of individual plots
)

246
QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n mean sd min Q1 median Q3 max percZero

1 Bob Belcher 8 5.875 1.885 4 4.75 6 6.00 10 0
2 Linda Belcher 8 8.000 1.195 6 7.75 8 8.25 10 0
3 Tina Belcher 8 7.875 1.553 5 7.00 8 9.00 10 0
4 Gene Belcher 8 5.250 0.707 4 5.00 5 6.00 6 0
5 Louise Belcher 8 8.375 0.916 7 8.00 8 9.00 10 0

247
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Quade test example

This example uses the formula notation indicating that Likert is the dependent variable, Instructor is
the independent variable, and Rater is the blocking variable. The data= option indicates the data
frame that contains the variables. For the meaning of other options, see ?quade.test.
quade.test(Likert ~ Instructor | Rater,
data = Data)

Quade test

Quade F = 8.0253, num df = 4, denom df = 28, p-value = 0.0001924

Post-hoc test: pairwise two-sample paired rank-sum test for multiple comparisons
To prevent the inflation of type I error rates, adjustments to the p-values can be made using the
p.adjust.method option. See ?p.adjust for details on available p-value adjustment methods.

Note that the data must be ordered by the blocking variable so that the first observation for Bob will be
paired with the first observation for Linda, and so on.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels=c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

Data

### Pairwise Mann–Whitney

PT = pairwise.wilcox.test(Data$Likert,
Data$Instructor,
p.adjust.method="fdr",
paired=TRUE)$p.value
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

### Convert PT to a full table and call it PT1

library(rcompanion)

PT1 = fullPTable(PT)

PT1

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
Linda Belcher 1.00000000 0.61269120 0.85010674 0.05090808 0.04637053
Louise Belcher 0.61269120 1.00000000 0.47198578 0.05090808 0.04637053

248
 QUADE TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Tina Belcher 0.85010674 0.47198578 1.00000000 0.05090808 0.04637053

Bob Belcher 0.05090808 0.05090808 0.05090808 1.00000000 0.65530421
Gene Belcher 0.04637053 0.04637053 0.04637053 0.65530421 1.00000000

# Produce compact letter display

library(multcompView)

multcompLetters(PT1,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "a" "a" "ab" "b"

### Values sharing a letter are not significantly different

Post-hoc test: Quade post-hoc test

A different post-hoc test for the Quade test can be conducted with the posthoc.quade.test function in
the PMCMR package. The output can be converted to a compact letter display using the
multcompLetters function in the multcompView package.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels=c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

Data

### Post-hoc test

library(PMCMR)

PT = posthoc.quade.test(y = Data$Likert,
groups = Data$Instructor,
blocks = Data$Rater,
p.adjust.method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Pairwise comparisons using posthoc-Quade test with TDist approximation

Linda Belcher Louise Belcher Tina Belcher Bob Belcher

Louise Belcher 0.00902 - - -
Tina Belcher 0.04533 0.50547 - -

249
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bob Belcher 0.36572 0.00108 0.00539 -

Gene Belcher 0.00544 0.77620 0.38132 0.00099

P value adjustment method: fdr

### Compact letter display

PT0 = as.matrix(PT$p.value)

library(rcompanion)

PT1 = fullPTable(PT0)

library(multcompView)
multcompLetters(PT1,
compare="<",
threshold=0.05,
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "b" "b" "a" "b"

Nonparametric Regression
There are different techniques that are considered to be forms of nonparametric regression. Kendall–
Theil regression fits a linear model between one x variable and one y variable using a completely
nonparametric approach. Quantile regression is a very flexible approach that can find a linear
relationship between a dependent variable and one or more independent variables. Local regression
fits a smooth curve to the dependent variable, and can accommodate multiple independent variables.
Generalized additive models are a powerful and flexible approach.

Packages used in this chapter

The packages used in this chapter include:

• psych
• mblm
• quantreg
• rcompanion
• mgcv
• lmtest

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}

250
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(mblm)){install.packages("mblm")}
if(!require(quantreg)){install.packages("quantreg")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(mgcv)){install.packages("mgcv")}
if(!require(lmtest)){install.packages("lmtest")}

Nonparametric correlation

Nonparametric correlation is discussed in the chapter Correlation and Linear Regression.

Nonparametric regression examples

Data for the examples in this chapter are borrowed from the Correlation and Linear Regression
chapter. In this hypothetical example, students were surveyed for their weight, daily caloric intake,
daily sodium intake, and a score on an assessment of knowledge gain

Input = ("
Instructor Grade Weight Calories Sodium Score
'Brendon Small' 6 43 2069 1287 77
'Brendon Small' 6 41 1990 1164 76
'Brendon Small' 6 40 1975 1177 76
'Brendon Small' 6 44 2116 1262 84
'Brendon Small' 6 45 2161 1271 86
'Brendon Small' 6 44 2091 1222 87
'Brendon Small' 6 48 2236 1377 90
'Brendon Small' 6 47 2198 1288 78
'Brendon Small' 6 46 2190 1284 89
'Jason Penopolis' 7 45 2134 1262 76
'Jason Penopolis' 7 45 2128 1281 80
'Jason Penopolis' 7 46 2190 1305 84
'Jason Penopolis' 7 43 2070 1199 68
'Jason Penopolis' 7 48 2266 1368 85
'Jason Penopolis' 7 47 2216 1340 76
'Jason Penopolis' 7 47 2203 1273 69
'Jason Penopolis' 7 43 2040 1277 86
'Jason Penopolis' 7 48 2248 1329 81
'Melissa Robins' 8 48 2265 1361 67
'Melissa Robins' 8 46 2184 1268 68
'Melissa Robins' 8 53 2441 1380 66
'Melissa Robins' 8 48 2234 1386 65
'Melissa Robins' 8 52 2403 1408 70
'Melissa Robins' 8 53 2438 1380 83
'Melissa Robins' 8 52 2360 1378 74
'Melissa Robins' 8 51 2344 1413 65
'Melissa Robins' 8 51 2351 1400 68
'Paula Small' 9 52 2390 1412 78
'Paula Small' 9 54 2470 1422 62
'Paula Small' 9 49 2280 1382 61
'Paula Small' 9 50 2308 1410 72
'Paula Small' 9 55 2505 1410 80
'Paula Small' 9 52 2409 1382 60
'Paula Small' 9 53 2431 1422 70

251
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Paula Small' 9 56 2523 1388 79

'Paula Small' 9 50 2315 1404 71
'Coach McGuirk' 10 52 2406 1420 68
'Coach McGuirk' 10 58 2699 1405 65
'Coach McGuirk' 10 57 2571 1400 64
'Coach McGuirk' 10 52 2394 1420 69
'Coach McGuirk' 10 55 2518 1379 70
'Coach McGuirk' 10 52 2379 1393 61
'Coach McGuirk' 10 59 2636 1417 70
'Coach McGuirk' 10 54 2465 1414 59
'Coach McGuirk' 10 54 2479 1383 61
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Kendall–Theil Sen Siegel nonparametric linear regression

Kendall–Theil regression is a completely nonparametric approach to linear regression where there is

one independent and one dependent variable. It is robust to outliers in the dependent variable. It
simply computes all the lines between each pair of points, and uses the median of the slopes of these
lines. This method is sometimes called Theil–Sen. A modified, and preferred, method is named after
Siegel.

The method yields a slope and intercept for the fit line, and a p-value for the slope can be determined
as well. Typically, no measure analogous to r-squared is reported.

The mblm function in the mblm package uses the Siegel method by default. The Theil–Sen procedure
can be chosen with the repeated=FALSE option. See library(mblm); ?mblm for more details.

252
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(mblm)

model.k = mblm(Calories ~ Sodium,

data=Data)

summary(model.k)

Coefficients:
Estimate MAD V value Pr(>|V|)
(Intercept) -208.5875 608.4540 230 0.000861 ***
Sodium 1.8562 0.4381 1035 5.68e-14 ***

### Values under Estimate are used to determine the fit line.
### MAD is the median absolute deviation, a robust measure of variability

Plot with statistics

plot(Calories ~ Sodium,
data = Data,
pch = 16)

abline(model.k,
col="blue",
lwd=2)

Pvalue = as.numeric(summary(model.k)$coefficients[2,4])
Intercept = as.numeric(summary(model.k)$coefficients[1,1])
Slope = as.numeric(summary(model.k)$coefficients[2,1])
R2 = NULL

t1 = paste0("p-value: ", signif(Pvalue, digits=3))

t2 = paste0("R-squared: ", "NULL")
t3 = paste0("Intercept: ", signif(Intercept, digits=3))
t4 = paste0("Slope: ", signif(Slope, digits=3))

text(1160, 2600, labels = t1, pos=4)

text(1160, 2500, labels = t2, pos=4)
text(1160, 2400, labels = t3, pos=4)
text(1160, 2300, labels = t4, pos=4)

253
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Quantile regression

While traditional linear regression models the conditional mean of the dependent variable, quantile
regression models the conditional median or other quantile. Medians are most common, but for
example, if the factors predicting the highest values of the dependent variable are to be investigated, a
95th percentile could be used. Likewise, models for several quantiles, e.g. 25th , 50th, 75th percentiles,
could be investigated simultaneously.

Quantile regression makes no assumptions about the distribution of the underlying data, and is robust
to outliers in the dependent variable. It does assume the dependent variable is continuous. However,
there are functions for other types of dependent variables in the qtools package. The model assumes
that the terms are linearly related. Quantile regression is sometimes considered “semiparametric”.

Quantile regression is very flexible in the number and types of independent variables that can be
added to the model. The example, here, however, confines itself to a simple case with one independent
variable and one dependent variable.

This example models the median of dependent variable, which is indicated with the tau = 0.5 option.

A p-value for the model can be found by using the anova function with the fit model and the null model.
A pseudo R-squared value can be found with the nagelkerke function in the rcompanion package.
library(quantreg)

model.q = rq(Calories ~ Sodium,

data = Data,
tau = 0.5)

summary(model.q)

tau: [1] 0.5

254
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Coefficients:
coefficients lower bd upper bd
(Intercept) -84.12409 -226.58102 134.91738
Sodium 1.76642 1.59035 1.89615

### Values under Coefficients are used to determine the fit line.
### bd appears to be a confidence interval for the coefficients

model.null = rq(Calories ~ 1,
data = Data,
tau = 0.5)

anova(model.q, model.null)

Quantile Regression Analysis of Deviance Table

Df Resid Df F value Pr(>F)

1 1 43 187.82 < 2.2e-16 ***

### p-value for model overall

library(rcompanion)

nagelkerke(model.q)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.115071
Cox and Snell (ML) 0.783920
Nagelkerke (Cragg and Uhler) 0.783921

Plot with statistics

plot(Calories ~ Sodium,
data = Data,
pch = 16)

abline(model,
col="blue",
lwd=2)

library(rcompanion)

Pvalue = anova(model.q, model.null)[[1]][1,4]

Intercept = as.numeric(summary(model.q)$coefficients[1,1])
Slope = as.numeric(summary(model.q)$coefficients[2,1])
R2 = nagelkerke(model.q)[[2]][3,1]

t1 = paste0("p-value: ", signif(Pvalue, digits=3))

t2 = paste0("R-squared: ", signif(R2, digits=3))
t3 = paste0("Intercept: ", signif(coefficients(model)[1], digits=3))

255
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

t4 = paste0("Slope: ", signif(coefficients(model)[2], digits=3))

text(1160, 2600, labels = t1, pos=4)

text(1160, 2500, labels = t2, pos=4)
text(1160, 2400, labels = t3, pos=4)
text(1160, 2300, labels = t4, pos=4)

Local regression

There are several techniques for local regression. The idea is to fit a curve to data by averaging, or
otherwise summarizing, data points that are next to one another. The amount of “wiggliness” of the
curve can be adjusted.

Local regression is useful for investigating the behavior of the response variable in more detail than
would be possible with a simple linear model.

The function loess in the native stats package can be used for one continuous dependent variable and
up to four independent variables. The process is essentially nonparametric, and is robust to outliers in
the dependent variable. Usually no p-value or r-squared are reported. Integer variables have to
coerced to numeric variables.

The plot below shows a basically linear response, but also shows an increase in Calories at the upper
end of Sodium.
Data$Sodium = as.numeric(Data$Sodium)

model.l = loess(Calories ~ Sodium,

data = Data,
span = 0.75, ### higher numbers for smoother fits
degree=2, ### use polynomials of order 2
family="gaussian") ### the default, use least squares to fit

256
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

summary(model.l)

Number of Observations: 45
Equivalent Number of Parameters: 4.19
Residual Standard Error: 91.97

Plot
library(rcompanion)

plotPredy(data = Data,
x = Sodium,
y = Calories,
model = model.l,
xlab = "Calories per day",
ylab = "Sodium intake per day")

Generalized additive models

Generalized additive models are very flexible, allowing for a variety of types of independent variables
and of dependent variables. A smoother function is often used to create a “wiggly” model analogous to
that used in local regression. The gam function in the mgcv package uses smooth functions plus a
conventional parametric component, and so would probably be classified as a semiparametric
approach. The summary function reports an R-squared value, and p-values for the terms. The anova
function can be used for one model, or to compare two models.
library(mgcv)

model.g = gam(Calories ~ s(Sodium),

data = Data,
family=gaussian())

257
 NONPARAMETRIC REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

summary(model.g)

Parametric coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2304.87 13.62 169.2 <2e-16 ***

Approximate significance of smooth terms:

edf Ref.df F p-value
s(Sodium) 1.347 1.613 66.65 4.09e-15 ***

R-sq.(adj) = 0.718 Deviance explained = 72.6%

GCV = 8811.5 Scale est. = 8352 n = 45

model.null = gam(Calories ~ 1,
data = Data,
family=gaussian())

anova(model.g,
model.null)

Analysis of Deviance Table

Model 1: Calories ~ s(Sodium)

Model 2: Calories ~ 1

Resid. Df Resid. Dev Df Deviance

1 42.387 356242
2 44.000 1301377 -1.6132 -945135

library(lmtest)

lrtest(model.g,
model.null)

Likelihood ratio test

Model 1: Calories ~ s(Sodium)

Model 2: Calories ~ 1

#Df LogLik Df Chisq Pr(>Chisq)

1 3.3466 -265.83
2 2.0000 -294.98 -1.3466 58.301 2.25e-14 ***

Plot with statistics

library(rcompanion)

plotPredy(data = Data,
x = Sodium,

258
 NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

y = Calories,
model = model.g,
xlab = "Calories per day",
ylab = "Sodium intake per day")

Pvalue = 2.25e-14
R2 = 0.718

t1 = paste0("p-value: ", signif(Pvalue, digits=3))

t2 = paste0("R-squared (adj.): ", signif(R2, digits=3))

text(1160, 2600, labels = t1, pos=4)

text(1160, 2500, labels = t2, pos=4)

### Note that the fit line is slightly curved.

Nonparametric Regression for Time Series

There are specific nonparametric techniques that are commonly used for time series data. The Mann–
Kendall trend test is commonly used to determine if a trend exists, and can handle seasonal patterns
within the data. The slope of the trend is often determined with Sen’s slope, which can also handle
seasonality in the data. Finally, Pettitt’s test is use to determine if there is a point at which the values
in the data change.

It is my understanding that the tests in this chapter do not take into consideration any autocorrelation
in the data. Autocorrelation would be for example if an observation with a high value would tend to be
followed by another observation with a high value. For the stage data in this chapter, since the stage of
a tidal river is affected by the phase of the moon and by the direction and strength of the wind, it
would be reasonable to assume that the average stage for one day would be correlated to the previous
day. Due to this limitation in these tests, there are two conditions under which these they are usually

259
 NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

used. 1) When the observations are far enough apart in time that autocorrelation is unlikely to be
important. For example, in the stage data in this chapter, monthly averages are used. 2) When the
values are likely to be influenced mostly by non-autocorrelated factors. For data where
autocorrelation is likely to be important, other models, such as autoregressive integrated moving
average (ARIMA), could be used.

Packages used in this chapter

The packages used in this chapter include:

• mice
• Kendall
• trend

The following commands will install these packages if they are not already installed:
if(!require(mice)){install.packages("mice")}
if(!require(Kendall)){install.packages("Kendall")}
if(!require(trend)){install.packages("trend")}

Nonparametric regression examples

The data used in this chapter is a times series of stage measurements of the tidal Cohansey River in
Greenwich, NJ. Stage is the height of the river, in this case given in feet, with an arbitrary 0 datum. The
data are from U.S. Geology Survey site 01413038, and are monthly averages.

The MannKendall and SeasonalMannKendall functions can handle missing data. However, the
sens.slope, sea.sens.slope, decompose, and stl functions, at the time of writing, cannot handle missing
data.

Because the data set has a few missing data points, we will impute those values with the mice function
in the mice package.

The function ts converts data into a time series object. The user needs to be careful using this function,
since it does not check for missing lines in the data. For example, if there is no observation for
February, the function will simply take the March the value for February and continue. However, one
could add an observation for February with a value of NA.
Greenwich = read.table("http://rcompanion.org/documents/Greenwich.csv",
header=TRUE, sep=",")

Greenwich[125:136,]

Agency Station Parameter TS_ID Year Month Stage

125 USGS 1413038 65 97255 2010 11 0.107
126 USGS 1413038 65 97255 2010 12 -0.297
127 USGS 1413038 65 97255 2011 1 -0.350
128 USGS 1413038 65 97255 2011 2 NA

260
NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

129 USGS 1413038 65 97255 2011 3 NA

130 USGS 1413038 65 97255 2011 4 NA
131 USGS 1413038 65 97255 2011 5 NA
132 USGS 1413038 65 97255 2011 6 NA
133 USGS 1413038 65 97255 2011 7 NA
134 USGS 1413038 65 97255 2011 8 0.347
135 USGS 1413038 65 97255 2011 9 0.738
136 USGS 1413038 65 97255 2011 10 0.499

library(mice)

Mousey = mice(Greenwich)

Greenwich = complete(Mousey)

Greenwich[125:136,]

Agency Station Parameter TS_ID Year Month Stage

125 USGS 1413038 65 97255 2010 11 0.107
126 USGS 1413038 65 97255 2010 12 -0.297
127 USGS 1413038 65 97255 2011 1 -0.350
128 USGS 1413038 65 97255 2011 2 -0.076
129 USGS 1413038 65 97255 2011 3 0.176
130 USGS 1413038 65 97255 2011 4 -0.314
131 USGS 1413038 65 97255 2011 5 0.051
132 USGS 1413038 65 97255 2011 6 0.035
133 USGS 1413038 65 97255 2011 7 0.347
134 USGS 1413038 65 97255 2011 8 0.347
135 USGS 1413038 65 97255 2011 9 0.738
136 USGS 1413038 65 97255 2011 10 0.499

TS = ts(Greenwich$Stage, frequency=12, start=c(2000,4))

TS

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2000 0.297 0.219 0.055 0.260 0.293 0.261 0.074 -0.013 -0.523
2001 -0.227 -0.465 -0.076 -0.008 0.183 0.157 0.206 0.198 0.412 0.037 -0.089 -0.263
2002 -0.513 -0.276 -0.522 -0.235 -0.120 0.172 0.135 0.253 0.362 0.448 -0.205 -0.276
2003 0.011 0.120 0.350 0.337 0.391 0.212 0.299 -0.132 -0.313 -0.534 -0.395 -0.310
2004 -0.419 -0.314 0.051 0.086 0.039 0.314 0.432 -0.099 -0.408 -0.267 -0.121 -0.140
2005 0.293 0.364 0.168 0.233 0.164 0.100 0.279 -0.375 -0.603 -0.427 -0.544 -0.532
2006 -0.122 0.164 -0.067 -0.106 0.115 0.446 0.034 0.160 -0.623 -0.692 -0.311 -0.731
2007 -0.199 -0.223 0.100 -0.063 0.113 -0.024 0.105 -0.284 -0.456 -0.591 -0.565 -0.381
2008 0.020 0.277 -0.037 -0.107 0.011 0.241 -0.038 -0.222 -0.806 -0.802 -0.654 -0.247
2009 -0.144 -0.119 0.510 0.274 0.097 0.412 0.418 0.350 -0.286 0.047 -0.323 0.393
2010 0.021 -0.049 0.141 0.035 0.318 0.263 0.023 0.107 -0.297 -0.350 0.051 0.289
2011 -0.063 0.164 0.314 0.037 0.347 0.738 0.499 0.087 -0.119 -0.355 -0.277 -0.025
2012 0.088 0.254 0.581 0.415 0.435 0.277 0.524 0.263 0.153 -0.290 -0.330 0.176
2013 0.054 0.023 0.289 0.341 0.392 0.378 0.536 -0.314 -0.265 -0.132 -0.408 -0.134
2014 0.158 0.300 0.401 0.172 0.456 0.492

plot(TS)

261
 NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Decomposing time series objects

It is helpful to decompose time series data into seasonal and trend components. The decompose
function in the native stats package uses “classical seasonal decomposition by moving averages”, and
the stl function in the native stats package uses “seasonal decomposition of time series by loess”.
plot(decompose(TS))

plot(stl(TS,
s.window="periodic"))

262
 NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Mann–Kendall trend test

The Mann–Kendall trend test is completely nonparametric. The MannKendall function in the Kendall
package can be used with a time series object. The SeasonalMannKendall function performs the test
while taking into account the seasonality of the data.
library(Kendall)

MK = MannKendall(TS)

summary(MK)

Score = 1371 , Var(Score) = 560382.3

denominator = 14524
tau = 0.0944, 2-sided pvalue =0.067233

### tau is a measure of the strength of the trend

### p-value for trend is also shown

library(Kendall)

SMK = SeasonalMannKendall(TS)

summary(SMK)

Score = 217 , Var(Score) = 4227

denominator = 1133.499
tau = 0.191, 2-sided pvalue =0.00084484

263
 NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### tau is a measure of the strength of the trend

### p-value for trend is also shown

Sen’s slope for time series data

The sens.slope function in the trend package is used with a time series object. The sea.sens.slope
function performs the test while taking into account the seasonality of the data.
library(trend)

sens.slope(TS)

Sen's slope and intercept

slope: 0.001
95 percent confidence intervall for slope
0.002 -1e-04

intercept: -0.0475
nr. of observations: 171

library(trend)

sea.sens.slope(TS)

Seasonal Sen's slope and intercept

slope: 0.0175

intercept: -1.6237
nr. of observations: 171

Pettitt’s test for change in value

The pettitt.test function in the trend package identifies a point at which the values in the data change.
The results here suggest that the stage values were higher after May 2009 than before.
library(trend)

pettitt.test(TS)

Pettitt's test for single change-point detection

K = 2453, p-value = 0.001526

alternative hypothesis: true change point is present in the series

sample estimates:
probable change point at tau
107

264
NONPARAMETRIC REGRESSION FOR TIME SERIES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Greenwich[107,]

Agency Station Parameter TS_ID Year Month Stage

107 USGS 1413038 65 97255 2009 5 -0.119

265
 INTRODUCTION TO PERMUTATION TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Permutation Tests for Ordinal Data

Introduction to Permutation Tests

Permutation tests are increasingly common tests to perform certain types of statistical analyses. They
do not rely on assumptions about the distribution of the data, as some other tests do. They are
therefore considered to be nonparametric tests. It is my understanding, however, that for certain
tests—for example those testing a difference in means—that there are assumptions about the
underlying data.

Permutation tests work by resampling the observed data many times in order to determine a p-value
for the test. Recall that the p-value is defined as the probability of getting data as extreme as the
observed data when the null hypothesis is true. If the data are shuffled many times in accordance with
the null hypothesis being true, the number of cases with data as extreme as the observed data could be
counted, and a p-value calculated.

The advantages of permutation tests are

• the lack of assumptions about the distribution of the underlying data,
• their flexibility in the kinds of data they can handle (nominal, ordinal, interval/ratio),
• and their being relatively straightforward to conduct and interpret.

The disadvantages of permutation tests are

• the limited complexity of designs they can handle (for the coin package at the time of
writing),
• the unfamiliarity of them for many readers.

The coin package

Permutation tests in this book will use the coin package, with either of two functions,
independence_test and symmetry_test. This book with use permutation tests with ordinal dependent
variables, but the coin package is able to handle nominal, ordinal, and interval/ratio data.

A few notes on using permutation tests:

• If the dependent variable is to be treated as an ordinal variable, it must be coded as an
ordered factor variable in R. It does not need to have numerals for levels. For example it
could have levels doctorate > masters > bachelors > associates > high.school. But also it
could have the levels 5 > 4 > 3 > 2 > 1.

• The general interpretation for significant results of these models isn’t that there is a
difference among medians, but that there is a significant effect of the independent variable on
the dependent variable, or that there is a significant difference among groups.

• Post-hoc tests for factors or groups can be conducted with pairwise tests of groups, or with
pairwise ordinal tests for paired data. The appropriate functions in the rcompanion package
are pairwisePermutationTest, pairwisePermutationMatrix, pairwisePermutationSymmetry, and
pairwisePermutationSymmetryMatrix.

266
 INTRODUCTION TO PERMUTATION TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Permutation tests for data arranged in contingency tables are presented in the Association
Tests for Ordinal Tables chapter.

Packages used in this chapter

The packages used in this chapter include:

• coin

The following commands will install these packages if they are not already installed:
if(!require(coin)){install.packages("coin")}

Permutation test example

The following example uses the left hand and right hand data from the Independent and Paired Values
chapter. For this example, we are interested in comparing the length of left hands and rights from 16
individuals. First we will compare the left hands to right hands as independent samples (analogous to
a Mann–Whitney test or t-test), then as paired values for each individual (analogous to a paired rank
sum test or paired t-test).

Input = ("
Individual Hand Length
A Left 17.5
B Left 18.4
C Left 16.2
D Left 14.5
E Left 13.5
F Left 18.9
G Left 19.5
H Left 21.1
I Left 17.8
J Left 16.8
K Left 18.4
L Left 17.3
M Left 18.9
N Left 16.4
O Left 17.5
P Left 15.0
A Right 17.6
B Right 18.5
C Right 15.9
D Right 14.9
E Right 13.7
F Right 18.9
G Right 19.5
H Right 21.5
I Right 18.5
J Right 17.1
K Right 18.9
L Right 17.5
M Right 19.5

267
INTRODUCTION TO PERMUTATION TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

N Right 16.5
O Right 17.4
P Right 15.6
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

Data

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

### Summarize data

library(FSA)

Summarize(Length ~ Hand,
data=Data,
digits=3)

Hand n nvalid mean sd min Q1 median Q3 max percZero

1 Left 16 16 17.356 1.948 13.5 16.35 17.50 18.52 21.1 0
2 Right 16 16 17.594 1.972 13.7 16.35 17.55 18.90 21.5 0

Box plot
boxplot(Length ~ Hand,
data=Data,
ylab="Length, cm")

268
 INTRODUCTION TO PERMUTATION TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Scatter plot with one-to-one line

In the plot below, each point represents a pair of paired values. Points that fall above and to the left of
the blue line indicate cases for which the value for Right was greater than for Left.
Left = Data$Length[Data$Hand=="Left"]

Right = Data$Length[Data$Hand=="Right"]

plot(Left, Right,
pch = 16, # shape of points
cex = 1.0, # size of points
xlim=c(13, 22), # limits of x axis
ylim=c(13, 22), # limits of y axis
xlab="Left hand",
ylab="Right hand")

abline(0,1, col="blue", lwd=2) # line with intercept of 0 and slope of 1

Permutation test of independence

This tests treats the two groups (left hand and right hand) as independent samples, and tests if there is
a difference in values between the two groups.
library(coin)

independence_test(Length ~ Hand,
data = Data)

Asymptotic General Independence Test

Z = -0.34768, p-value = 0.7281

269
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

Permutation test of symmetry

This tests treats the two groups (left hand and right hand) as having paired or repeated data, paired
within Individual. Practically, it tests if there is a difference in values between the two groups.
library(coin)

symmetry_test(Length ~ Hand | Individual,

data = Data)

Asymptotic General Symmetry Test

Z = -2.6348, p-value = 0.008418

References

For more information on permutation tests and the coin package, see:

Hothorn, T., K. Hornik, M.A. van de Wiel, and A. Zeileis. 2015. Implementing a Class of
Permutation Tests: The coin Package. cran.r-
project.org/web/packages/coin/vignettes/coin_implementation.pdf.

library(coin); help(package="coin")

One-way Permutation Test of Independence for

Ordinal Data
When to use this test

A permutation test of independence can be used for one-way data with an ordinal dependent variable.
It will determine if there is a difference in the response variable among groups. There can be two or
more groups.

This test is analogous to designs used with Mann–Whitney, Kruskal–Wallis, two-sample t-test, one-
way anova, one-way anova with blocks, and ordinal regression equivalents.

The test assumes that the observations are independent. That is, it is not appropriate for paired
observations or repeated measures data. It does not make assumptions about the distribution of
values.

The test is performed with the independence_test function in the coin package.

Post-hoc testing can be conducted with pairwise permutation tests across groups.

270
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

It is important that the dependent variable be specified as an ordered factor variable in R.

Appropriate data
• One-way data. That is, one measurement variable in two or more groups
• Dependent variable is ordinal, and specified in R as an ordered factor variable
• Independent variable is a factor with two or more levels. That is, two or more groups
• Observations between groups are independent. That is, not paired or repeated measures
data

Hypotheses
• Null hypothesis: The values of the dependent variable among groups are equal.
• Alternative hypothesis (two-sided): The values of the dependent variable among groups are
not equal.

Interpretation
• Reporting significant results for the omnibus test as “Significant differences were found
among values for groups.” is acceptable. Alternatively, “A significant effect for Independent
Variable on Dependent Variable was found.”
• Reporting significant results for mean separation post-hoc tests as “Value of Dependent
Variable for group A was different than that for group B.” is acceptable.

Other notes and alternative tests

Ordinal regression is an alternative.

As are the traditional nonparametric tests Mann–Whitney U or Kruskal–Wallis are alternatives in

cases where there is not a blocking variable.

Packages used in this chapter

The packages used in this chapter include:

• psych
• lattice
• FSA
• coin
• rcompanion
• multcompView
• ggplot2

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(coin)){install.packages("coin")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(ggplot2)){install.packages("ggplot2")}

271
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

One-way ordinal permutation test example

This example re-visits the Pooh, Piglet, and Tigger data from the Descriptive Statistics with the likert
Package chapter.

It answers the question, “Are the scores significantly different among the three speakers?”

Note that a new variable, Likert.f, is created for the Likert scores as an ordered factor variable. It is
necessary that the dependent variable passed to independence_test be an ordered factor variable for it
to be treated as an ordinal variable.
Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2
Piglet 3
Tigger 4
Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

272
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
cross tabulated over values of Speaker. The prop.table function translates a table into proportions.
The margin=1 option indicates that the proportions are calculated for each row.
xtabs( ~ Speaker + Likert.f,
data = Data)

Likert.f
Speaker 1 2 3 4 5
Pooh 0 0 1 6 3
Piglet 1 6 2 1 0
Tigger 0 0 2 6 2

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5
Pooh 0.0 0.0 0.1 0.6 0.3
Piglet 0.1 0.6 0.2 0.1 0.0
Tigger 0.0 0.0 0.2 0.6 0.2

273
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

Bar plots by group

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
presented over values of Speaker.
library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,3)) # columns and rows of individual plots

Summarize data treating Likert scores as numeric

It may be useful to look at the minimum, first quartile, median, third quartile, and maximum for Likert
for each group.
library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n mean sd min Q1 median Q3 max percZero

1 Pooh 10 4.2 0.632 3 4 4 4.75 5 0
2 Piglet 10 2.3 0.823 1 2 2 2.75 4 0
3 Tigger 10 4.0 0.667 3 4 4 4.00 5 0

274
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

One-way ordinal permutation test

Note that the dependent variable is an ordered factor variable, Likert.f. Speaker is the independent
variable. The data= option indicates the data frame that contains the variables. For the meaning of
other options, see library(coin); ?independence_test. There is the option of adding a blocking variable to
the formula with e.g. | Block.
library(coin)

independence_test(Likert.f ~ Speaker,
data = Data)

Asymptotic General Independence Test

maxT = 4.2006, p-value = 7.685e-05

Post-hoc test: pairwise permutation tests

If the independence test is significant, a post-hoc analysis can be performed to determine which
groups differ from which other groups.

The pairwisePermutationTest and pairwisePermutationMatrix functions in the rcompanion package

conduct permutation tests across groups in a pairwise manner. See library(rcompanion);
?pairwisePermutationTest for further details.

Because the post-hoc test will produce multiple p-values, adjustments to the p-values can be made to
avoid inflating the possibility of making a type-I error. Here, the method of adjustment is indicated
with the method option. There are a variety of methods for controlling the familywise error rate or for
controlling the false discovery rate. See ?p.adjust for details on these methods.

Before conducting the pairwise tests, we will re-order the levels of the grouping variable by the
median of each group. This makes interpretation of the pairwise comparisons and compact letter
display easier.

Table output and compact letter display

### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Pairwise permutation tests

library(rcompanion)

PT = pairwisePermutationTest(x = Data$Likert.f,
g = Data$Speaker,
method = "fdr")

PT

Comparison Stat p.value p.adjust

275
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

1 Pooh - Tigger = 0 0.698 0.4852 0.485200

2 Pooh - Piglet = 0 3.515 0.000439 0.001238
3 Tigger - Piglet = 0 -3.344 0.0008254 0.001238

### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 Pooh a a
2 Tigger a a
3 Piglet b b

Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix output and compact letter display

A compact letter display condenses a table of p-values into a simpler format. In the output, groups
sharing a same letter are not significantly different. Compact letter displays are a clear and succinct
way to present results of multiple comparisons.

Here the fdr p-value adjustment method is used. See ?p.adjust for details on available methods.

The code creates a matrix of p-values called PM, which is then passed to the multcompLetters function
to be converted to a compact letter display.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Conduct pairwise permutation tests

library(rcompanion)

PM = pairwisePermutationMatrix(x = Data$Likert,
g = Data$Speaker,
exact = NULL,
method = "fdr")$Adjusted
PM

### Produce compact letter display

library(multcompView)

multcompLetters(PM,

276
 ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Pooh Tigger Piglet

"a" "a" "b"

Groups sharing a letter are not significantly different (alpha = 0.05).

Plot of medians and confidence intervals

The following code uses the groupwiseMedian function to produce a data frame of medians for each
speaker along with the 95% confidence intervals for each median with the percentile method. See
library(rcompanion); ?groupwiseMedian for further options.

These medians are then plotted, with their confidence intervals shown as error bars. The grouping
letters from the multiple comparisons are added.
### Order groups by original order

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Piglet", "Tigger"))

### Create data frame of medians and confidence intervals

library(rcompanion)

Sum = groupwiseMedian(data=Data,
group="Speaker",
var="Likert",
conf=0.95,
R=5000,
percentile=TRUE,
bca=FALSE,
digits=3)

Sum

### Prepare labels

X = 1:3

Y = Sum$Percentile.upper + 0.2

Label = c("a", "b", "a")

### Plot

library(ggplot2)

277
ONE-WAY PERMUTATION TEST OF INDEPENDENCE FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

ggplot(Sum, ### The data frame to use.

aes(x = Speaker,
y = Median)) +
geom_errorbar(aes(ymin = Percentile.lower,
ymax = Percentile.upper),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Median Likert score")+

annotate("text",
x = X,
y = Y,
label = Label)

Plot of median Likert versus Speaker. Error bars indicate the 95% confidence
intervals for the median with the percentile method.

Comparison of methods for independence tests for ordinal data

Data Test p-value Mean separation
Pooh Piglet Permutation 0.0004
Pooh Piglet Ordinal regression 0.00003
Pooh Piglet Mann–Whitney 0.0005

Pooh Tigger Piglet Permutation 0.00008 a a b

Pooh Tigger Piglet Ordinal regression 0.000008 a a b
Pooh Tigger Piglet Kruskal–Wallis 0.0002 a a b

278
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

One-way Permutation Test of Symmetry for

Paired Ordinal Data
When to use this test

A permutation test of symmetry can be used for one-way data with an ordinal dependent variable
where observations are paired within a blocking variable. It will determine if there is a difference in
the response variable among groups when controlling for the effect of the blocking variable. There can
be two or more groups.

This test is analogous to designs used with Friedman, Quade, paired t-test, repeated measures one-
way anova, and ordinal regression equivalents.

The test does not make assumptions about the distribution of values.

The test is performed with the symmetry_test function in the coin package.

Post-hoc testing can be conducted with pairwise permutation tests across groups.

It is important that the dependent variable be specified as an ordered factor variable in R.

Appropriate data
• One-way data plus a blocking variable. That is, one measurement variable in two or more
groups, where observations are paired within levels of a blocking variable
• Dependent variable is ordinal, and specified in R as an ordered factor variable
• Independent variable is a factor with two or more levels. That is, two or more groups. The
blocking variable is also a factor variable
• The data is arranged in a complete block design, with one or more observations per cell

Hypotheses
• Null hypothesis: The response of the dependent variable among groups are equal.
• Alternative hypothesis (two-sided): The response of the dependent variable among groups
are not equal.

Interpretation
• Reporting significant results for the omnibus test as “Significant differences were found in the
response among groups.” is acceptable. Alternatively, “A significant effect for Independent
Variable on Dependent Variable was found when controlling for the effect of Blocking
Variable.”
• Reporting significant results for mean separation post-hoc tests as “Response of Dependent
Variable for group A was different than that for group B.” is acceptable.

279
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

Other notes and alternative tests

Ordinal regression is an alternative.

The traditional nonparametric tests Friedman, Quade, Paired rank-sum test, or Sign test may be
alternatives depending on the design of the experiment.

Packages used in this chapter

The packages used in this chapter include:

• psych
• lattice
• FSA
• coin
• rcompanion
• multcompView
• ggplot2

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(coin)){install.packages("coin")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(ggplot2)){install.packages("ggplot2")}

One-way ordinal permutation test of symmetry example

This example re-visits the Belcher data from the Friedman Test chapter. Note that each instructor is
rated by each of eight raters. Because of this, we want to stratify the responses by Rater.

It answers the question, “Are the scores significantly different among the five speakers when
controlling for the effect of the different raters?”

Note that a new variable, Likert.f, is created for the Likert scores as an ordered factor variable. It is
necessary that the dependent variable passed to symmetry_test be an ordered factor variable for it to
be treated as an ordinal variable.
Input =("
Instructor Rater Likert
'Bob Belcher' a 4
'Bob Belcher' b 5
'Bob Belcher' c 4
'Bob Belcher' d 6
'Bob Belcher' e 6
'Bob Belcher' f 6
'Bob Belcher' g 10

280
ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

'Bob Belcher' h 6
'Linda Belcher' a 8
'Linda Belcher' b 6
'Linda Belcher' c 8
'Linda Belcher' d 8
'Linda Belcher' e 8
'Linda Belcher' f 7
'Linda Belcher' g 10
'Linda Belcher' h 9
'Tina Belcher' a 7
'Tina Belcher' b 5
'Tina Belcher' c 7
'Tina Belcher' d 8
'Tina Belcher' e 8
'Tina Belcher' f 9
'Tina Belcher' g 10
'Tina Belcher' h 9
'Gene Belcher' a 6
'Gene Belcher' b 4
'Gene Belcher' c 5
'Gene Belcher' d 5
'Gene Belcher' e 6
'Gene Belcher' f 6
'Gene Belcher' g 5
'Gene Belcher' h 5
'Louise Belcher' a 8
'Louise Belcher' b 7
'Louise Belcher' c 8
'Louise Belcher' d 8
'Louise Belcher' e 9
'Louise Belcher' f 9
'Louise Belcher' g 8
'Louise Belcher' h 10
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

281
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
cross tabulated over values of Instructor The prop.table function translates a table into proportions.
The margin=1 option indicates that the proportions are calculated for each row.

xtabs( ~ Instructor + Likert.f,

data = Data)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 2 1 4 0 0 0 1
Linda Belcher 0 0 1 1 4 1 1
Tina Belcher 0 1 0 2 2 2 1
Gene Belcher 1 4 3 0 0 0 0
Louise Belcher 0 0 0 1 4 2 1

XT = xtabs( ~ Instructor + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 0.250 0.125 0.500 0.000 0.000 0.000 0.125
Linda Belcher 0.000 0.000 0.125 0.125 0.500 0.125 0.125
Tina Belcher 0.000 0.125 0.000 0.250 0.250 0.250 0.125
Gene Belcher 0.125 0.500 0.375 0.000 0.000 0.000 0.000
Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

Bar plots by group

Note that the variable we want to count is Likert.f, which is a factor variable. Counts for Likert.f are
presented over values of Speaker. Also note that the bar plots don’t show the effect of the blocking
variable.
library(lattice)

histogram(~ Likert.f | Instructor,

data=Data,
layout=c(1,5) # columns and rows of individual plots
)

282
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

Summarize data treating Likert scores as numeric

It may be useful to look at the minimum, first quartile, median, third quartile, and maximum for Likert
for each group.
library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n mean sd min Q1 median Q3 max percZero

1 Bob Belcher 8 5.875 1.885 4 4.75 6 6.00 10 0
2 Linda Belcher 8 8.000 1.195 6 7.75 8 8.25 10 0
3 Tina Belcher 8 7.875 1.553 5 7.00 8 9.00 10 0
4 Gene Belcher 8 5.250 0.707 4 5.00 5 6.00 6 0
5 Louise Belcher 8 8.375 0.916 7 8.00 8 9.00 10 0

Permutation symmetry test

Note that the dependent variable is an ordered factor variable, Likert.f. Instructor is the independent
variable, and Rater is the blocking variable. The data= option indicates the data frame that contains
the variables. For the meaning of other options, see library(coin); ?symmetry_test.

283
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

library(coin)

symmetry_test(Likert.f ~ Instructor | Rater,

data = Data)

Asymptotic General Symmetry Test

maxT = 3.4036, p-value = 0.003416

Post-hoc test: pairwise permutation tests

If the symmetry test is significant, a post-hoc analysis can be performed to determine which groups
differ from which other groups.

The pairwisePermutationSymmetry and pairwisePermutationSymmetryMatrix functions in the

rcompanion package conduct permutation tests across groups in a pairwise manner. See
library(rcompanion); ?pairwisePermutationSymmetry for further details.

Because the post-hoc test will produce multiple p-values, adjustments to the p-values can be made to
avoid inflating the possibility of making a type-I error. Here, the method of adjustment is indicated
with the method option. There are a variety of methods for controlling the familywise error rate or for
controlling the false discovery rate. See ?p.adjust for details on these methods.

Before conducting the pairwise tests, we will re-order the levels of the grouping variable by the
median of each group. This makes interpretation of the pairwise comparisons and compact letter
display easier.

Table output and compact letter display

### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise permutation tests

library(rcompanion)

PT = pairwisePermutationSymmetry(x = Data$Likert.f,
g = Data$Instructor,
b = Data$Rater,
method = "fdr")

PT

Comparison W p.value p.adjust

1 Linda Belcher - Louise Belcher = 0 -0.9045 0.3657 0.42680

284
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

2 Linda Belcher - Tina Belcher = 0 0.378 0.7055 0.70550

3 Linda Belcher - Bob Belcher = 0 -2.38 0.01729 0.03458
4 Linda Belcher - Gene Belcher = 0 2.593 0.009522 0.03458
5 Louise Belcher - Tina Belcher = 0 -1.155 0.2482 0.35460
6 Louise Belcher - Bob Belcher = 0 -2.265 0.02354 0.03923
7 Louise Belcher - Gene Belcher = 0 -2.744 0.006068 0.03458
8 Tina Belcher - Bob Belcher = 0 -2.412 0.01586 0.03458
9 Tina Belcher - Gene Belcher = 0 2.528 0.01147 0.03458
10 Bob Belcher - Gene Belcher = 0 0.8704 0.3841 0.42680

### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 LindaBelcher a a
2 LouiseBelcher a a
3 TinaBelcher a a
4 BobBelcher b b
5 GeneBelcher b b

Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix output and compact letter display

A compact letter display condenses a table of p-values into a simpler format. In the output, groups
sharing a same letter are not significantly different. Compact letter displays are a clear and succinct
way to present results of multiple comparisons.

Here the fdr p-value adjustment method is used. See ?p.adjust for details on available methods.

The code creates a matrix of p-values called PM which is then passed to the multcompLetters function
to be converted to a compact letter display.
### Order groups by median

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise permutation tests

library(rcompanion)

PM = pairwisePermutationSymmetryMatrix(x = Data$Likert.f,
g = Data$Instructor,

285
 ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

b = Data$Rater,
method = "fdr")

PM

# Produce compact letter display

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "a" "a" "b" "b"

Groups sharing a letter are not significantly different (alpha = 0.05).

Plot of medians and confidence intervals

The following code uses the groupwiseMedian function to produce a data frame of medians for each
speaker along with the 95% confidence intervals for each median with the percentile method. See
library(rcompanion); ?groupwiseMedian for further options.

These medians are then plotted, with their confidence intervals shown as error bars. The grouping
letters from the multiple comparisons are added.
### Order groups by original order

Data$Instructor = factor(Data$Instructor,
levels = c("Bob Belcher", "Linda Belcher", "Tina Belcher",
"Gene Belcher", "Louise Belcher"))

### Create data frame of medians and confidence intervals

library(rcompanion)

Sum = groupwiseMedian(data=Data,
group="Instructor",
var="Likert",
conf=0.95,
R=5000,
percentile=TRUE,
bca=FALSE,
digits=3)

Sum

286
ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

### Prepare labels

X = 1:5

Y = Sum$Percentile.upper + 0.4

Label = c("b", "a", "a", "b", "a")

### Plot

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Instructor,
y = Median)) +
geom_errorbar(aes(ymin = Percentile.lower,
ymax = Percentile.upper),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Median Likert score")+

annotate("text",
x = X,
y = Y,
label = Label)

287
ONE-WAY PERMUTATION TEST OF SYMMETRY FOR PAIRED ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

Plot of median Likert versus Instructor. Error bars indicate the 95% confidence
intervals for the median with the percentile method.

Comparison of methods for symmetry tests for ordinal data

Data Test p-value Mean separation
Pooh Time Permutation symmetry 0.19
Pooh Time Ordinal regression 0.002
Pooh Time Two-sample paired rank-sum 0.02

Bob Belcher et al. Permutation symmetry 0.003 a a a b b

Bob Belcher et al. Ordinal regression 0.00000009 a a a b b
Bob Belcher et al. Friedman (Conover) 0.0001 a a a b b
Bob Belcher et al. Quade (pairwise) 0.0002 a a a ab b

288
 INTRODUCTION TO LINEAR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Concepts for Linear Models

Introduction to Linear Models

Linear models are used for a wide variety of statistical analyses. The basic concept is that a dependent
variable can be predicted from a set of independent variables that are related in a linear fashion. This
framework leads to models that are very flexible, and can include a variety of continuous or
categorical independent variables.

All linear models make some assumptions about the underlying data. For one thing, the data should
accord with the fact the model is composed of a linear combination of the effects. Non-linearity would
suggest that the model effects be modified or that a different kind of model be used. In addition, each
type of linear model usually makes some assumptions about the distribution of the data. For the
reported statistics to be valid, it is essential to understand and check these assumptions for the
specific type of model being used.

Types of linear models

In general, the type of model to be used is determined by the nature of the dependent variable.

General linear models

Readers may be familiar with linear regression, multiple linear regression, or analysis of variance
(ANOVA).

These models can be considered part of larger category of linear models called general linear model.
The dependent variable is continuous interval/ratio, and there are assumptions about the distribution
of the data. These models are usually called parametric models or tests, although technically other
types of models which assume properties about the distribution of the data are also parametric in
nature.

Generalized linear models

Models for other types of dependent variables can be developed in a generalized linear model
framework. This approach is similar to general linear model approach, except that there are different
assumptions about the distribution of the data.

For example, ordinal dependent variables can be modeled with cumulative link models. Binary
(yes/no) dependent variables can be modeled with logistic regression. Dependent variables of
discrete counted quantities can be modeled with Poisson regression and similar techniques. Percent
and proportion data can be modeled with beta regression.

Other linear models

Not all linear models are included in the general linear model and generalized linear model categories.
For example, common quantile regression is a type of linear model not included in these categories.

Fitting models
For any type of linear model, some method is used to find the value for the parameters for the model
which best fit the data. For simple models like linear regression or a simple analysis of variance, these

289
 INTRODUCTION TO LINEAR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

parameters estimates could be found by hand calculations. Luckily, more complex models can be fit by
computer algorithm. General linear models are typically fit by ordinary least squares (OLS), whereas
generalized linear models are typically fit by maximum likelihood estimation (MLE). OLS is actually a
specific case of MLE that is valid only when the conditional distribution of the data are normal. These
approaches to estimation are generally reliable, but the reader should be aware that there are cases
where MLE may fail.

Formulae for specifying models in R

Most packages for specifying types of models in R use a similar grammar in the model formula.

The formula
y ~ x1 + x2 + x1:x2

Specifies y as the dependent variables, and x1, x2, and the interaction of x1 and x2 as the independent
variables.

The formula

y ~ x1 | g

Specifies g as a stratification variable or as a conditional variable, that is x1 given g.

The symbol 1 specifies an intercept for the model, so that

y ~ x1 - 1

indicates that the intercept should not be included in the model, and

y ~ 1

indicates that only the intercept is to be included in the right side of this model.

The King article in the “References” section has some more detail on model specification syntax in R.

Data in data frames

Usually the variables in the model will be taken from a data frame. For example

y ~ x

will look for y and x in the global environment, whereas

y ~ x, data = Data

will look for y and x in the data frame called Data.

290
 INTRODUCTION TO LINEAR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Random effects
The syntax varies somewhat for random effects in models across packages, but

y ~ x, random = ~1|Subject

or

y ~ x +(1|Subject)

Specifies that y is the dependent variable, x is an independent variable, and Subject is an independent
variable treated as random variable, specifically with an intercept fit for each level of Subject.

More on specifying random effects is discussed in the chapter Repeated Measures ANOVA.

Extracting model information from R

Each package varies on the methods used to extract information about the model, but some are
relatively common across several packages. For other types of model objects, there may be methods
to extract similar information with different functions.

Assuming that model has been defined as a model object by an appropriate function,

summary(model)

produces a summary of the model with estimates of the coefficients, and sometimes other useful
information like a p-value for the model, or an r-squared or pseudo R-squared for the model.

library(car)

Anova(model)

for some models, will produce an analysis of variance table, or an analysis of deviance table.

plot(model)

will produce plots of the model, usually diagnostic plots.

anova(model1, model2)

will compare two models by an analysis of variance.

predict(model)

will report predicted values in the dependent variable from the model for each observation that went
in to the model.

residuals(model)

291
 USING RANDOM EFFECTS IN MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

will report residual values from the model.

str(model)

reports the structure of the information stored in the model object.

References

King, W.B. Model Formulae. ww2.coastal.edu/kingw/statistics/R-tutorials/formulae.html.

Using Random Effects in Models

This book will not investigate the concept of random effects in models in any substantial depth. The
goal of this chapter is to empower the reader to include random effects in models in cases of paired
data or repeated measures.

Random effects in models for paired and repeated measures

As an example, if we are measuring the left hand and right of several individuals, the measurements
are paired within each individual. That is, we want to statistically match the left hand of Individual A to
the right hand of Individual A, since we suppose that someone with a large left hand will have a large
right hand. Therefore, the variable Individual will be included in the model as a random variable. Each
Individual could be thought of has a block including a measurement for the left hand and a
measurement for the right hand.

As a second example, imagine we are measuring the scores of instructors multiple times, and we can
match each rater’s score at each time to the same rater’s score at the other times. So, we want to
statistically match the score of Rater A at Time 1 to the score of Rater A at Time 2, since we suppose
that someone who scores an instructor high at one time might score an instructor high at another
time. Therefore, the variable Rater will be included in the model as a random variable. Each Rater
could be thought of as a block including a measurement for each of the times.

The concept of random variables

Getting a firm grasp of the concept of random variables is not an easy task, so it is okay to not fret too
much about completely understanding the concept at this point. It is also helpful to keep in mind that
whether a variable should be considered a fixed or random variable will depend on the interpretation
of the person doing the analysis.

In the previous chapter, the variable Speaker was used as a fixed effect in the model. Conceptually, the
idea is that we are interested in the effect of each of the specific levels of that variable. That is, we care
specifically about Pooh and Piglet and their scores.

292
 USING RANDOM EFFECTS IN MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In many cases of blocking variables, though, we don’t necessarily care about the values for those
specific blocks.

For example, if we conducted a study in two different schools focusing on two different curricula, we
may want to know about the effect of the curricula, but more or less chose the schools at random. We
don’t really care if Springfield Elementary had a higher score than Shelbyville Elementary. The two
schools represent any two random schools, not those two specific schools. But we definitely want to
include the School effect in the model, because we suppose that one school could do better with both
curricula than would the other school.

Another example of this is when we have instructors rated by student raters. If we are studying the
performance of the instructors, we don’t necessarily care about how Nelson or Milhouse or Ralph as
individuals rate instructors. They are representing students chosen at random. We just need to
include the effect of the variable Rater in the model to statistically account for the fact that each rater
might tend to rate all instructors lower or higher than other raters.

In these examples, School and Rater could be included in their respective models as random effects.

Mixed effects models

When a model includes both fixed effects and random effects, it is called a mixed effects model.

Optional technical note: Random effects in more complex models

For more complex models, specifying random effects can become difficult. Random effects can be
crossed with one another or can be nested within one another. Also, correlation structures for the
random effects can be specified. However, approaching this complexity is beyond the scope of this
book.

The examples in this book treat random variables as simple intercept-only variables, as simple blocks
that are not nested within other variables.

Packages used in this chapter

The packages used in this chapter include:

• FSA
• psych
• lme4
• lmerTest
• nlme
• car

The following commands will install these packages if they are not already installed:
if(!require(FSA)){install.packages("FSA")}
if(!require(psych)){install.packages("psych")}
if(!require(lme4)){install.packages("lme4")}
if(!require(lmerTest)){install.packages("lmerTest")}
if(!require(nlme)){install.packages("nlme")}
if(!require(car)){install.packages("car")}

293
 USING RANDOM EFFECTS IN MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

An example of a mixed model

For this example we will revisit the hand size example from the Independent and Paired Values
chapter. The variable hand Length will be treated as an interval/ratio variable. We will return to
using cumulative link models for Likert data in subsequent chapters.

Note that the model includes the blocking variable, Individual, and so data do not need to be in a
certain order to match the paired observations.

Also note that mixed models may make certain assumptions about the distributions of the data. For
simplicity, this example will not check if those assumption are met.

By default, an analysis of variance for a mixed model doesn’t test the significance of the random effects
in the model. However, the effect of random terms can be tested by comparing the model to a model
including only the fixed effects and excluding the random effects, or with the rand function from the
lmerTest package if the lme4 package is used to specify the model.

As a technical note, the lmerTest package has options to use Satterthwaite or Kenward–Roger degrees
of freedom, and options for type-III or type-II tests in the analysis of variance, if the lme4 package is
used to specify the model.
Input = ("
Individual Hand Length
A Left 17.5
B Left 18.4
C Left 16.2
D Left 14.5
E Left 13.5
F Left 18.9
G Left 19.5
H Left 21.1
I Left 17.8
J Left 16.8
K Left 18.4
L Left 17.3
M Left 18.9
N Left 16.4
O Left 17.5
P Left 15.0
A Right 17.6
B Right 18.5
C Right 15.9
D Right 14.9
E Right 13.7
F Right 18.9
G Right 19.5
H Right 21.5
I Right 18.5
J Right 17.1

294
 USING RANDOM EFFECTS IN MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

K Right 18.9
L Right 17.5
M Right 19.5
N Right 16.5
O Right 17.4
P Right 15.6
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Mixed model with lmer

One way to construct a mixed effects model for ratio/interval data is with the lmer function in the lme4
package. The lmerTest package is used to produce an analysis of variance with p-values for model
effects.

Notice the grammar in the lmer function that defines the model: the term (1|Individual) is added to the
model to indicate that Individual is the random term.

As a technical note, the 1 indicates that an intercept is to be fitted for each level of the random variable.

As another technical note, REML stands for restricted maximum likelihood. It is a method of fitting the
model, and is often considered better than fitting with a conventional ML (maximum likelihood)
method.

Define model and conduct analysis of variance

library(lme4)

library(lmerTest)

model = lmer(Length ~ Hand + (1|Individual),

data=Data,
REML=TRUE)

anova(model)

295
 USING RANDOM EFFECTS IN MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Analysis of Variance Table of type III with Satterthwaite

approximation for degrees of freedom

Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)

Hand 0.45125 0.45125 1 15 11.497 0.004034 **

Test the random effects in the model

The rand function from the lmerTest package will test the random effects in the model.
rand(model)

Analysis of Random effects Table

Chi.sq Chi.DF p.value

Individual 58.4 1 2e-14 ***

Mixed model with nlme

Another way to construct a mixed effects model for interval/ratio data is with the lme function in the
nlme package.

Notice the grammar in the lme function that defines the model: the option random=~1|Individual is
added to the model to indicate that Individual is the random term.

As a technical note, the 1 indicates that an intercept is to be fitted for each level of the random variable.

As another technical note, REML stands for restricted maximum likelihood. It is a method of fitting the
model, and is often considered better than fitting with a conventional ML (maximum likelihood)
method.

Define model and conduct analysis of deviance

library(nlme)

model = lme(Length ~ Hand, random=~1|Individual,

data=Data,
method="REML")

library(car)

Anova(model)

Analysis of Deviance Table (Type II tests)

Chisq Df Pr(>Chisq)
Hand 11.497 1 0.0006972 ***

296
 WHAT ARE LEAST SQUARE MEANS? SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Test the random effects in the model

The random effects in the model can be tested by comparing the model to a model fitted with just the
fixed effects and excluding the random effects. Because there are not random effects in this second
model, the gls function in the nlme package is used to fit this model.

We will use a similar method for cumulative link models.

model.fixed = gls(Length ~ Hand,
data=Data,
method="REML")

anova(model,
model.fixed)

Model df AIC BIC logLik Test L.Ratio p-value

model 1 4 80.62586 86.23065 -36.31293
model.fixed 2 3 137.06356 141.26715 -65.53178 1 vs 2 58.4377 <.0001

What are Least Square Means?

Least square means are means for groups that are adjusted for means of other factors in the model.

Imagine a case where you are measuring the height of 7th-grade students in two classrooms, and want
to see if there is a difference between the two classrooms. You are also recording the sex of the
students, and at this age girls tend to be taller than boys. Say classroom A happens to have far more
girls than boys. If you were to look at the mean height in the classrooms, you might find that
classroom A had a higher mean, but this may not be an effect of the different classrooms, but because
of the difference in the counts of boys and girls in each. In this case, reporting least square means for
the classrooms may give a more representative result. Reporting least square means for studies
where there are not equal observations for each combination of treatments is sometimes
recommended. We say the design of these studies is unbalanced.

The following example details this hypothetical example. Looking at the means from the Summarize
function in FSA, we might think there is a meaningful difference between the classrooms, with a mean
height of 153.5 cm vs. 155.0 cm. But looking at the least square means (lsmeans), which are adjusted
for the difference in boys and girls in each classroom, this difference disappears. Each classroom has a
least squared mean of 153.5 cm, indicating the mean of classroom B was inflated due to the higher
proportion of girls.

Note that the following example uses a linear model with the lm function. Here, Height is being treated
as an interval/ratio variable.

This kind of analysis makes certain assumptions about the distribution of the data, but for simplicity,
this example will ignore the need to determine that the data meet these assumptions.

297
 WHAT ARE LEAST SQUARE MEANS? SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Packages used in this chapter

The packages used in this chapter include:

• FSA
• psych
• lsmeans
• car

The following commands will install these packages if they are not already installed:
if(!require(FSA)){install.packages("FSA")}
if(!require(psych)){install.packages("psych")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(car)){install.packages("car")}

Least square means example

Input =("
Classroom Sex Height
A Male 151
A Male 150
A Male 152
A Male 149
A Female 155
A Female 156
A Female 157
A Female 158
B Male 151
B Male 150
B Female 155
B Female 156
B Female 157
B Female 158
B Female 156
B Female 157
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

298
 LEAST SQUARE MEANS FOR MULTIPLE COMPARISONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

rm(Input)

Least square means

library(FSA)

Summarize(Height ~ Classroom,
data=Data,
digits=3)

Classroom n nvalid mean sd min Q1 median Q3 max percZero

1 A 8 8 153.5 3.423 149 150.8 153.5 156.2 158 0
2 B 8 8 155.0 2.928 150 154.0 156.0 157.0 158 0

model = lm(Height ~ Classroom + Sex + Classroom:Sex,

data = Data)

library(lsmeans)

lsmeans(model,
pairwise ~ Classroom,
adjust="tukey")

Classroom lsmean SE df lower.CL upper.CL

A 153.5 0.4082483 12 152.6105 154.3895
B 153.5 0.4714045 12 152.4729 154.5271

Note that an analysis of variance also would have told us that there is a difference between levels of
Sex, but not between levels of Classroom.
library(car)

Anova(model)

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Classroom 0 1 0.0 1
Sex 126 1 94.5 4.857e-07 ***
Classroom:Sex 0 1 0.0 1
Residuals 16 12

Least Square Means for Multiple Comparisons

299
 LEAST SQUARE MEANS FOR MULTIPLE COMPARISONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Comparisons of values across groups in linear models, cumulative link models, and others can be
conducted easily with the lsmeans package. Importantly, it can make comparisons among interactions
of factors.

L.S. means stands for least square means. Least square means are means for treatment levels that are
adjusted for means of other factors in the model. For a more complete explanation, see the What are
least square means? chapter.

p-value adjustments for multiple comparisons

Adjustment of p-values for multiple comparisons is indicated with the adjust= option in the lsmeans
function. Options are "tukey", "scheffe", "sidak", "bonferroni", "dunnettx", "mvt", and "none". For
further information on these options, see the summary function section in:

Lenth, R.V. and M. Hervé. 2015. Package ‘lsmeans’. cran.r-

project.org/web/packages/lsmeans/lsmeans.pdf.

Note that the adjust= option should also be applied to the cld function if a compact letter display is
requested.

Ignore values of lsmeans for clm and clmm models

Typically you should ignore the values of the LS means themselves (lsmeans) when using them with
clm and clmm models. With default settings, the values of the LS means and the values of differences
among the LS means are not easy to interpret. See “clm” in help(models, package="lsmeans") for
details and for other options.

Packages used in this chapter

The packages used in this chapter include:

• psych
• ordinal
• car
• RVAideMemoire
• lsmeans
• multcompView

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(multcompView)){install.packages("multcompView")}

Multiple comparisons with lsmeans

300
LEAST SQUARE MEANS FOR MULTIPLE COMPARISONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

This example uses data set and model from the One-way Ordinal ANOVA with CLM chapter.
Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2
Piglet 3
Tigger 4
Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

301
 LEAST SQUARE MEANS FOR MULTIPLE COMPARISONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Define model and conduct analysis of deviance

library(ordinal)

model = clm(Likert.f ~ Speaker,

data = Data)

library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Speaker 23.395 2 8.315e-06 ***

Group separations by lsmeans in table format

In the lsmeans function, model specifies the model object that was previously fitted. Note the
specialized formula where pairwise indicates that all pairwise comparisons should be conducted, and
Speaker indicates the variable whose levels will be compared.

Remember to ignore the estimate, SE, and lsmean values when using lsmeans with a clm or clmm
model object, unless specific options in lsmeans are selected.
library(lsmeans)

lsmeans(model,
pairwise ~ Speaker,
adjust="tukey") ### Tukey-adjusted comparisons

$contrasts
contrast estimate SE df z.ratio p.value
Pooh - Piglet 4.943822 1.3764706 NA 3.5916658 0.0010
Pooh - Tigger 0.633731 0.9055691 NA 0.6998152 0.7636
Piglet - Tigger -4.310091 1.3173294 NA -3.2718403 0.0031

P value adjustment: tukey method for comparing a family of 3 estimates

302
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

### Remember to ignore “estimate” and “SE” of differences with CLM,

### as well as “lsmeans” in the output not shown here

Compact letter display

Here we’ll create an object of the lsmeans output called leastsquare. Then we’ll pass this object to the
cld function to create a compact letter display.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Speaker,
adjust="tukey") ### Tukey-adjusted comparisons

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

Speaker lsmean SE df asymp.LCL asymp.UCL .group

Piglet -1.783599 0.7755958 NA -3.6355182 0.06832068 a
Tigger 2.526492 0.8413200 NA 0.5176407 4.53534421 b
Pooh 3.160223 0.8912968 NA 1.0320404 5.28840661 b

Confidence level used: 0.95

Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

### Groups sharing a letter in .group are not significantly different

### Remember to ignore “lsmean”, “SE”, “LCL”, and “UCL” for CLM
### unless certain options are used.

Factorial ANOVA: Main Effects, Interaction

Effects, and Interaction Plots
Two-way or multi-way data often come from experiments with a factorial design. A factorial design
has at least two factor variables for its independent variables, and multiple observation for every
combination of these factors.

The weight gain example below show factorial data. In this example, there are three observations for
each combination of Diet and Country.

303
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

With this kind of data, we are usually interested in testing the effect of each factor variable (main
effects) and then the effect of their combination (interaction effect).

For two-way data, an interaction plot shows the mean or median value for the response variable for
each combination of the independent variables. This type of plot, especially if it includes error bars to
indicate the variable of data within each group, gives us some understanding of the effect of the main
factors and their interaction.

When main effects or interaction effects are statistically significant, post-hoc testing can be conducted
to determine which groups differ significantly from other groups. With a factorial experiment, there
are a few guidelines for determining when to do post-hoc testing. The following guidelines are
presented for two-way data for simplicity.

• When neither the main effects nor the interaction effect is statistically significant, no post-hoc
mean-separation testing should be conducted.

• When one or more of the main effects are statistically significant and the interaction effect is
not, post-hoc mean-separation testing should be conducted on significant main effects only.
(This is shown in the first weight gain example below)

• When the interaction effect is statistically significant, post-hoc mean-separation testing should
be conducted on the interaction effect only. This is the case even when the main effects are
also statistically significant.
(This is shown in the second weight gain example below)

The final guideline above is not always followed. There are times when people will present the mean-
separation tests for significant main effects even when the interaction effect is significant. In general,
though, if there is a significant interaction, the mean-separation tests for interaction will better explain
the results of the analysis, and the mean-separation tests for the main effects will be of less interest.

Packages used in this chapter

The packages used in this chapter include:

• psych
• car
• multcompView
• lsmeans
• FSA
• ggplot2
• phia

The following commands will install these packages if they are not already installed:
if(!require(car)){install.packages("car")}
if(!require(psych)){install.packages("psych")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}

304
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

if(!require(FSA)){install.packages("FSA")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(phia)){install.packages("phia")}

Two-way ANOVA example with interaction effect

Imagine for this example an experiment in which people were put on one of three diets to encourage
weight gain. The amount of weight gained will be the dependent variable, and will be considered an
interval/ratio variable. For independent variables, there are three different diets and the country in
which subjects live.

The model will be fit with the lm function, whose syntax is similar to that of the clm function.

This kind of analysis makes certain assumptions about the distribution of the data, but for simplicity,
this example will ignore the need to determine that the data meet these assumptions.

Input =("
Diet Country Weight_change
A USA 0.120
A USA 0.125
A USA 0.112
A UK 0.052
A UK 0.055
A UK 0.044
B USA 0.096
B USA 0.100
B USA 0.089
B UK 0.025
B UK 0.029
B UK 0.019
C USA 0.149
C USA 0.150
C USA 0.142
C UK 0.077
C UK 0.080
C UK 0.066
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Country = factor(Data$Country,
levels=unique(Data$Country))

### Check the data frame

library(psych)

headTail(Data)

305
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Simple interaction plot

The interaction.plot function creates a simple interaction plot for two-way data. The options shown
indicate which variables will used for the x-axis, trace variable, and response variable. The fun=mean
option indicates that the mean for each group will be plotted. For the meaning of other options, see
?interaction.plot.

This style of interaction plot does not show the variability of each group mean, so it is difficult to use
this style of plot to determine if there are significant differences among groups.

The plot shows that mean weight gain for each diet was lower for the UK compared with USA. And
that this difference was relatively constant for each diet, as is evidenced by the lines on the plot being
parallel. This suggests that there is no large or significant interaction effect. That is, the difference
among diets is consistent across countries. And vice-versa, the difference in countries is consistent
across diets.

A couple of other styles of interaction plot are shown at the end of this chapter.
interaction.plot(x.factor = Data$Country,
trace.factor = Data$Diet,
response = Data$Weight_change,
fun = mean,
type="b",
col=c("black","red","green"), ### Colors for levels of trace var.
pch=c(19, 17, 15), ### Symbols for levels of trace var.
fixed=TRUE, ### Order by factor order in data
leg.bty = "o")

306
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

Specify the linear model and conduct an analysis of variance

A linear model is specified with the lm function. Weight_change is the dependent variable. Country
and Diet are the independent variables, and including Country:Diet in the formula adds the interaction
term for Country and Diet to the model.

The ANOVA table indicates that the main effects are significant, but that the interaction effect is not.

model = lm(Weight_change ~ Country + Diet + Country:Diet,

data = Data)

library(car)

Anova(model,
type = "II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Country 0.022472 1 657.7171 7.523e-12 ***
Diet 0.007804 2 114.2049 1.547e-08 ***
Country:Diet 0.000012 2 0.1756 0.8411
Residuals 0.000410 12

Post-hoc testing with lsmeans

Because the main effects were significant, we will want to perform post-hoc mean separation tests for
each main effect factor variable.

For this, we will use the lsmeans package. The linear model under consideration is called model,
created the lm function above. The formula in the lsmeans function indicates that pairwise

307
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

comparisons should be conducted for the variable Country in the first call, and for the variable Diet in
the second call.
library(lsmeans)

lsmeans(model,
pairwise ~ Country,
adjust="tukey") ### Tukey-adjusted comparisons

$contrasts
contrast estimate SE df t.ratio p.value
USA - UK 0.07066667 0.002755466 12 25.646 <.0001

library(lsmeans)

lsmeans(model,
pairwise ~ Diet,
adjust="tukey") ### Tukey-adjusted comparisons

$contrasts
contrast estimate SE df t.ratio p.value
A - B 0.025 0.003374743 12 7.408 <.0001
A - C -0.026 0.003374743 12 -7.704 <.0001
B - C -0.051 0.003374743 12 -15.112 <.0001

Extended example with additional country

For the following example, the hypothetical data have been amended to include a third country, New
Zealand.
Input =("
Diet Country Weight_change
A USA 0.120
A USA 0.125
A USA 0.112
A UK 0.052
A UK 0.055
A UK 0.044
A NZ 0.080
A NZ 0.090
A NZ 0.075
B USA 0.096
B USA 0.100
B USA 0.089
B UK 0.025
B UK 0.029
B UK 0.019
B NZ 0.055
B NZ 0.065
B NZ 0.050
C USA 0.149
C USA 0.150
C USA 0.142

308
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

C UK 0.077
C UK 0.080
C UK 0.066
C NZ 0.055
C NZ 0.065
C NZ 0.050
C NZ 0.054
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Country = factor(Data$Country,
levels=unique(Data$Country))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Simple interaction plot

The plot suggests that the effect of diet is not consistent across all three countries. While Diet C
showed the greatest mean weight gain for USA and UK, for NZ it has a lower mean than Diet A. This
suggests there may be a meaningful or significant interaction effect, but we will need to do a statistical
test to confirm this hypothesis.
interaction.plot(x.factor = Data$Country,
trace.factor = Data$Diet,
response = Data$Weight_change,
fun = mean,
type="b",
col=c("black","red","green"), ### Colors for levels of trace var.
pch=c(19, 17, 15), ### Symbols for levels of trace var.
fixed=TRUE, ### Order by factor order in data
leg.bty = "o")

309
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

Specify the linear model and conduct an analysis of variance

The ANOVA table indicates that the interaction effect is significant, as are both main effects.
model = lm(Weight_change ~ Country + Diet + Country:Diet,
data = Data)

library(car)

Anova(model,
type = "II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Country 0.0256761 2 318.715 2.426e-15 ***
Diet 0.0051534 2 63.969 3.634e-09 ***
Country:Diet 0.0040162 4 24.926 2.477e-07 ***
Residuals 0.0007653 19

Post-hoc testing with lsmeans

Because the interaction effect was significant, we would like to compare all group means from the
interaction. Even though the main effects were significant, the typical advice is to not conduct
pairwise comparisons for main effects when their interaction is significant. This is because focusing
on the groups in the interaction better describe the results of the analysis.

We will use the lsmeans package, but ask for a compact letter display with the cld function. First we
create an object, named leastsquare, with the results of the call to lsmeans. Notice here that the
formula indicates that pairwise comparisons should be conducted for the interaction of Country and
Diet, indicated with Country:Diet.

310
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

Be sure to read the Least Square Means for Multiple Comparisons chapter for correct interpretation of
least square means. For lm model objects, the values for lsmean, SE, LCL, and UCL values are
meaningful.

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Country:Diet,
adjust="tukey") ### Tukey-adjusted comparisons

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

Country Diet lsmean SE df lower.CL upper.CL .group

UK B 0.02433333 0.003664274 19 0.01291388 0.03575278 a
UK A 0.05033333 0.003664274 19 0.03891388 0.06175278 b
NZ C 0.05600000 0.003173354 19 0.04611047 0.06588953 b
NZ B 0.05666667 0.003664274 19 0.04524722 0.06808612 bc
UK C 0.07433333 0.003664274 19 0.06291388 0.08575278 cd
NZ A 0.08166667 0.003664274 19 0.07024722 0.09308612 de
USA B 0.09500000 0.003664274 19 0.08358055 0.10641945 e
USA A 0.11900000 0.003664274 19 0.10758055 0.13041945 f
USA C 0.14700000 0.003664274 19 0.13558055 0.15841945 g

Confidence level used: 0.95

Conf-level adjustment: sidak method for 9 estimates
P value adjustment: tukey method for comparing a family of 9 estimates
significance level used: alpha = 0.05

### Groups sharing a letter are not significantly different

Interaction plot with error bars using ggplot2

The interaction plots created with the interaction.plot function above are handy to investigate trends
in the data, but have the disadvantage of not showing the variability in data within each group.

The package ggplot2 can be used to create attractive interaction plots with error bars. Here, we will
use standard error of each mean for the error bars.
### Create a data frame called Sum with means and standard deviations

library(FSA)

Sum = Summarize(Weight_change ~ Country + Diet,

data=Data,
digits=3)

311
FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

### Add standard error of the mean to the Sum data frame

Sum$se = Sum$sd / sqrt(Sum$n)

Sum$se = signif(Sum$se, digits=3)

Sum

Country Diet n nvalid mean sd min Q1 median Q3 max percZero se

1 USA A 3 3 0.119 0.007 0.112 0.116 0.120 0.122 0.125 0 0.00404
2 UK A 3 3 0.050 0.006 0.044 0.048 0.052 0.054 0.055 0 0.00346
3 NZ A 3 3 0.082 0.008 0.075 0.078 0.080 0.085 0.090 0 0.00462
4 USA B 3 3 0.095 0.006 0.089 0.092 0.096 0.098 0.100 0 0.00346
5 UK B 3 3 0.024 0.005 0.019 0.022 0.025 0.027 0.029 0 0.00289
6 NZ B 3 3 0.057 0.008 0.050 0.052 0.055 0.060 0.065 0 0.00462
7 USA C 3 3 0.147 0.004 0.142 0.146 0.149 0.150 0.150 0 0.00231
8 UK C 3 3 0.074 0.007 0.066 0.072 0.077 0.078 0.080 0 0.00404
9 NZ C 4 4 0.056 0.006 0.050 0.053 0.054 0.058 0.065 0 0.00300

### Order levels of the factor; otherwise R will alphabetize them

Sum$Country = factor(Sum$Country,
levels=unique(Sum$Country))

### Produce interaction plot

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x = Country,

y = mean,
color = Diet)) +
geom_errorbar(aes(ymin = mean - se,
ymax = mean + se),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold") +
scale_colour_manual(values= c("black","red","green")) +
ylab("Mean weight change")

### You may see an error, “ymax not defined”

### In this case, it does not appear to affect anything

312
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

Interaction plot with mean separation letters manually added

It is common to add mean separation letters from post-hoc analyses to interaction plots. Typically,
letters would be added manually in either image manipulation software like Photoshop or GIMP, or in
a word processor or other software that can handle graphic manipulation. Letters can also be added
to the plot by ggplot2 with the annotate function.

It is probably more common for means to be lettered so that the greatest mean is indicated with a.
However, lsmeans by default labels the least mean with a. The order of letters can be reversed
manually.

313
 FACTORIAL ANOVA: MAIN EFFECTS, INTERACTION EFFECTS, AND SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
INTERACTION PLOTS EVALUATION IN R

Plot of mean weight change for three diets in three countries. Means sharing a
letter are not significantly different according to pairwise comparisons of least
square means with Tukey adjustment for multiple comparisons.

Interaction plot with the phia package

The phia package can be used to create interaction plots quickly. By default the error bars indicate
standard error of the means. For additional options, see ?interactionMeans.

Note that model is the linear model specified above.

library(phia)

IM = interactionMeans(model)

IM

Country Diet adjusted mean std. error

1 USA A 0.11900000 0.003664274
2 UK A 0.05033333 0.003664274
3 NZ A 0.08166667 0.003664274
4 USA B 0.09500000 0.003664274
5 UK B 0.02433333 0.003664274
6 NZ B 0.05666667 0.003664274
7 USA C 0.14700000 0.003664274
8 UK C 0.07433333 0.003664274
9 NZ C 0.05600000 0.003173354

314
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Produce interaction plot

plot(IM)

### Return the graphics device to its default 1-plot-per-window state||

par(mfrow=c(1,1))

p-values and R-square Values for Models

When developing more complex models it is often desirable to report a p-value for the model as a
whole as well as an R-square for the model.

p-values for models

The p-value for a model determines the significance of the model compared with a null model. For a
linear model, the null model is defined as the dependent variable being equal to its mean. So, the p-
value for the model is answering the question, Does this model explain the data significantly better than
would just looking at the average value of the dependent variable?

315
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

R-squared and pseudo R-squared

The R-squared value is a measure of how well the model explains the data. It is an example of a
goodness-of-fit statistic.

R-squared for linear (ordinary least squares) models

In R, models fit with the lm function are linear models fit with ordinary least squares (OLS). For these
models, R-squared indicates the proportion of the variability in the dependent variable that is
explained by model. That is, an R-squared of 0.60 indicates that 60% of the variability in the
dependent variable is explained by the model.

Pseudo R-squared
For many types of models, R-squared is not defined. These include relatively common models like
logistic regression and the cumulative link models used in this book. For these models, pseudo R-
squared measures can be calculated. A pseudo R-squared is not directly comparable to the R-squared
for OLS models. Nor can it can be interpreted as the proportion of the variability in the dependent
variable that is explained by model. Instead pseudo R-squared measures are relative measures among
similar models indicating how well the model explains the data.

In general I favor the Nagelkerke pseudo R-squared, but there is no agreement as to which pseudo R-
squared measurement should be used.

This book uses three pseudo R-squared measures: McFadden, Cox and Snell (also referred to as ML),
Nagelkerke (also referred to as Cragg and Uhler).

p-values and R-squared values.

p-values and R-squared values measure different things. The p-value indicates if there is a significant
relationship described by the model, and the R-squared measures the degree to which the data is
explained by the model. It is therefore possible to get a significant p-value with a low R-squared value.
This often happens when there is a lot of variability in the dependent variable, but there are enough
data points for a significant relationship to be indicated.

Packages used in this chapter

The packages used in this chapter include:

• psych
• lmtest
• boot
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(lmtest)){install.packages("lmtest")}
if(!require(boot)){install.packages("boot")}
if(!require(rcompanion)){install.packages("rcompanion")}

316
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Example of model p-value, R-squared, and pseudo R-squared

The following example uses some hypothetical data of a sample of people for which typing speed
(Words.per.minute) and age were measured. After plotting the data, we decide to construct a
polynomial model with Words.per.minute as the dependent variable and Age and Age2 as the
independent variables. Notice in this example that all variables are treated as interval/ratio variables,
and that the independent variables are not factor variables, but are continuous variables.

The data will first be fit with a linear model with the lm function. Passing this model to the summary
function will display the p-value for the model and the R-squared value for the model.

The same data will then be fit with a generalized linear model with the glm function. This type of
model allows more latitude in the types of data that can be fit, but in this example, we’ll use the
family=gaussian option, which will mimic the model fit with the lm function, though the underlying
math is different.

Importantly, the summary of the glm function does not produce a p-value for the model nor an R-
squared for the model.

For the model fit with glm, the p-value can be determined with the anova function comparing the fitted
model to a null model. The null model is fit with only an intercept term on the right side of the model.
As an alternative, the nagelkerke function described below also reports a p-value for the model, using
the likelihood ratio test.

There is no R-squared defined for a glm model. Instead a pseudo R-squared can be calculated. The
function nagelkerke produces pseudo R-squared values for a variety of models. It reports three types:
McFadden, Cox and Snell, and Nagelkerke. In general I recommend using the Nagelkerke measure,
though there is no agreement on which pseudo R-squared measurement to use, if any at all.

Note that the Nagelkerke is the same as the Cox and Snell, except that the value is adjusted upward so
that the Nagelkerke has a maximum value of 1. It has been suggested that a McFadden value of 0.2–0.4
indicates a good fit.

Note that these models makes certain assumptions about the distribution of the data, but for
simplicity, this example will ignore the need to determine if the data met these assumptions.

Input =("
Age Words.per.minute
12 23
12 32
12 25
13 27
13 30
15 29
15 33
16 37
18 29
22 33

317
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

23 37
24 33
25 43
27 35
33 30
42 25
53 22
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Plot the data

plot(Words.per.minute ~ Age,
data = Data,
pch=16,
xlab = "Age",
ylab = "Words per minute")

318
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Prepare data
### Create new variable for the square of Age

Data$Age2 = Data$Age ^ 2

### Double check data frame

library(psych)

headTail(Data)

Age Words.per.minute Age2

1 12 23 144
2 12 32 144
3 12 25 144
4 13 27 169
... ... ... ...
14 27 35 729
15 33 30 1089
16 42 25 1764
17 53 22 2809

Linear model
model = lm (Words.per.minute ~ Age + Age2,
data=Data)

summary(model) ### Shows coefficients,

### p-value for model, and R-squared

Multiple R-squared: 0.5009, Adjusted R-squared: 0.4296

F-statistic: 7.026 on 2 and 14 DF, p-value: 0.00771

### p-value and (multiple) R-squared value

Simple plot of data and model

For bivariate data, the function plotPredy will plot the data and the predicted line for the model. It also
works for polynomial functions, if the order option is changed.
library(rcompanion)

plotPredy(data = Data,
y = Words.per.minute,
x = Age,
x2 = Age2,
model = model,
order = 2,
xlab = "Words per minute",
ylab = "Age")

319
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Generalized linear model

model = glm (Words.per.minute ~ Age + Age2,
data = Data,
family = gaussian)

summary(model) ### Shows coefficients

Calculate p-value for model

In R, the most common way to calculate the p-value for a fitted model is to compare the fitted model to
a null model with the anova function. The null model is usually formulated with just a constant on the
right side.
null = glm (Words.per.minute ~ 1, ### Create null model
data = Data, ### with only an intercept on the right side
family = gaussian)

anova (model,
null,
test="Chisq") ### Tests options "Rao", "LRT",
### "Chisq", "F", "Cp"
### But some work with only some model types

Resid. Df Resid. Dev Df Deviance Pr(>Chi)

1 14 243.07
2 16 487.06 -2 -243.99 0.0008882 ***

Calculate pseudo R-squared and p-value for model

An alternative test for the p-value for a fitted model, the nagelkerke function will report the p-value for
a model using the likelihood ratio test.

320
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The nagelkerke function also reports the McFadden, Cox and Snell, and Nagelkerke pseudo R-squared
values for the model.
library(rcompanion)

nagelkerke(model)

$Pseudo.R.squared.for.model.vs.null

Pseudo.R.squared
McFadden 0.112227
Cox and Snell (ML) 0.500939
Nagelkerke (Cragg and Uhler) 0.501964

$Likelihood.ratio.test

Df.diff LogLik.diff Chisq p.value

-2 -5.9077 11.815 0.0027184

Likelihood ratio test for p-value for model

The p-value for a model by the likelihood ratio test can also be determined with the lrtest function in
the lmtest package.
library(lmtest)

lrtest(model)

#Df LogLik Df Chisq Pr(>Chisq)

1 4 -46.733
2 2 -52.641 -2 11.815 0.002718 **

Simple plot of data and model

library(rcompanion)

plotPredy(data = Data,
y = Words.per.minute,
x = Age,
x2 = Age2,
model = model,
order = 2,
xlab = "Words per minute",
ylab = "Age",
col = "red") ### line color

321
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional analyses: Confidence intervals for p-values and R-squared values

It is relatively easy to produce confidence intervals for p-values, R-squared values, or other parameters
from model fitting, such as coefficients for regression terms. This can be accomplished with
bootstrapping. Here the boot.ci function from the boot package is used.

The code below is a little complicated, but relatively flexible.

Function can contain any function of interest, as long as it includes an input vector or data frame (input
in this case) and an indexing variable (index in this case). Stat is set to produce the actual statistic of
interest on which to perform the bootstrap (r.squared from the summary of the lm in this case).

The code Function(Data, 1:17) is there simply to test Function on the data frame Data. In this case, it
will produce the output of Function for the first n rows of Data. Since n is defined as the length of the
first column in Data, this should return the value for Stat for the whole data frame, if Function is set up
correctly.
Input =("
Age Words.per.minute
12 23
12 32
12 25
13 27
13 30
15 29
15 33
16 37
18 29
22 33
23 37
24 33
25 43
27 35

322
P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

33 30
42 25
53 22
")

Data = read.table(textConnection(Input),header=TRUE)

Data$Age2 = Data$Age ^ 2

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Linear model

R-squared value
library(boot)

Function = function(input, index){

Input = input[index,]
Result = lm (Words.per.minute ~ Age + Age2,
data = Input)
Stat = summary(Result)$r.squared
return(Stat)}

### Test Function

n = length(Data[,1])

Function(Data, 1:n)

[1] 0.5009385

### Produce Stat estimate by bootstrap

Boot = boot(Data,
Function,
R=5000)

323
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

mean(Boot$t[,1])

[1] 0.5754582

### Produce confidence interval by bootstrap

boot.ci(Boot,
conf = 0.95,
type = "perc")

### Options: "norm", "basic", "stud", "perc", "bca", "all"

BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS

Based on 1000 bootstrap replicates

Intervals :
Level Percentile
95% ( 0.3796, 0.7802 )
Calculations and Intervals on Original Scale

### Other information

Boot

hist(Boot$t[,1],
col = "darkgray")

p-value
Unfortunately the p-value for the model is not stored neatly in the lm model object, so it has to be
calculated in the Stat definition.
library(boot)

Function = function(input, index){

Input = input[index,]
Result = lm (Words.per.minute ~ Age + Age2,
data = Input)
Stat = pf(summary(Result)$fstatistic[1],
summary(Result)$fstatistic[2],
summary(Result)$fstatistic[3],
lower.tail = FALSE)
return(Stat)}

### Test Function

n = length(Data[,1])

Function(Data, 1:n)

324
 P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

value
0.007710425

### Produce Stat estimate by bootstrap

Boot = boot(Data,
Function,
R=5000)

mean(Boot$t[,1])

[1] 0.00672866

### Produce confidence interval by bootstrap

boot.ci(Boot,
conf = 0.95,
type = "perc")

### Options: "norm", "basic", "stud", "perc", "bca", "all"

BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS

Based on 1000 bootstrap replicates

Intervals :
Level Percentile
95% ( 0.0000, 0.0368 )
Calculations and Intervals on Original Scale

### Other information

Boot

hist(Boot$t[,1],
col = "darkgray")

Generalized linear model

Pseudo R-squared value

The nagelkerke function produces a list. The second item in the list is a matrix named

Pseudo.R.squared.for.model.vs.null.

The third element of this matrix is the value for the Nagelkerke pseudo R-squared. So,

nagelkerke(Result, Null)[[2]][3]

yields the value of the Nagelkerke pseudo R-squared.

325
P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(boot)

library(rcompanion)

Function = function(input, index){

Input = input[index,]
Result = glm (Words.per.minute ~ Age + Age2,
data = Input,
family="gaussian")
Null = glm (Words.per.minute ~ 1,
data = Input,
family="gaussian")
Stat = nagelkerke(Result, Null)[[2]][3]
return(Stat)}

### Test Function

n = length(Data[,1])

Function(Data, 1:n)

[1] 0.501964

### Produce Stat estimate by bootstrap

Boot = boot(Data,
Function,
R=1000)

### In this case, even 1000 iterations can take a while

mean(Boot$t[,1])

[1] 0.5803598

### Produce confidence interval by bootstrap

boot.ci(Boot,
conf = 0.95,
type = "perc")

### Options: "norm", "basic", "stud", "perc", "bca", "all"

BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS

Based on 1000 bootstrap replicates

Intervals :
Level Percentile
95% ( 0.3836, 0.7860 )
Calculations and Intervals on Original Scale

326
P-VALUES AND R-SQUARE VALUES FOR MODELS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Other information

Boot

hist(Boot$t[,1],
col = "darkgray")

p-value
library(boot)

library(rcompanion)

Function = function(input, index){

Input = input[index,]
Result = glm (Words.per.minute ~ Age + Age2,
data = Input,
family="gaussian")
Null = glm (Words.per.minute ~ 1,
data = Input,
family="gaussian")
Stat = nagelkerke(Result, Null)[[3]][1,4]
return(Stat)}

### Test Function

n = length(Data[,1])

Function(Data, 1:n)

[1] 0.0027184

### Produce Stat estimate by bootstrap

Boot = boot(Data,
Function,
R=1000)

### In this case, even 1000 iterations can take a while

mean(Boot$t[,1])

[1] 0.002723259

### Produce confidence interval by bootstrap

boot.ci(Boot,
conf = 0.95,
type = "perc")

327
 INTRODUCTION TO CUMULATIVE LINK MODELS (CLM) FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

### Options: "norm", "basic", "stud", "perc", "bca", "all"

BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS

Based on 1000 bootstrap replicates

Intervals :
Level Percentile
95% ( 0.0000, 0.0173 )
Calculations and Intervals on Original Scale

### Other information

Boot

hist(Boot$t[,1],
col = "darkgray")

References

Kabacoff, R.I. Bootstrapping. Quick-R. www.statmethods.net/advstats/bootstrapping.html.

Ordinal Tests with Cumulative Link Models

Introduction to Cumulative Link Models (CLM)

for Ordinal Data
In the section on nonparametric tests in this book, each test is used for data from a specific situation or
design, such as comparing groups from two-sample unpaired data, or two-sample paired data, or with
an unreplicated complete block design.

Cumulative link models are a different approach to analyzing ordinal data. Models can be chosen to
handle simple or more complex data and situations.

This approach is very powerful and flexible, and might be considered the best approach for data with
ordinal dependent variables in many cases.

However, two disadvantages to using these models is that 1) your audience may not familiar with
them, and 2) their results can be somewhat tricky to interpret or explain.

These models are also called ordinal regression models, or proportional odds models.

The ordinal package

These models and tests will use the ordinal package, and either of two functions, clm and clmm.

328
 INTRODUCTION TO CUMULATIVE LINK MODELS (CLM) FOR ORDINAL SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
DATA EVALUATION IN R

A few notes on using cumulative link models:

• The dependent variable must be an ordered factor variable. It does not need to have
numerals for levels. For example it could have levels doctorate > masters > bachelors >
associates > high.school. But also it could have the levels 4 > 3 > 2 > 1.

• Independent variables can be factors, ordered factors, or interval/ratio variables.

• The general interpretation for significant results of these models is that there is a significant
effect of the independent variable on the dependent variable, or that there is a significant
difference among groups.

• Post-hoc tests for factors or groups can be conducted with the lsmeans package. An optional
approach for post-hoc tests is to use pairwise ordinal tests of groups, or with pairwise ordinal
tests for paired data. These can be conducted with the functions pairwiseOrdinalTest and
pairwiseOrdinalPairedTest from the rcompanion package.

• The threshold = “equidistant” and threshold = “symmetric” options can be used to indicate to
the software that levels of the response variable are equally spaced or symmetrically spaced,
respectively. This is useful to indicate when these conditions are assumed to be true, but are
also useful to try if the model procedure produces errors. Likert items using symmetrical
language in the range of responses could be considered symmetric. Likert items with several
numbered options with anchor terms only at the ends of scale might be considered
equidistant.

Analysis of deviance
The significance of the effects of independent variables will be tested with an analysis of deviance
(ANODE) approach. This is analogous to the analysis of variance (ANOVA) used in linear models.

Model assumptions for CLM

Most statistical models have some assumptions about the underlying data. In order for the model to
be valid, these assumptions have to be met.

For CLM, the assumption of concern is called the proportional odds assumption. An explanation of this
assumption can be found in the Wikipedia or IDRE articles cited below.

The ordinal package can test for the proportional odds assumption with the nominal_test and
scale_test functions (Christensen 2015b). If any independent variable fails these tests (that is, a
significant p-value is returned), that variable can be handled differently in the model using the nominal
and scale options in the clm function.

References

For more information on these models and the ordinal package, see:

329
 TWO-SAMPLE ORDINAL TEST WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Christensen, H.R.B. 2015a. Analysis of ordinal data with cumulative link models—estimation
with the R-package ordinal. cran.r-
project.org/web/packages/ordinal/vignettes/clm_intro.pdf.

• library(ordinal); help(package="ordinal")

• Wikipedia. 2015. “The model and the proportional odds assumption” in Ordered logit.
en.wikipedia.org/wiki/Ordered_logit#The_model_and_the_proportional_odds_assumption.

• IDRE . 2015. R Data Analysis Examples: Ordinal Logistic Regression. UCLA.

www.ats.ucla.edu/stat/r/dae/ologit.htm.

• Christensen, R.H.B. 2015b. Package ‘ordinal’. cran.r-

project.org/web/packages/ordinal/ordinal.pdf.

• Hervé, M. 2014. “72. Analyser des notes” in Aide-mémoire de statistique appliquée à la

biologie. cran.r-project.org/doc/contrib/Herve-Aide-memoire-statistique.pdf.

Two-sample Ordinal Test with CLM

Appropriate data
• Two-sample data. That is, one-way data with two groups only
• Dependent variable is ordered factor
• Independent variable is a factor with two levels. That is, two groups
• Observations between groups are independent. That is, not paired or repeated measures
data

Interpretation
A significant result can be interpreted as, “There was a significant difference between groups.”
Or, “There was a significant effect of Independent Variable.”

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ordinal
• car
• RVAideMemoire

The following commands will install these packages if they are not already installed:

330
TWO-SAMPLE ORDINAL TEST WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}

Two-sample ordinal model example

Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2
Piglet 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Create a new variable which is the Likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

331
 TWO-SAMPLE ORDINAL TEST WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Speaker + Likert.f,
data = Data)

Likert.f
Speaker 1 2 3 4 5
Piglet 1 6 2 1 0
Pooh 0 0 1 6 3

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5
Piglet 0.1 0.6 0.2 0.1 0.0
Pooh 0.0 0.0 0.1 0.6 0.3

Bar plots of data by group

library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,2) # columns and rows of individual plots
)

332
 TWO-SAMPLE ORDINAL TEST WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

Speaker n mean sd min Q1 median Q3 max percZero

1 Piglet 10 2.3 0.823 1 2 2 2.75 4 0
2 Pooh 10 4.2 0.632 3 4 4 4.75 5 0

Two-sample ordinal model example

The model is specified using formula notation. Here, Likert.f is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?clm.

The p-value for the effect of Speaker is determined using the Anova function in the packages
RVAideMemoire and car.

Define model
library(ordinal)

model = clm(Likert.f ~ Speaker,

data = Data)

Analysis of deviance

333
 TWO-SAMPLE PAIRED ORDINAL TEST WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Speaker 17.505 1 2.866e-05 ***

Check model assumptions

nominal_test(model)

Tests of nominal effects

formula: Likert.f ~ Speaker

Df logLik AIC LRT Pr(>Chi)
<none> -20.199 50.397
Speaker

### No p-value produced for this example.

scale_test(model)

Tests of scale effects

formula: Likert.f ~ Speaker

Df logLik AIC LRT Pr(>Chi)
<none> -20.199 50.397
Speaker 1 -20.148 52.295 0.1019 0.7496

### No violation of model assumptions

Two-sample Paired Ordinal Test with CLMM

Note that the order of the data doesn’t matter, as it did in the paired signed-rank test example, because
here the blocking variable, Student, is entered explicitly in the model.

Note that the clmm function is used here instead of the clm function. The clmm function specifies a
mixed effects model. This type of model is appropriate for paired and repeated measures analyses.

Appropriate data
• Two-sample data. That is, one-way data with two groups only
• Dependent variable is ordered factor

334
 TWO-SAMPLE PAIRED ORDINAL TEST WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Independent variable is a factor with two levels. That is, two groups
• Observations between groups are paired

Interpretation
A significant result can be interpreted as, “There was a significant difference between groups.”
Or, “There was a significant effect of Independent Variable.”

Packages used in this chapter

The packages used in this chapter include:

• psych
• ordinal
• car
• RVAideMemoire

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}

Two-sample paired ordinal model example

Input =("
Speaker Time Student Likert
Pooh 1 a 1
Pooh 1 b 4
Pooh 1 c 3
Pooh 1 d 3
Pooh 1 e 3
Pooh 1 f 3
Pooh 1 g 4
Pooh 1 h 3
Pooh 1 i 3
Pooh 1 j 3
Pooh 2 a 4
Pooh 2 b 5
Pooh 2 c 4
Pooh 2 d 5
Pooh 2 e 4
Pooh 2 f 5
Pooh 2 g 3
Pooh 2 h 4
Pooh 2 i 3
Pooh 2 j 4
")

Data = read.table(textConnection(Input),header=TRUE)

335
 TWO-SAMPLE PAIRED ORDINAL TEST WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Create a new variable which is the Likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Plot the paired data

Scatter plot with one-to-one line

Paired data can be visualized with a scatter plot of the paired cases. In the plot below, points that fall
above and to the left of the blue line indicate cases for which the value for Time 2 was greater than for
Time 1.

Note that the points in the plot are jittered slightly so that points that would fall directly on top of one
another can be seen.

First, two new variables, Time.1 and Time.2, are created by extracting the values of Likert for
observations with the Time variable equal to 1 or 2, respectively.

Note that for this plot, the data must be ordered so that the first observation where Time = 1 is paired
to the first observation where Time = 2, and so on.
Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]

plot(Time.1, jitter(Time.2), # jitter offsets points so you can see them all
pch = 16, # shape of points
cex = 1.0, # size of points
xlim=c(1, 5.5), # limits of x axis
ylim=c(1, 5.5), # limits of y axis
xlab="Time 1",
ylab="Time 2"
)
abline(0,1, col="blue", lwd=2) # line with intercept of 0 and slope of 1

336
 TWO-SAMPLE PAIRED ORDINAL TEST WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bar plot of differences

Note that for this plot, the data must be ordered so that the first observation where Time = 1 is paired
to the first observation where Time = 2, and so on.
Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]

Difference = Time.2 - Time.1

barplot(Difference,
col="dark gray",
xlab="Observation",
ylab="Difference (Time 2 – Time 1)")

Two-sample paired ordinal model

The model is specified using formula notation with the clmm function (not clm). Here, Likert.f is the
dependent variable and Time is the independent variable. Student is specified as the blocking variable.

337
 TWO-SAMPLE PAIRED ORDINAL TEST WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

It is identified as a random variable in the mixed effects model. The data= option indicates the data
frame that contains the variables. For the meaning of other options, see ?clmm.

The p-value for the effect of Time is determined using the Anova function in the packages
RVAideMemoire and car.

Define model
library(ordinal)

model = clmm(Likert.f ~ Time + (1|Student),

data = Data)

Analysis of deviance
library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Time 9.2033 1 0.002416 **

Determine the effect of the random variable

The effect of the random variable (Student, in this case) can be determined by comparing the full
model to a model with only the fixed effects (Time, in this case).

First, model.fixed is defined, and then the two models are passed to the anova function.
model.fixed = clm(Likert.f ~ Time,
data = Data)

anova(model,
null = model.fixed)

no.par AIC logLik LR.stat df Pr(>Chisq)

model.null 4 45.262 -18.631
model 5 47.242 -18.621 0.02 1 0.8875

Check model assumptions

At the time of writing, the nominal_test and scale_test functions don’t work with clmm model objects,
so we will define a similar model with clm and no random effects, and test the proportional odds
assumption on this model. In this example, neither the nominal_test nor scale_test functions produced
a p-value, so the results of these tests are not conclusive.
model.clm = clm(Likert.f ~ Time + Student,

338
 ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

data = Data)

nominal_test(model.clm)

Tests of nominal effects

formula: Likert.f ~ Time + Student

Df logLik AIC LRT Pr(>Chi)
<none> -11.821 49.642
Time
Student

### No p-value produced for this example.

scale_test(model.clm)

Tests of scale effects

formula: Likert.f ~ Time + Student

Df logLik AIC LRT Pr(>Chi)
<none> -11.821 49.642
Time
Student

Warning messages:
1: (-2) Model failed to converge: degenerate Hessian with 1 negative eigenvalues
In addition: maximum number of consecutive Newton modifications reached
2: (-1) Model failed to converge with max|grad| = 6.85558e-06 (tol = 1e-06)
In addition: maximum number of consecutive Newton modifications reached

### No p-value produced for this example.

### Errors produced for this example.

One-way Ordinal ANOVA with CLM

Appropriate data
• One-way data with two or more groups
• Dependent variable is ordered factor
• Independent variable is a factor with at least two levels or groups
• Observations between groups are not paired or repeated measures

Interpretation
A significant result can be interpreted as, “There was a significant difference among groups.”
Or, “There was a significant effect of Independent Variable.”

A significant post-hoc analysis indicates, “There was a significant difference between Group A
and Group B”, and so on.

339
 ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ordinal
• car
• RVAideMemoire
• multcompView
• lsmeans
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-way ordinal ANOVA example

Input =("
Speaker Likert
Pooh 3
Pooh 5
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 4
Pooh 5
Pooh 5
Piglet 2
Piglet 4
Piglet 2
Piglet 2
Piglet 1
Piglet 2
Piglet 3
Piglet 2
Piglet 2
Piglet 3
Tigger 4

340
ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Tigger 4
Tigger 4
Tigger 4
Tigger 5
Tigger 3
Tigger 5
Tigger 4
Tigger 4
Tigger 3
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Speaker + Likert.f,
data = Data)

Likert.f
Speaker 1 2 3 4 5
Pooh 0 0 1 6 3
Piglet 1 6 2 1 0
Tigger 0 0 2 6 2

341
ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

XT = xtabs( ~ Speaker + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Speaker 1 2 3 4 5
Pooh 0.0 0.0 0.1 0.6 0.3
Piglet 0.1 0.6 0.2 0.1 0.0
Tigger 0.0 0.0 0.2 0.6 0.2

Bar plots by group

library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Speaker,
data=Data,
digits=3)

342
 ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Speaker n mean sd min Q1 median Q3 max percZero

1 Pooh 10 4.2 0.632 3 4 4 4.75 5 0
2 Piglet 10 2.3 0.823 1 2 2 2.75 4 0
3 Tigger 10 4.0 0.667 3 4 4 4.00 5 0

One-way ordinal ANOVA

The model is specified using formula notation. Here, Likert.f is the dependent variable and Speaker is
the independent variable. The data= option indicates the data frame that contains the variables. For
the meaning of other options, see ?clm.

The p-value for the effect of Speaker is determined using the Anova function in the packages
RVAideMemoire and car.

Define model
library(ordinal)

model = clm(Likert.f ~ Speaker,

data = Data)

Analysis of deviance
library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Speaker 23.395 2 8.315e-06 ***

Post-hoc test with lsmeans

In the lsmeans function, model specifies the model object that was previously fitted. Note the
specialized formula where pairwise indicates that all pairwise comparisons should be conducted, and
Instructor indicates the variable whose levels will be compared.

Here we’ll create an object of the lsmeans output called leastsquare. Then we’ll pass this object to the
cld function to create a compact letter display.

Be sure to read the Least Square Means for Multiple Comparisons chapter for correct interpretation of
least square means. For clm model objects, the lsmean, SE, LCL, and UCL values should be ignored,
unless specific options in lsmeans are selected.

343
ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Speaker,
adjust="tukey") ### Tukey-adjusted comparisons

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

Speaker lsmean SE df asymp.LCL asymp.UCL .group

Piglet -1.783599 0.7755958 NA -3.6355182 0.06832068 a
Tigger 2.526492 0.8413200 NA 0.5176407 4.53534421 b
Pooh 3.160223 0.8912968 NA 1.0320404 5.28840661 b

Confidence level used: 0.95

Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

### Groups sharing a letter in .group are not significantly different

### Remember to ignore “lsmean”, “SE”, “LCL”, and “UCL” for CLM

Check model assumptions

nominal_test(model)

Tests of nominal effects

formula: Likert.f ~ Speaker

Df logLik AIC LRT Pr(>Chi)
<none> -30.149 72.298
Speaker

### No p-value

scale_test(model)

Tests of scale effects

formula: Likert.f ~ Speaker

Df logLik AIC LRT Pr(>Chi)
<none> -30.149 72.298
Speaker 2 -29.839 75.677 0.62096 0.7331

### No violation in assumptions

344
 ONE-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional analyses

Post-hoc test: pairwise ordinal tests for multiple comparisons of groups

Table format
Post-hoc testing can be conducted with the pairwiseOrdinalTest function in the rcompanion package,
which produces a table of p-values comparing each pair of groups, or with pairwiseOrdinalMatrix
function, which produces a matrix of p-values. This can then be converted into a compact letter
display.

To prevent the inflation of type I error rates, adjustments to the p-values can be made using the
p.adjust.method option. See ?p.adjust for details on available p-value adjustment methods.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Pairwise ordinal regression tests

library(rcompanion)

PT = pairwiseOrdinalTest(x = Data$Likert.f,
g = Data$Speaker,
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Comparison p.value p.adjust

1 Pooh - Tigger = 0 0.4763 4.763e-01
2 Pooh - Piglet = 0 2.866e-05 8.598e-05
3 Tigger - Piglet = 0 9.477e-05 1.422e-04

#### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 Pooh a a
2 Tigger a a
3 Piglet b b

Groups sharing a letter are not significantly different (alpha = 0.05).

345
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Matrix format and compact letter display

Usually you would want to first order your treatment variable (Speaker) by median or other location
statistic for the letters in the compact letter display to be in their proper order.
### Order groups by median

Data$Speaker = factor(Data$Speaker,
levels=c("Pooh", "Tigger", "Piglet"))

### Pairwise ordinal regression tests

library(rcompanion)

PM = pairwiseOrdinalMatrix(x = Data$Likert.f,
g = Data$Speaker,
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PM

$Adjusted
Pooh Tigger Piglet
Pooh 1.000e+00 0.4763000 8.598e-05
Tigger 4.763e-01 1.0000000 1.422e-04
Piglet 8.598e-05 0.0001422 1.000e+00

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Pooh Tigger Piglet

"a" "a" "b"

### Values sharing a letter are not significantly different

One-way Repeated Ordinal ANOVA with CLMM

Appropriate data
• Two-way data with one treatment variable and one blocking variable
• Dependent variable is ordered factor

346
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Treatment variable is a factor with at least two levels or groups

• Blocking variable is a factor with at least two levels
• Observations can be paired or repeated measures within blocks

Interpretation
A significant result can be interpreted as, “There was a significant difference among groups.”
Or, “There was a significant effect of Independent Variable.”

A significant post-hoc analysis indicates, “There was a significant difference between Group A
and Group B”, and so on.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ordinal
• car
• RVAideMemoire
• multcompView
• lsmeans
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-way repeated ordinal ANOVA example

In this example, we want to determine if there is a difference in scores among five instructors from the
Belcher family. Because we know which rater gave each score, we will use the Rater as a blocking
variable, with the suspicion that one rater might rate consistently higher than another one. If we
hadn’t recorded this information about raters, we could use a one-way ordinal anova.
Input =("
Instructor Rater Likert
'Bob Belcher' a 4
'Bob Belcher' b 5
'Bob Belcher' c 4

347
ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Bob Belcher' d 6
'Bob Belcher' e 6
'Bob Belcher' f 6
'Bob Belcher' g 10
'Bob Belcher' h 6
'Linda Belcher' a 8
'Linda Belcher' b 6
'Linda Belcher' c 8
'Linda Belcher' d 8
'Linda Belcher' e 8
'Linda Belcher' f 7
'Linda Belcher' g 10
'Linda Belcher' h 9
'Tina Belcher' a 7
'Tina Belcher' b 5
'Tina Belcher' c 7
'Tina Belcher' d 8
'Tina Belcher' e 8
'Tina Belcher' f 9
'Tina Belcher' g 10
'Tina Belcher' h 9
'Gene Belcher' a 6
'Gene Belcher' b 4
'Gene Belcher' c 5
'Gene Belcher' d 5
'Gene Belcher' e 6
'Gene Belcher' f 6
'Gene Belcher' g 5
'Gene Belcher' h 5
'Louise Belcher' a 8
'Louise Belcher' b 7
'Louise Belcher' c 8
'Louise Belcher' d 8
'Louise Belcher' e 9
'Louise Belcher' f 9
'Louise Belcher' g 8
'Louise Belcher' h 10
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered=TRUE)

### Check the data frame

348
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Instructor + Likert.f,
data = Data)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 2 1 4 0 0 0 1
Linda Belcher 0 0 1 1 4 1 1
Tina Belcher 0 1 0 2 2 2 1
Gene Belcher 1 4 3 0 0 0 0
Louise Belcher 0 0 0 1 4 2 1

XT = xtabs( ~ Instructor + Likert.f,

data = Data)

prop.table(XT,
margin = 1)

Likert.f
Instructor 4 5 6 7 8 9 10
Bob Belcher 0.250 0.125 0.500 0.000 0.000 0.000 0.125
Linda Belcher 0.000 0.000 0.125 0.125 0.500 0.125 0.125
Tina Belcher 0.000 0.125 0.000 0.250 0.250 0.250 0.125
Gene Belcher 0.125 0.500 0.375 0.000 0.000 0.000 0.000
Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

Bar plots by group

Note that the bar plots don’t show the effect of the blocking variable.
library(lattice)

histogram(~ Likert.f | Instructor,

data=Data,
layout=c(1,5) # columns and rows of individual plots
)

349
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n mean sd min Q1 median Q3 max percZero

1 Bob Belcher 8 5.875 1.885 4 4.75 6 6.00 10 0
2 Linda Belcher 8 8.000 1.195 6 7.75 8 8.25 10 0
3 Tina Belcher 8 7.875 1.553 5 7.00 8 9.00 10 0
4 Gene Belcher 8 5.250 0.707 4 5.00 5 6.00 6 0
5 Louise Belcher 8 8.375 0.916 7 8.00 8 9.00 10 0

One-way repeated ordinal ANOVA

The model is specified using formula notation with the clmm function (not clm). Here, Likert.f is the
dependent variable and Instructor is the independent variable. Rater is specified as the blocking
variable. Technically, it is identified as the random variable in the mixed model. The data= option
indicates the data frame that contains the variables. For the meaning of other options, see ?clmm.

350
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The p-value for the effect of Instructor is determined using the Anova function in the packages
RVAideMemoire and car.

Define model and conduct analysis of deviance

Here the threshold = "equidistant" option is used in order to avoid errors in the optional post-hoc tests.
This option does not need to be used routinely.
library(ordinal)

model = clmm(Likert.f ~ Instructor + (1|Rater),

data = Data,
threshold = "equidistant")

library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Instructor 38.51 4 8.794e-08 ***

Determine the effect of the random variable

The effect of the random variable (Rater, in this case) can be determined by comparing the full model
to a model with only the fixed effects (Instructor, in this case).

First, model.fixed is defined, and then the two models are passed to the anova function.

Be sure that the threshold option in model.fixed is the same as in the original model.
model.fixed = clm(Likert.f ~ Instructor,
data = Data,
threshold = "equidistant")

anova(model,
null = model.fixed)

Likelihood ratio tests of cumulative link models:

no.par AIC logLik LR.stat df Pr(>Chisq)

model.fixed 6 135.19 -61.596
model 7 129.46 -57.732 7.7275 1 0.005439 **

351
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Check model assumptions

At the time of writing, the nominal_test and scale_test functions don’t work with clmm model objects,
so we will define a similar model with clm and no random effects, and test the proportional odds
assumption on this model.
model.clm = clm(Likert.f ~ Instructor + Rater,
data = Data,
threshold = "equidistant")

nominal_test(model.clm)

Tests of nominal effects

Df logLik AIC LRT Pr(>Chi)

<none> -47.032 120.06
Instructor 4 -42.680 119.36 8.7053 0.0689 .
Rater 7 -39.544 119.09 14.9766 0.0363 *

### No violation in assumptions for Instructor.

### We can ignore Rater since it was a random effect in the original model.

scale_test(model.clm)

Warning messages:
1: (-2) Model failed to converge: degenerate Hessian with 1 negative eigenvalues
In addition: maximum number of consecutive Newton modifications reached

### This test failed. It would be possible to adjust the fitting parameters,
### and try to get the model to converge.

Post-hoc comparisons with lsmeans

Because the main effect of Instructor in the model was significant, we want to know which instructor
has statistically different scores than which other instructors. Comparing LS means for each instructor
will allow us to do this.

Be sure to read the Least Square Means for Multiple Comparisons chapter for correct interpretation of
least square means. For clmm model objects, the lsmean, SE, LCL, and UCL values should be ignored,
unless specific options in lsmeans are selected.

Here we’ll create an object of the lsmeans output called leastsquare. Then we’ll pass this object to the
cld function to create a compact letter display.

The ordinal model under consideration is called model, created the clmm function above. The formula
in the lsmeans function indicates that pairwise comparisons should be conducted for the variable
Instructor.
library(multcompView)

library(lsmeans)

352
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust="tukey") ### Tukey-adjusted comparisons

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

Instructor lsmean SE df asymp.LCL asymp.UCL .group

Gene Belcher -3.957035 0.9610526 NA -6.4257521 -1.4883188 a
Bob Belcher -2.893186 0.8879872 NA -5.1742153 -0.6121576 a
Tina Belcher 2.018826 0.7027604 NA 0.2136007 3.8240505 b
Linda Belcher 2.378266 0.7417485 NA 0.4728901 4.2836425 b
Louise Belcher 3.329647 0.8470188 NA 1.1538560 5.5054375 b

Results are averaged over the levels of: Rater

Confidence level used: 0.95
Conf-level adjustment: sidak method for 5 estimates
P value adjustment: tukey method for comparing a family of 5 estimates
significance level used: alpha = 0.05

### Groups sharing a letter in .group are not significantly different

### Remember to ignore “lsmean”, “SE”, “LCL”, and “UCL” for CLMM

Optional analyses
Post-hoc test: pairwise paired ordinal tests for multiple comparisons
The post-hoc analysis can be accomplished with the pairwiseOrdinalPairedTest or
pairwiseOrdinalPairedMatrix functions. These functions extract the p-values from paired cumulative
link models for each pair of treatments.

Usually you would want to first order your treatment variable (Speaker) by median or other location
statistic for the letters in the compact letter display to be in their proper order.

Note that for this function, the data must be ordered by the blocking variable so that the first
observation where Instructor = Bob is paired to the first observation where Instructor = Linda, and so
on.

Here the p-values for the pairwise comparisons are adjusted with the fdr method. See ?p.adjust for
options and details on p-value adjustment methods.

Here the threshold = "equidistant" option is used in order to avoid errors. This option does not need to
be used routinely.
### Order groups by median

353
 ONE-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Instructor = factor(Data$Instructor,
levels = c("Linda Belcher", "Louise Belcher",
"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise ordinal regression tests

library(rcompanion)

PT = pairwiseOrdinalPairedTest(x = Data$Likert.f,
g = Data$Instructor,
b = Data$Rater,
threshold = "equidistant",
method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Comparison p.value p.adjust

1 Linda Belcher - Louise Belcher = 0 0.3565 4.456e-01
2 Linda Belcher - Tina Belcher = 0 0.5593 5.593e-01
3 Linda Belcher - Bob Belcher = 0 0.00231 3.850e-03
4 Linda Belcher - Gene Belcher = 0 1.485e-05 7.425e-05
5 Louise Belcher - Tina Belcher = 0 0.2347 3.353e-01
6 Louise Belcher - Bob Belcher = 0 0.001071 2.142e-03
7 Louise Belcher - Gene Belcher = 0 1.247e-07 1.247e-06
8 Tina Belcher - Bob Belcher = 0 0.001013 2.142e-03
9 Tina Belcher - Gene Belcher = 0 0.0001283 4.277e-04
10 Bob Belcher - Gene Belcher = 0 0.5152 5.593e-01

### Compact letter display

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 LindaBelcher a a
2 LouiseBelcher a a
3 TinaBelcher a a
4 BobBelcher b b
5 GeneBelcher b b

Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix output and compact letter display

### Order groups by median

Data$Instructor = factor(Data$Instructor,

354
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

levels = c("Linda Belcher", "Louise Belcher",

"Tina Belcher", "Bob Belcher",
"Gene Belcher"))

### Pairwise ordinal regression tests

library(rcompanion)

PM = pairwiseOrdinalPairedMatrix(x = Data$Likert.f,
g = Data$Instructor,
b = Data$Rater,
threshold="equidistant",
method="fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PM

$Adjusted
Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
Linda Belcher 1.000e+00 4.456e-01 0.5593000 0.003850 7.425e-05
Louise Belcher 4.456e-01 1.000e+00 0.3353000 0.002142 1.247e-06
Tina Belcher 5.593e-01 3.353e-01 1.0000000 0.002142 4.277e-04
Bob Belcher 3.850e-03 2.142e-03 0.0021420 1.000000 5.593e-01
Gene Belcher 7.425e-05 1.247e-06 0.0004277 0.559300 1.000e+00

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
"a" "a" "a" "b" "b"

### Values sharing a letter are not significantly different

Two-way Ordinal ANOVA with CLM

A two-way ordinal analysis of variance can address an experimental design with two independent
variables, each of which is a factor variable. The main effect of each independent variable can be
tested, as well as the effect of the interaction of the two factors.

The example here looks at ratings for three instructors across four different questions. The analysis
will attempt to answer the questions: a) Is there a significant difference in scores for different

355
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

instructors? b) Is there a significant difference in scores for different questions? c) Is there a

significant interaction effect of Instructor and Question?

Main effects and interaction effects are explained further in the Factorial ANOVA: Main Effects,
Interaction Effects, and Interaction Plots chapter.

The clm function can specify more complex models with multiple independent variables of different
types, but this book will not explore more complex examples.

The p-values for the main and interaction effects can be determined with the Anova function from
RVAideMemoire, which produces an analysis of deviance table for these effects. In addition, a p-value
for the model as a whole will be determined, along with a pseudo R-squared for the model as a whole.

Post-hoc analysis to determine which groups are different can be conducted on each significant main
effect and on the interaction effect if it is significant.

Appropriate data
• Two-way data
• Dependent variable is ordered factor
• Independent variables are factors with at least two levels or groups each
• Observations between groups are not paired or repeated measures

Interpretation
A significant main effect can be interpreted as, “There was a significant difference among
groups.” Or, “There was a significant effect of Independent Variable.”

A significant interaction effect can be interpreted as, “There was a significant interaction effect
between Independent Variable A and Independent Variable B.”

A significant post-hoc analysis indicates, “There was a significant difference between Group A
and Group B”, and so on.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ggplot2
• ordinal
• car
• RVAideMemoire
• multcompView
• lsmeans
• rcompanion

356
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(rcompanion)){install.packages("rcompanion")}

Two-way ordinal ANOVA example

Input =("
Instructor Question Likert
Fuu Informative 8
Fuu Informative 9
Fuu Informative 9
Fuu Informative 8
Fuu Delivery 10
Fuu Delivery 9
Fuu Delivery 8
Fuu Delivery 8
Fuu VisualAides 7
Fuu VisualAides 7
Fuu VisualAides 6
Fuu VisualAides 7
Fuu AnswerQuest 8
Fuu AnswerQuest 9
Fuu AnswerQuest 9
Fuu AnswerQuest 8
Jin Informative 7
Jin Informative 8
Jin Informative 6
Jin Informative 5
Jin Delivery 8
Jin Delivery 8
Jin Delivery 9
Jin Delivery 6
Jin VisualAides 5
Jin VisualAides 6
Jin VisualAides 7
Jin VisualAides 7
Jin AnswerQuest 8
Jin AnswerQuest 7
Jin AnswerQuest 6
Jin AnswerQuest 6
Mugen Informative 5
Mugen Informative 4
Mugen Informative 3
Mugen Informative 4

357
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Mugen Delivery 8
Mugen Delivery 9
Mugen Delivery 8
Mugen Delivery 7
Mugen VisualAides 5
Mugen VisualAides 4
Mugen VisualAides 4
Mugen VisualAides 5
Mugen AnswerQuest 6
Mugen AnswerQuest 7
Mugen AnswerQuest 6
Mugen AnswerQuest 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Instructor + Likert.f,
data = Data)

Likert.f
Instructor 3 4 5 6 7 8 9 10
Fuu 0 0 0 1 3 6 5 1
Jin 0 0 2 5 4 4 1 0

358
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Mugen 1 4 3 2 3 2 1 0

xtabs( ~ Question + Likert.f,

data = Data)

Likert.f
Question 3 4 5 6 7 8 9 10
AnswerQuest 0 0 0 4 3 3 2 0
Delivery 0 0 0 1 1 6 3 1
Informative 1 2 2 1 1 3 2 0
VisualAides 0 2 3 2 5 0 0 0

xtabs( ~ Instructor + Likert.f + Question,

data = Data)

, , Question = AnswerQuest

Likert.f
Instructor 3 4 5 6 7 8 9 10
Fuu 0 0 0 0 0 2 2 0
Jin 0 0 0 2 1 1 0 0
Mugen 0 0 0 2 2 0 0 0

, , Question = Delivery

Likert.f
Instructor 3 4 5 6 7 8 9 10
Fuu 0 0 0 0 0 2 1 1
Jin 0 0 0 1 0 2 1 0
Mugen 0 0 0 0 1 2 1 0

, , Question = Informative

Likert.f
Instructor 3 4 5 6 7 8 9 10
Fuu 0 0 0 0 0 2 2 0
Jin 0 0 1 1 1 1 0 0
Mugen 1 2 1 0 0 0 0 0

, , Question = VisualAides

Likert.f
Instructor 3 4 5 6 7 8 9 10
Fuu 0 0 0 1 3 0 0 0
Jin 0 0 1 1 2 0 0 0
Mugen 0 2 2 0 0 0 0 0

Bar plots by group

library(lattice)

histogram(~ Likert.f | Instructor,

359
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

library(lattice)

histogram(~ Likert.f | Question,

data=Data,
layout=c(1,4) # columns and rows of individual plots
)

360
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(lattice)

histogram(~ Likert.f | Instructor + Question,

data=Data,
layout=c(3,4) # columns and rows of individual plots
)

361
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n mean sd min Q1 median Q3 max percZero

1 Fuu 16 8.125 1.025 6 7.75 8.0 9 10 0
2 Jin 16 6.812 1.167 5 6.00 7.0 8 9 0
3 Mugen 16 5.750 1.770 3 4.00 5.5 7 9 0

library(FSA)

Summarize(Likert ~ Question,
data=Data,
digits=3)

Question n mean sd min Q1 median Q3 max percZero

1 AnswerQuest 12 7.250 1.138 6 6.00 7.0 8 9 0

362
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

2 Delivery 12 8.167 1.030 6 8.00 8.0 9 10 0

3 Informative 12 6.333 2.103 3 4.75 6.5 8 9 0
4 VisualAides 12 5.833 1.193 4 5.00 6.0 7 7 0

library(FSA)

Summarize(Likert ~ Instructor + Question,

data=Data,
digits=3)

Instructor Question n mean sd min Q1 median Q3 max percZero

1 Fuu AnswerQuest 4 8.50 0.577 8 8.00 8.5 9.00 9 0
2 Jin AnswerQuest 4 6.75 0.957 6 6.00 6.5 7.25 8 0
3 Mugen AnswerQuest 4 6.50 0.577 6 6.00 6.5 7.00 7 0
4 Fuu Delivery 4 8.75 0.957 8 8.00 8.5 9.25 10 0
5 Jin Delivery 4 7.75 1.258 6 7.50 8.0 8.25 9 0
6 Mugen Delivery 4 8.00 0.816 7 7.75 8.0 8.25 9 0
7 Fuu Informative 4 8.50 0.577 8 8.00 8.5 9.00 9 0
8 Jin Informative 4 6.50 1.291 5 5.75 6.5 7.25 8 0
9 Mugen Informative 4 4.00 0.816 3 3.75 4.0 4.25 5 0
10 Fuu VisualAides 4 6.75 0.500 6 6.75 7.0 7.00 7 0
11 Jin VisualAides 4 6.25 0.957 5 5.75 6.5 7.00 7 0
12 Mugen VisualAides 4 4.50 0.577 4 4.00 4.5 5.00 5 0

Produce interaction plot with medians and quantiles

library(FSA)

Sum = Summarize(Likert ~ Instructor + Question,

data=Data,
digits=3)

Sum

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x=Instructor,
y=median,
color=Question)) +
geom_errorbar(aes(ymin=Q1,
ymax=Q3),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold"))+
ylab("Median Likert score")

363
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Two-way ordinal ANOVA

In the model notation in the clm function, here, Likert.f is the dependent variable and Instructor and
Question are the independent variables. The term Instructor:Question adds the interaction effect of
these two independent variables to the model. The data= option indicates the data frame that contains
the variables. For the meaning of other options, see ?clm.

The p-values for the two main effects and the interaction effect is determined using the Anova function
in the packages RVAideMemoire and car.

Here the threshold = "symmetric" option is used in order to avoid errors. This option does not need to
be used routinely.

Define model and analyses of deviance

library(ordinal)

model = clm(Likert.f ~ Instructor + Question + Instructor:Question,

data = Data,
threshold="symmetric")

library(car)

library(RVAideMemoire)

364
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Instructor 32.157 2 1.040e-07 ***
Question 28.248 3 3.221e-06 ***
Instructor:Question 24.326 6 0.0004548 ***

p-value for model and pseudo R-squared

The p-value for the model and a pseudo R-squared value can be determined with the nagelkerke
function.
library(rcompanion)

nagelkerke(model)

$Pseudo.R.squared.for.model.vs.null

Pseudo.R.squared
McFadden 0.400602
Cox and Snell (ML) 0.775956
Nagelkerke (Cragg and Uhler) 0.794950

$Likelihood.ratio.test

Df.diff LogLik.diff Chisq p.value

-11 -35.902 71.804 5.5398e-11

Post-hoc tests with lsmeans for multiple comparisons of groups

Be sure to read the Least Square Means for Multiple Comparisons chapter for correct interpretation of
least square means. For clm model objects, the lsmean, SE, LCL, and UCL values should be ignored,
unless specific options in lsmeans are selected.

Because the interaction term in the model was significant, the group separation for the interaction
effect is explored.
library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor + Question,
adjust="tukey") ### Tukey-adjusted comparisons

leastsquare

cld(leastsquare,
alpha = 0.05,

365
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Letters = letters, ### Use lower-case letters for .group

adjust = "tukey") ### Tukey-adjusted comparisons

Instructor Question lsmean SE df asymp.LCL asymp.UCL .group

Mugen Informative -6.663413e+00 1.4186237 NA -10.7176160 -2.609209 a
Mugen VisualAides -5.262834e+00 1.2789949 NA -8.9180007 -1.607668 ab
Jin VisualAides -4.347138e-01 0.9435048 NA -3.1311021 2.261675 bc
Jin Informative -8.326673e-17 1.0546366 NA -3.0139854 3.013985 bcd
Mugen AnswerQuest 4.718448e-16 0.8484277 NA -2.4246729 2.424673 c
Jin AnswerQuest 4.347138e-01 0.9435048 NA -2.2616745 3.131102 cde
Fuu VisualAides 5.525651e-01 0.8464847 NA -1.8665549 2.971685 cd
Jin Delivery 3.490051e+00 1.3194708 NA -0.2807891 7.260890 cde
Mugen Delivery 3.713121e+00 1.2254534 NA 0.2109685 7.215274 cde
Fuu AnswerQuest 5.262834e+00 1.2789949 NA 1.6076682 8.918001 de
Fuu Informative 5.262834e+00 1.2789949 NA 1.6076682 8.918001 de
Fuu Delivery 5.782817e+00 1.3782347 NA 1.8440397 9.721595 e

Confidence level used: 0.95

Conf-level adjustment: sidak method for 12 estimates
P value adjustment: tukey method for comparing a family of 12 estimates
significance level used: alpha = 0.05

### Groups sharing a letter in .group are not significantly different

### Remember to ignore “lsmean”, “SE”, “LCL”, and “UCL” with CLM

Interaction plot with group separation letters

Group separation letters can be added manually to an interaction plot.

Groups sharing a letter are not significantly different. In this case, because so many groups share a
letter, it is difficult to interpret this plot. One approach would be to look at differences among
instructors for each question. Looking at AnswerQuest, Fuu’s scores are not statistically different than
Jin’s (because they share the letters d and e), but Fuu’s scores are different than Mugen’s (because they
share no letters). So, we can conclude for this question, that Fuu’s scores are significantly greater than
Mugen’s. And so on.

366
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Check model assumptions

nominal_test(model)

Tests of nominal effects

formula: Likert.f ~ Instructor + Question + Instructor:Question

Df logLik AIC LRT Pr(>Chi)
<none> -53.718 137.44
Instructor 6 -49.812 141.62 7.8121 0.2522
Question 9 -50.182 148.37 7.0711 0.6297
Instructor:Question

### No violation in assumptions.

scale_test(model)

Tests of scale effects

formula: Likert.f ~ Instructor + Question + Instructor:Question

Df logLik AIC LRT Pr(>Chi)
<none> -53.718 137.44
Instructor 2 -51.669 137.34 4.0985 0.12883
Question 3 -50.534 137.07 6.3669 0.09506 .
Instructor:Question

367
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Warning message:
(-1) Model failed to converge with max|grad| = 1.70325e-06 (tol = 1e-06)
In addition: maximum number of consecutive Newton modifications reached

### This test failed, but the results suggest no violation of assumptions.

Optional analyses

Post-hoc test: pairwise ordinal tests for multiple comparisons of groups

A different method to compare groups is to conduct pairwise ordinal tests for groups.

Here, the threshold = "flexible" option is used in order to avoid errors for this data set. This option does
not need to be used routinely. Note that potentially important errors are produced in the pairwise
post-hoc tests on the interaction factor. One reason for this is that there are few observations for each
combination of Instructor and Question.

Table format
### Create a new variable for interaction term

Data$Interaction = interaction(Data$Instructor, Data$Question)

Data$Interaction = factor(Data$Interaction,
levels=unique(Data$Interaction))

library(psych)

headTail(Data)

### Order groups by median

Data$Interaction = factor(Data$Interaction,
levels = c("Fuu.AnswerQuest", "Fuu.Delivery",
"Fuu.Informative", "Mugen.Delivery",
"Jin.Delivery", "Fuu.VisualAides",
"Mugen.AnswerQuest", "Jin.AnswerQuest",
"Jin.VisualAides"," Jin.Informative",
"Mugen.VisualAides", "Mugen.Informative"))

### Pairwise ordinal regression tests

library(rcompanion)

PT = pairwiseOrdinalTest(x = Data$Likert.f,
g = Data$Interaction,
threshold = "flexible",
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

368
 TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

1 Fuu.AnswerQuest - Fuu.Delivery = 0 0.7335 0.795100

2 Fuu.AnswerQuest - Fuu.Informative = 0 1 1.000000
3 Fuu.AnswerQuest - Mugen.Delivery = 0 0.2996 0.380300

< output snipped >

64 Mugen.VisualAides - Mugen.Informative = 0 0.2996 0.380300

65 Mugen.VisualAides - Jin.Informative = 0 0.007007 0.012500
66 Mugen.Informative - Jin.Informative = 0 0.003926 0.009597

warnings()

Hessian is numerically singular: parameters are not uniquely determined

In addition: Absolute convergence criterion was met, but relative criterion was
not met

### Results may not be reliable for this data set for these pairwise tests

### Compact letter display

library (rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 Fuu.AnswerQuest a a
2 Fuu.Delivery a a
3 Fuu.Informative a a
4 Mugen.Delivery ab ab
5 Jin.Delivery abc abc
6 Fuu.VisualAides c c
7 Mugen.AnswerQuest c c
8 Jin.AnswerQuest bc bc
9 Jin.VisualAides c c
10 Mugen.VisualAides d d
11 Mugen.Informative d d
12 Jin.Informative bc bc

Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix format and compact letter display

Usually you would want to first order your treatment variable (Interaction) by median or other
location statistic for the letters in the compact letter display to be in their proper order.
### Create a new variable for interaction term

Data$Interaction = interaction(Data$Instructor, Data$Question)

369
TWO-WAY ORDINAL ANOVA WITH CLM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Interaction = factor(Data$Interaction,
levels=unique(Data$Interaction))

library(psych)

headTail(Data)

### Order groups by median

Data$Interaction = factor(Data$Interaction,
levels = c("Fuu.AnswerQuest", "Fuu.Delivery",
"Fuu.Informative", "Mugen.Delivery",
"Jin.Delivery", "Fuu.VisualAides",
"Mugen.AnswerQuest", "Jin.AnswerQuest",
"Jin.VisualAides"," Jin.Informative",
"Mugen.VisualAides", "Mugen.Informative"

### Pairwise ordinal regression tests

library(rcompanion)

PM = pairwiseOrdinalMatrix(x = Data$Likert.f,
g = Data$Interaction,
threshold = "flexible",
method = "fdr")
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PM

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Fuu.AnswerQuest Fuu.Delivery Fuu.Informative Mugen.Delivery

"a" "a" "a" "ab"

Jin.Delivery Fuu.VisualAides
"abc" "c"

Mugen.AnswerQuest Jin.AnswerQuest Jin.VisualAides Mugen.VisualAides

"c" "bc" "c" "d"

Mugen.Informative Jin.Informative
"d" "bc"

### Values sharing a letter are not significantly different

370
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

warnings()

Hessian is numerically singular: parameters are not uniquely determined

In addition: Absolute convergence criterion was met, but relative criterion was
not met

### Results may not be reliable for this data set for these pairwise tests

Interaction plot with separation letters

Two-way Repeated Ordinal ANOVA with CLMM

A two-way repeated ordinal analysis of variance can address an experimental design with two
independent variables, each of which is a factor variable, plus a blocking variable. The main effect of
each independent variable can be tested, as well as the effect of the interaction of the two factors.

The example here looks at students’ knowledge scores for three speakers across two different times.
Because we know which student has each score at each time, we will use the Student as a blocking

371
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

variable, with the suspicion that one student might have consistently lower knowledge than another
student. If we hadn’t recorded this information about students, we could use a two-way ordinal anova.

As a matter of practical interpretation, we may not care about the absolute scores for each of the three
speakers, only if the knowledge scores for students increased from Time 1 to Time 2. At Time 1,
students’ knowledge scores are lower for Piglet that for the other two speakers. This isn’t the fault of
Piglet; he is just speaking about a topic that the students have little knowledge of.

The clmm function can specify more complex models with multiple independent variables of different
types, but this book will not explore more complex examples.

The p-values for the main and interaction effects can be determined with the Anova function from
RVAideMemoire, which produces an analysis of deviance table for these effects. In addition, a p-value
for the model as a whole will be determined, along with a pseudo R-squared for the model as a whole.

Post-hoc analysis to determine which groups are different can be conducted on each significant main
effect and on the interaction effect if it is significant.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice
• ggplot2
• ordinal
• car
• RVAideMemoire
• lsmeans
• multcompView
• plyr
• boot
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(ordinal)){install.packages("ordinal")}
if(!require(car)){install.packages("car")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(plyr)){install.packages("plyr")}
if(!require(boot)){install.packages("boot")}
if(!require(rcompanion)){install.packages("rcompanion")}

372
TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Two-way repeated ordinal ANOVA example

Input =("
Speaker Student Time Likert
Pooh a 1 3
Pooh b 1 5
Pooh c 1 4
Pooh d 1 4
Pooh e 1 4
Pooh f 1 3
Pooh g 1 4
Pooh h 1 4

Pooh a 2 4
Pooh b 2 5
Pooh c 2 5
Pooh d 2 5
Pooh e 2 5
Pooh f 2 4
Pooh g 2 5
Pooh h 2 5

Piglet i 1 1
Piglet j 1 2
Piglet k 1 1
Piglet l 1 1
Piglet m 1 1
Piglet n 1 2
Piglet o 1 3
Piglet p 1 1

Piglet i 2 2
Piglet j 2 4
Piglet k 2 2
Piglet l 2 1
Piglet m 2 2
Piglet n 2 2
Piglet o 2 4
Piglet p 2 2

Eeyore q 1 4
Eeyore r 1 5
Eeyore s 1 4
Eeyore t 1 4
Eeyore u 1 4
Eeyore v 1 4
Eeyore w 1 4
Eeyore x 1 4

Eeyore q 2 5
Eeyore r 2 4
Eeyore s 2 4

373
TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Eeyore t 2 4
Eeyore u 2 3
Eeyore v 2 4
Eeyore w 2 4
Eeyore x 2 4
")

Data = read.table(textConnection(Input),header=TRUE)

### Change Time into a factor variable

Data$Time = factor(Data$Time)

### Order levels of the factor; otherwise R will alphabetize them

Data$Speaker = factor(Data$Speaker,
levels=unique(Data$Speaker))

### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
ordered = TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data treating Likert scores as factors

xtabs( ~ Time + Likert.f + Speaker,
data = Data)

Likert.f
Time 1 2 3 4 5
1 0 0 2 5 1
2 0 0 0 2 6

, , Speaker = Piglet

374
TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Likert.f
Time 1 2 3 4 5
1 5 2 1 0 0
2 1 5 0 2 0

, , Speaker = Eeyore

Likert.f
Time 1 2 3 4 5
1 0 0 0 7 1
2 0 0 1 6 1

Bar plots by group

library(lattice)

histogram(~ Likert.f | Speaker,

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

library(lattice)

histogram(~ Likert.f | Time,

data=Data,
layout=c(1,2) # columns and rows of individual plots
)

375
TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(lattice)

histogram(~ Likert.f | Time + Speaker,

data=Data,
layout=c(2,3) # columns and rows of individual plots
)

Summarize data treating Likert scores as numeric

library(FSA)

376
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Summarize(Likert ~ Speaker + Time,

data=Data,
digits=3)

Speaker Time n nvalid mean sd min Q1 median Q3 max percZero

1 Pooh 1 8 8 3.875 0.641 3 3.75 4 4.0 5 0
2 Piglet 1 8 8 1.500 0.756 1 1.00 1 2.0 3 0
3 Eeyore 1 8 8 4.125 0.354 4 4.00 4 4.0 5 0
4 Pooh 2 8 8 4.750 0.463 4 4.75 5 5.0 5 0
5 Piglet 2 8 8 2.375 1.061 1 2.00 2 2.5 4 0
6 Eeyore 2 8 8 4.000 0.535 3 4.00 4 4.0 5 0

Plot Likert scores before and after

Since we are most interested in pairing scores from Time 1 to Time 2, we can plot the data in a way
that pairs points.

Points above and to left of the blue line indicate cases in which the score for Time 2 is greater than that
for Time 1.

Note that the data must be ordered by Student and Speaker for this code to plot the data correctly.
That is, the first observation for Time==1 must be the same student and speaker as the first
observation for Time==2.

Also note that the points on the plot are jittered so that multiple points with the same values will be
visible.
Time.1 = Data$Likert[Data$Time==1]
Time.2 = Data$Likert[Data$Time==2]
Speaker = Data$Speaker[Data$Time==2]

plot(Time.1, jitter(Time.2),
pch = 16,
cex = 1,
col = Speaker,
xlab="Time 1",
ylab="Time 2")

abline(0,1, col="blue", lwd=2)

legend('bottomright',
legend = unique(Speaker),
col = 1:3,
cex = 1,
pch = 16)

377
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Produce interaction plot with medians and confidence intervals

The following plot will show the interaction between Speaker and Time. The median for each Speaker
x Time combination is plotted, and the confidence intervals for each are shown with error bars. In this
case some of the medians and confidence limits are the same values, so some error bars are not visible.

The plot makes it easy to see the change in median score for each speaker from Time 1 to Time 2.
library(rcompanion)

Sum = groupwiseMedian(data = Data,

group = c("Speaker", "Time"),
var = "Likert",
conf = 0.95,
R = 5000,
percentile = TRUE,
bca = FALSE,
digits = 3)
Sum

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x=Time,
y=Median,
color=Speaker)) +
geom_errorbar(aes(ymin=Percentile.lower,
ymax=Percentile.upper),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +

378
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

theme(axis.title = element_text(face = "bold")) +

ylab("Median Likert score")

Two-way repeated ordinal ANOVA

In the model notation in the clmm function, here, Likert.f is the dependent variable and Speaker and
Time are the independent variables. The term Time:Spreaker adds the interaction effect of these two
independent variables to the model. Student is used as a blocking variable, and is entered as a random
variable. The data= option indicates the data frame that contains the variables. For the meaning of
other options, see ?clmm.

The p-values for the two main effects and the interaction effect is determined using the Anova function
in the packages RVAideMemoire and car.

Here the threshold = "equidistant" option is used in order to avoid errors for this data set. This option
does not need to be used routinely.

Optional technical note

In the model below Student is treated as a simple random blocking variable, specified by (1|Student).
Note that in the data, students were lettered a–x straight through the Speakers. If the students had
been lettered a–h for each Speaker, but there were actually 24 students, we would need to tell the
model that Student was nested within Speaker, by specifying that the random effect was
(1|Speaker/Student), or that the random effect of interest was the interaction of Speaker and Student,
by specifying (1|Speaker:Student). This second specification is equivalent to that in the code below,
with the original data, and will produce the same results.

379
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

I consider it a good practice to assign unique individuals with unique labels. This practice is not
always followed, but following the practice will avoid mis-specifying models. In this case with 24
students, if they had been lettered a–h for each Speaker, and the original model with the (1|Student)
effect had been used, the results would be wrong because the model would treat Pooh’s student a as
the same individual as Piglet’s student a.

Define model and conduct analysis of deviance

library(ordinal)

model = clmm(Likert.f ~ Time + Speaker + Time:Speaker + (1|Student),

data = Data,
threshold = "equidistant")

library(car)

library(RVAideMemoire)

Anova(model,
type = "II")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Time 9.774 1 0.00177 **
Speaker 34.485 2 3.249e-08 ***
Time:Speaker 28.202 2 7.517e-07 ***

p-value for model and pseudo R-squared

In order to use the nagelkerke function for a clmm model object, a null model has to be specified
explicitly. (This is not the case with a clm model object).

We want the null model to have no fixed effects except for an intercept, indicated with a 1 on the right
side of the ~. But we also may want to include the random effects, in this case (1 | Student).

We can compare our full model against a model with an overall intercept and an intercept for each
student.

Be sure that the threshold option in model.null is the same as in the original model.
model.null = clmm(Likert.f ~ 1 + (1 | Student),
data = Data,
threshold = "equidistant")

library(rcompanion)

nagelkerke(fit = model,
null = model.null)

$Pseudo.R.squared.for.model.vs.null

380
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Pseudo.R.squared
McFadden 0.544198
Cox and Snell (ML) 0.787504
Nagelkerke (Cragg and Uhler) 0.836055

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-5 -37.172 74.344 1.275e-14

Another approach to determining the p-value and pseudo R-squared for a clmm model is comparing
the model to a null model with only an intercept and neither the fixed nor the random effects.
model.null.2 = clm(Likert.f ~ 1,
data = Data,
threshold = "equidistant")

library(rcompanion)

nagelkerke(fit = model,
null = model.null.2)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.587734
Cox and Snell (ML) 0.842667
Nagelkerke (Cragg and Uhler) 0.880526

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-6 -44.385 88.771 5.4539e-17

Determine the effect of the random variable

The effect of the random variable (Student, in this case) can be determined by comparing the full
model to a model with only the fixed effects (Speaker, Time, and Speaker:Time in this case).

First, model.fixed is defined, and then the two models are passed to the anova function.

Be sure that the threshold option in model.fixed is the same as in the original model.
model.fixed = clm(Likert.f ~ Time + Speaker + Time:Speaker,
data = Data,
threshold = "equidistant")

anova(model,
model.fixed)

Likelihood ratio tests of cumulative link models:

no.par AIC logLik LR.stat df Pr(>Chisq)

model.fixed 7 101.197 -43.598
model 8 78.268 -31.134 24.928 1 5.95e-07 ***

381
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Post-hoc tests with lsmeans for multiple comparisons of groups

Be sure to read the Least Square Means for Multiple Comparisons chapter for correct interpretation of
least square means. For clmm model objects, the lsmean, SE, LCL, and UCL values should be ignored,
unless specific options in lsmeans are selected. See ?lsmeans::models for more information.

Because the interaction term in the model was significant, the group separations for the interaction is
explored.
library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Speaker + Time,
adjust="tukey") ### Tukey-adjusted comparisons
leastsquare

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lower-case letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Speaker Time lsmean SE df asymp.LCL asymp.UCL .group

Piglet 1 -36.08909 0.002735251 NA -36.09629 -36.08190 a
Piglet 2 -17.88982 0.003252687 NA -17.89838 -17.88127 b
Eeyore 2 17.46501 2.047095879 NA 12.07902 22.85100 c
Pooh 1 18.19627 0.002535027 NA 18.18960 18.20294 c
Eeyore 1 19.30751 2.030625019 NA 13.96486 24.65016 c
Pooh 2 36.72566 0.002910484 NA 36.71800 36.73332 d

Confidence level used: 0.95

Conf-level adjustment: sidak method for 6 estimates
P value adjustment: tukey method for comparing a family of 6 estimates
significance level used: alpha = 0.05

### Groups sharing a letter in .group are not significantly different

### Remember to ignore “lsmean”, “LCL”, “LCL”, and “UCL” with CLMM

Focusing on meaningful comparisons

Because we wanted to focus on the comparing each speaker at Time1 and Time 2, let’s rearrange the
lsmeans cld table to focus on these comparisons.

With the results in this format, it is easy to see that the scores for both Pooh and Piglet improved from
Time 1 to Time 2, but that the scores for poor Eeyore did not.

Speaker Time lsmean SE df asymp.LCL asymp.UCL .group

Pooh 1 18.19627 0.002535027 NA 18.18960 18.20294 c
Pooh 2 36.72566 0.002910484 NA 36.71800 36.73332 d

382
 TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Speaker Time lsmean SE df asymp.LCL asymp.UCL .group

Piglet 1 -36.08909 0.002735251 NA -36.09629 -36.08190 a
Piglet 2 -17.88982 0.003252687 NA -17.89838 -17.88127 b

Speaker Time lsmean SE df asymp.LCL asymp.UCL .group

Eeyore 1 19.30751 2.030625019 NA 13.96486 24.65016 c
Eeyore 2 17.46501 2.047095879 NA 12.07902 22.85100 c

Interaction plot with group separation letters

Group separation letters can be added manually to the plot of medians.

Check model assumptions

At the time of writing, the nominal_test and scale_test functions don’t work with clmm model objects,
so we will define a similar model with clm and no random effects, and test the proportional odds
assumption on this model.
model.clm = clm(Likert.f ~ Time + Speaker + Time:Speaker + Student,
data = Data,
threshold = "equidistant")

Warning message:
(1) Hessian is numerically singular: parameters are not uniquely determined
In addition: Absolute convergence criterion was met, but relative criterion was
not met.

### The fitting of this model failed. We’ll try a model without student.

383
TWO-WAY REPEATED ORDINAL ANOVA WITH CLMM SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

model.clm = clm(Likert.f ~ Time + Speaker + Time:Speaker,

data = Data,
threshold = "equidistant")

nominal_test(model.clm)

Tests of nominal effects

formula: Likert.f ~ Time + Speaker + Time:Speaker

Df logLik AIC LRT Pr(>Chi)
<none> -43.598 101.197
Time 1 -43.149 102.298 0.8983 0.34325
Speaker 2 -40.053 98.106 7.0903 0.02886 *
Time:Speaker 5 -39.393 102.786 8.4110 0.13499

### There is a violation of proportional odds assumption for this model.

### Results from this model fitting may not be reliable.

scale_test(model.clm)

Warning messages:
1: (-1) Model failed to converge with max|grad| = 2.26258e-05 (tol = 1e-06)
In addition: iteration limit reached

### This test failed. It would be possible to adjust the fitting parameters,
### and try to get the model to converge.

384
 INTRODUCTION TO TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Tests for Nominal Data

Introduction to Tests for Nominal Variables

The tests for nominal variables presented in this book are commonly used to determine if there is an
association between two nominal variables or if counts of observations for a nominal variable match a
theoretical set of proportions for that variable.

Tests of symmetry, or marginal homogeneity, can determine if frequencies for one nominal variable
are greater than that for another, or if there was a change in frequencies from sampling at one time to
another.

As a more advanced approach, models can be specified with nominal dependent variables. A common
type of model with a nominal dependent variable is logistic regression.

Descriptive statistics and plots for nominal data

Descriptive statistics for nominal data are discussed in the “Descriptive statistics for nominal data”
section in the Descriptive Statistics chapter.

Descriptive plots for nominal data are discussed in the “Examples of basic plots for nominal data”
section in the Basic Plots chapter.

Contingency tables

Nominal data are often arranged in a contingency table of counts of observations for each cell of the
table. For example, if there were 6 males and 4 females reading Sappho, 5 males and 4 females
reading Steven Crane, and 7 males and 10 females reading Judith Viorst, the data could be arranged as:
Sex
Male Female
Poet
Sappho 6 4
Crane 3 4
Viorst 2 5

This table can be read into R in the following manner.

Input =("
Poet Male Female
Sappho 6 4
Crane 3 4
Viorst 2 5
")

Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix

385
 INTRODUCTION TO TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Male Female
Sappho 6 4
Crane 3 4
Viorst 2 5

It is helpful to look at totals for columns and rows.

colSums(Matrix)

Male Female
11 13

rowSums(Matrix)

Sappho Crane Viorst

10 7 7

Bar plots
Simple bar charts and mosaic plots are also helpful.
barplot(Matrix,
beside = TRUE,
legend = TRUE,
ylim = c(0, 8), ### y-axis: used to prevent legend overlapping bars
cex.names = 0.8, ### Text size for bars
cex.axis = 0.8, ### Text size for axis
args.legend = list(x = "topright", ### Legend location
cex = 0.8, ### Legend text size
bty = "n")) ### Remove legend box

Matrix.t = t(Matrix) ### Transpose Matrix for the next plot

386
 INTRODUCTION TO TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

barplot(Matrix.t,
beside = TRUE,
legend = TRUE,
ylim = c(0, 8), ### y-axis: used to prevent legend overlapping bars
cex.names = 0.8, ### Text size for bars
cex.axis = 0.8, ### Text size for axis
args.legend = list(x = "topright", ### Legend location
cex = 0.8, ### Legend text size
bty = "n")) ### Remove legend box

Mosaic plots
Mosaic plots are very useful for visualizing the association between two nominal variables, but can be
somewhat tricky to interpret for those unfamiliar with them. Note that the column width is
determined by the number of observation in that category. In this case, the Sappho column is wider
because more students are reading Sappho than the other two poets. Note, too, that the number of
observations in each cell is determined by the area of the cell, not its height. In this case, the Sappho–
Female cell and the Crane–Female cell have the same count (4), and so the same area. The Crane–
Female cell is taller than the Sappho–Female because it is a higher proportion of observations for that
author (4 out of 7 Crane readers compared with 4 out of 10 Sappho readers).
mosaicplot(Matrix,
color=TRUE,
cex.axis=0.8)

387
 INTRODUCTION TO TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional analyses: converting long-format data to a matrix

In R, most simple analyses for nominal data expect the data to be in a matrix format. However, data
may be in a long format, either with each row representing a single observation, or with each row
containing a count of observations.

It is relatively easy to convert long-format data to a matrix. The xtabs function will produce a table
that can be used for most functions expecting data to be formatted as a matrix object.

Long-format with each row as an observations

Input =("
Poet Sex
Sappho Male
Sappho Male
Sappho Male
Sappho Male
Sappho Male
Sappho Male
Sappho Female
Sappho Female
Sappho Female
Sappho Female
Crane Male
Crane Male
Crane Male
Crane Female
Crane Female
Crane Female
Crane Female
Viorst Male
Viorst Male
Viorst Female
Viorst Female
Viorst Female
Viorst Female
Viorst Female

388
INTRODUCTION TO TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, xtabs will alphabetize them

Data$Poet = factor(Data$Poet,
levels=unique(Data$Poet))

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Table = xtabs(~ Poet + Sex, data=Data)

Table

Sex
Poet Male Female
Sappho 6 4
Crane 3 4
Viorst 2 5

Long-format with counts of observations

Input =("
Poet Sex Count
Sappho Male 6
Sappho Female 4
Crane Male 3
Crane Female 4
Viorst Male 2
Viorst Female 5
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, xtabs will alphabetize them

Data$Poet = factor(Data$Poet,
levels=unique(Data$Poet))

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Table = xtabs(Count ~ Poet + Sex, data=Data)

Table

Sex
Poet Male Female

389
 CONFIDENCE INTERVALS FOR PROPORTIONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Sappho 6 4
Crane 3 4
Viorst 2 5

Optional analyses: obtaining information about a matrix object

Matrix

Male Female
Sappho 6 4
Crane 3 4
Viorst 2 5

class(Matrix)

[1] "matrix"

typeof(Matrix)

[1]"integer"

attributes(Matrix)

$dim
[1] 3 2

$dimnames

$dimnames[[1]]
[1] "Sappho" "Crane" "Viorst"

$dimnames[[2]]
[1] "Male" "Female"

str(Matrix)

int [1:3, 1:2] 6 3 2 4 4 5

- attr(*, "dimnames")=List of 2
..$ : chr [1:3] "Sappho" "Crane" "Viorst"
..$ : chr [1:2] "Male" "Female"

Confidence Intervals for Proportions

A binomial proportion has counts for two levels of a nominal variable. An example would be counts of
students of only two sexes, male and female. If there are 20 students in a class, and 12 are female, then

390
 CONFIDENCE INTERVALS FOR PROPORTIONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

the proportion of females are 12/20, or 0. 6, and the proportion of males are 8/20 or 0.4. This is a
binomial proportion.
Sex Count Proportion
Female 12 0.60
Male 8 0.40
------ -- -----------
Total 20 1.00

A multinomial proportion has counts for more than two levels of a nominal variable. For example, we
might have the following levels and counts for sex for students in a class:
Sex Count Proportion
Female 12 0.60
Male 6 0.30
Other 1 0.05
Prefer not to answer 1 0.05
-------------------- -- -----------
Total 20 1.00

Confidence intervals can be produced for either binomial or multinomial proportions.

The binom.test function in the native stats package will provide the Clopper-Pearson confidence
interval for a binomial proportion. Other methods for a binomial proportion are provided by the
BinomCI function in the DescTools package, as well as by various functions in the PropCIs package.

Confidence intervals for multinomial proportions can be produced with the MultinomCI function in the
DescTools package.

Packages used in this chapter

The packages used in this chapter include:

• DescTools
• PropCIs

The following commands will install these packages if they are not already installed:
if(!require(DescTools)){install.packages("DescTools")}
if(!require(PropCIs)){install.packages("PropCIs")}

Example of confidence intervals for a binomial proportion

As part of a demographic survey of her scrapbooking 4-H course, Seras Victoria asks students if they have
ever done scrapbooking before. The following are the data from her course:
Experience Count
Yes 7
No 14

391
 CONFIDENCE INTERVALS FOR PROPORTIONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

---------- --
Total 21

Note that when calculating confidence intervals for a binomial variable, one level of the nominal
variable is chosen to be the “success” level. This is an arbitrary decision, but you should be cautious to
remember that the confidence interval is reported for the proportion of “success” responses. The
BinomCI function in the DescTools package can produce the confidence interval for both “success” and
“failure” in one step.

The binom.test function output includes a confidence interval for the proportion, and the proportion of
“success” as a decimal number. The binom.test function uses the Clopper–Pearson method for
confidence intervals.

binom.test(7, 21,
0.5,
alternative="two.sided",
conf.level=0.95)

95 percent confidence interval:

0.1458769 0.5696755

sample estimates:
probability of success
0.3333333

The BinomCI function in the DescTools package has several methods for calculating confidence
intervals for a binomial proportion.

library(DescTools)

BinomCI(7, 21,
conf.level = 0.95,
method = "clopper-pearson")

### Methods: "wilson", "wald", "agresti-coull", "jeffreys",

### "modified wilson", "modified jeffreys",
### "clopper-pearson", "arcsine", "logit", "witting", "pratt"

est lwr.ci upr.ci

[1,] 0.3333333 0.1458769 0.5696755

The BinomCI function in the DescTools package can also produce the confidence intervals for “success”
and “failure” in one step.
library(DescTools)

observed = c(7, 14)

total = sum(observed)

392
 CONFIDENCE INTERVALS FOR PROPORTIONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

BinomCI(observed, total,
conf.level = 0.95,
method = "clopper-pearson")

### Methods: "wilson", "wald", "agresti-coull", "jeffreys",

### "modified wilson", "modified jeffreys",
### "clopper-pearson", "arcsine", "logit", "witting", "pratt"

est lwr.ci upr.ci

[1,] 0.3333333 0.1458769 0.5696755
[2,] 0.6666667 0.4303245 0.8541231

The PropCIs package has functions for calculating confidence intervals for a binomial proportion.

The exactci function uses the Clopper–Pearson exact method.

7/21

[1] 0.3333333

library(PropCIs)

exactci(7, 21,
conf.level=0.95)

95 percent confidence interval:

0.1458769 0.5696755

The blakerci function uses the Blaker exact method.

7/21

[1] 0.3333333

library(PropCIs)

blakerci(7, 21,
conf.level=0.95)

95 percent confidence interval:

0.1523669 0.5510455

Example of confidence intervals for a multinomial proportion

As part of a demographic survey of her scrapbooking 4-H course, Seras Victoria asks students to report their
sex. The following are the data from her course:
Sex Count

393
 CONFIDENCE INTERVALS FOR PROPORTIONS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Female 10
Male 9
Other 1
No answer 1
---------- -----
Total 21

library(DescTools)

observed = c(10, 9, 1, 1)

MultinomCI(observed,
conf.level=0.95,
method="sisonglaz")

### Methods: "sisonglaz", "cplus1", "goodman"

est lwr.ci upr.ci

[1,] 0.47619048 0.2857143 0.7009460
[2,] 0.42857143 0.2380952 0.6533270
[3,] 0.04761905 0.0000000 0.2723746
[4,] 0.04761905 0.0000000 0.2723746

Optional analysis: confidence intervals for a difference in proportions

As part of a demographic survey of their scrapbooking 4-H courses, Seras Victoria and Integra Hellsing ask
students if they have experience in scrapbooking. They want to determine the difference of proportions of
students having experience in each class, and calculate a confidence interval for that difference. The
following are the data:

Seras Victoria Integra Hellsing

Experience Count Experience Count

Yes 7 Yes 13
No 14 No 4
---------- ----- ---------- -----
Total 21 Total 17

Two functions in the PropCIs package can determine a confidence interval for a difference for in
independent proportions.
7/21

[1] 0.3333333

13/17

[1] 0.7647059

394
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

(7/21) - (13/17)

[1] -0.4313725

library(PropCIs)

diffscoreci(7, 21, 13, 17,

conf.level=0.95)

95 percent confidence interval:

-0.6685985 -0.1103804

wald2ci(7, 21, 13, 17,

conf.level=0.95,
adjust = "Wald")

### adjust = "AC", "Wald"

95 percent confidence interval:

-0.7165199 -0.1462252

sample estimates:
[1] -0.4313725

References
“Confidence Limits” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of Biological Statistics,
version 1.09. rcompanion.org/rcompanion/c_04.html.

Goodness-of-Fit Tests for Nominal Variables

Goodness-of-fit tests are used to compare proportions of levels of a nominal variable to theoretical or
expected proportions. Common goodness-of-fit tests are G-test, chi-square, and binomial or
multinomial exact tests.

In general, there are no assumptions about the distribution of data for these tests. However, the
results of chi-square tests and G-tests can be inaccurate if cell counts are low. A rule of thumb is that
all cell counts should be 5 or greater for chi-square- and G-tests. For a more complete discussion, see
McDonald in the “Optional Readings” section for details on what constitutes low cell counts.

One approach is to use exact tests, which are not bothered by low cell counts. However, if there are
not low cell counts, using G-test or chi-square test is fine. G-test is probably technically a better test

395
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

than chi-square. The advantage of chi-square tests is that your audience may be more familiar with
them.

G-tests are also called likelihood ratio tests, and “Likelihood Ratio Chi-Square” by SAS.

Appropriate data
• A nominal variable with two or more levels
• Theoretical, typical, expected, or neutral values for the proportions for this variable are
needed for comparison
• G-test and chi-square test may not be appropriate if there are cells with low counts in them

Hypotheses
• Null hypothesis: The proportions for the levels for the nominal variable are not different
from the expected proportions.
• Alternative hypothesis (two-sided): The proportions for the levels for the nominal variable
are different from the expected proportions.

Interpretation
Significant results can be reported as “The proportions for the levels for the nominal variable
were statistically different from the expected proportions.”

Packages used in this chapter

The packages used in this chapter include:

• EMT
• DescTools
• ggplot2

The following commands will install these packages if they are not already installed:
if(!require(EMT)){install.packages("EMT")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(ggplot2)){install.packages("ggplot2")}

Goodness-of-fit tests for nominal variables example

As part of a demographic survey of students in this environmental issues webinar series, Alucard
recorded the race and ethnicity of his students. He wants to compare the data for his class to the
demographic data for Cumberland County, New Jersey as a whole
Race Alucard’s_class County_proportion
White 20 0.775
Black 9 0.132
American Indian 9 0.012
Asian 1 0.054
Pacific Islander 1 0.002
Two or more races 1 0.025
----------------- --- ------

396
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Total 41 1.000

Ethnicity Alucard’s_class County_proportion

Hispanic 7 0.174
Not Hispanic 34 0.826
----------------- --- -----
Total 41 1.000

Exact tests for goodness-of-fit

Race data
observed = c(20, 9, 9, 1, 1, 1)
expected = c(0.775, 0.132, 0.012, 0.054, 0.002, 0.025)

library(EMT)

multinomial.test(observed, expected)

### This can take a long time!

Exact Multinomial Test, distance measure: p

Events pObs p.value

1370754 0 0

### A faster, approximate test by Monte Carlo simulation

observed = c(20, 9, 9, 1, 1, 1)
expected = c(0.775, 0.132, 0.012, 0.054, 0.002, 0.025)

library(EMT)

multinomial.test(observed, expected,
MonteCarlo = TRUE)

Exact Multinomial Test, distance measure: p

Events pObs p.value

1370754 0 0

Ethnicity data
x = 7
n = 41
expected = 0.174

binom.test(x, n, expected)

397
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Exact binomial test

number of successes = 7, number of trials = 41, p-value = 1

G-test for goodness-of-fit

Race data
observed = c(20, 9, 9, 1, 1, 1)
expected = c(0.775, 0.132, 0.012, 0.054, 0.002, 0.025)

library(DescTools)

GTest(x=observed,
p=expected,
correct="none")

### Correct: "none" "williams" "yates"

Log likelihood ratio (G-test) goodness of fit test

data: observed
G = 46.317, X-squared df = 5, p-value = 7.827e-09

Ethnicity data
observed = c(7, 34)
expected = c(0.174, 0.826)

library(DescTools)

GTest(x=observed,
p=expected,
correct="none")

### Correct: "none" "williams" "yates"

Log likelihood ratio (G-test) goodness of fit test

G = 0.0030624, X-squared df = 1, p-value = 0.9559

Chi-square test for goodness-of-fit

Race data
observed = c(20, 9, 9, 1, 1, 1)
expected = c(0.775, 0.132, 0.012, 0.054, 0.002, 0.025)

chisq.test(x = observed,
p = expected)

398
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Chi-squared test for given probabilities

X-squared = 164.81, df = 5, p-value < 2.2e-16

Ethnicity data
observed = c(7, 34)
expected = c(0.174, 0.826)

chisq.test(x = observed,
p = expected)

Chi-squared test for given probabilities

X-squared = 0.0030472, df = 1, p-value = 0.956

Multinomial test example with plot and confidence intervals

This is an example of a multinomial test that includes a bar plot showing confidence intervals. The
data is a simple vector of counts, like the above example.

For a similar example using two-way count data which is organized into a data frame, see the
"Examples of basic plots for nominal data" section in the Basic Plots chapter.

Walking to the store, Jerry Coyne observed the colors of nail polish on women’s toes (Coyne, 2016).
Presumably because that’s the kind of thing retired professors are apt to do. He concluded that red
was a more popular color but didn’t do any statistical analysis to support his conclusion.
Color of polish Count
Red 19
None or clear 3
White 1
Green 1
Purple 2
Blue 2

We will use a multinomial goodness of fit test to determine if there is an overall difference in the
proportion of colors (multinomial.test function in the EMT package). The confidence intervals for each
proportion can be found with the MultinomCI function in the DescTools package. The data then needs
to be manipulated some so that we can plot the data as counts and not proportions.

Note here that the expected counts are simply 1 divided by the number of treatments. In this case the
null hypothesis is that the observed proportions are all the same. In the examples above, the null
hypothesis was that the observed proportions were not different than expected proportions. It’s two
ways to think of the same null hypothesis.

399
 GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The confidence intervals for each proportion can be used as a post-hoc test, to determine which
proportions differ from each other, or to determine which proportions differ from 0.
nail.color = c("Red", "None", "White", "Green", "Purple", "Blue")
observed = c( 19, 3, 1, 1, 2, 2 )
expected = c( 1/6, 1/6, 1/6, 1/6, 1/6, 1/6 )

library(EMT)

multinomial.test(observed,
expected)

### This may take a while. Use Monte Carlo for large numbers.

Exact Multinomial Test, distance measure: p

Events pObs p.value

237336 0 0

library(DescTools)

MCI = MultinomCI(observed,
conf.level=0.95,
method="sisonglaz")

MCI

est lwr.ci upr.ci

[1,] 0.67857143 0.5357143 0.8423162
[2,] 0.10714286 0.0000000 0.2708876
[3,] 0.03571429 0.0000000 0.1994590
[4,] 0.03571429 0.0000000 0.1994590
[5,] 0.07142857 0.0000000 0.2351733
[6,] 0.07142857 0.0000000 0.2351733

Nail.color = factor(nail.color,
levels=unique(nail.color))

### Order the levels, otherwise R will alphabetize them

### For plot, Create variables of counts, and then wrap them into a data frame

Total = sum(observed)
Count = observed
Lower = MCI[,'lwr.ci'] * Total
Upper = MCI[,'upr.ci'] * Total

Data = data.frame(Count, Lower, Upper)

Data

400
GOODNESS-OF-FIT TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Count Lower Upper

1 19 15 23.584853
2 3 0 7.584853
3 1 0 5.584853
4 1 0 5.584853
5 2 0 6.584853
6 2 0 6.584853

library(ggplot2)

ggplot(Data, ### The data frame to use.

aes(x = Nail.color,
y = Count)) +

geom_bar(stat = "identity",
color = "black",
fill = "gray50",
width = 0.7) +

geom_errorbar(aes(ymin = Lower,
ymax = Upper),
width = 0.2,
size = 0.7,
position = pd,
color = "black"
) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +

ylab("Count of observations") +
xlab("Nail color")

Bar plot of the count of the color of women’s toenail polish observed by Jerry

401
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Coyne while walking to the store. Error bars indicate 95% confidence intervals
(Sison and Glaz method).

Optional readings
“Small numbers in chi-square and G–tests” in McDonald, J.H. 2014. Handbook of Biological Statistics.
www.biostathandbook.com/small.html.

References
Coyne, J. A. 2016. Why is red nail polish so popular? Why Evolution is True.
whyevolutionistrue.wordpress.com/2016/07/02/why-is-red-nail-polish-so-popular/.

“Chi-square Test of Goodness-of-Fit” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_03.html.

“Exact Test of Goodness-of-Fit” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_01.html.

“G–test of Goodness-of-Fit” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/b_04.html.

“Repeated G–tests of Goodness-of-Fit” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_09.html.

Association Tests for Nominal Variables

These tests for nominal variables are used to determine if two nominal variables are associated.
Sometimes the term “independent” is used to mean that there is no association.

In general, there are no assumptions about the distribution of data for these tests.

Note that for these tests of association there shouldn’t be paired values. For example, if experimental
units—the things you are counting—are “students before” and “students after”, or “left hands” and
“right hands”, the tests in the chapter Tests for Paired Nominal Data may be more appropriate.

Also note that these tests will not be accurate if there are “structural zeros” in the contingency table. If
you were counting pregnant and non-pregnant individuals across categories like male and female, the
male–pregnant cell may contain a structural zero if you assume your population cannot have pregnant
males.

Low cell counts

The results of chi-square tests and G-tests can be inaccurate if cell counts are low. A rule of thumb is
that all cell counts should be 5 or greater for chi-square- and G-tests. For a more complete discussion,
see McDonald in the “Optional Readings” section on what constitutes low cell counts.
402
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One approach when there are low counts is to use exact tests, such as Fisher’s exact test, which are not
bothered by low cell counts.

Another approach is to apply a continuity correction to the test. The chisq.test function automatically
applies the Yates's continuity correction for 2 x 2 tables. The GTest function has options for Yates or
Williams corrections.

Fisher’s exact test

Technically, Fisher’s exact test assumes that the table has fixed margins, in at least in one dimension.
That is, if the table has counts of some nominal variable for both males and females, the test assumes
that the number of males tested and the number of females tested was determined ahead of time.

It appears to me that this assumption is commonly ignored, but I don’t know if this is because of
ignorance of this assumption, or just that some people assume that advantages of the exact test
outweigh using it in cases when this assumption is violated.

Choosing among tests

If there are not low cell counts, using G-test or chi-square test is fine. G-test is probably technically a
better test than chi-square. The advantage of chi-square tests is that your audience may be more
familiar with them. That being said, some authors recommend the using Fisher’s exact test routinely
for tables of less than, say, 1000 observations, with the assumption that exact tests yield more
accurate p-values.

Appropriate data
• Two nominal variables with two or more levels each.
• Experimental units aren’t paired.
• There are no structural zeros in the contingency table.
• G-test and chi-square test may not be appropriate if there are cells with low counts in them.

Hypotheses
• Null hypothesis: There is no association between the two variables.
• Alternative hypothesis (two-sided): There is an association between the two variables.

Interpretation
Significant results can be reported as “There was a significant association between variable A
and variable B.”

Post-hoc analysis
Post hoc analysis for tests on a contingency table larger than 2 x 2 can be conducted by
conducting tests for the component 2 x 2 tables. A correction for multiple tests should be
applied.

Other notes and alternative tests

• For paired data, see the tests in the chapter Tests for Paired Nominal Data.
• For matrices with 3 dimensions, the Cochran–Mantel–Haenszel test can be used.

403
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Packages used in this chapter

The packages used in this chapter include:

• DescTools
• multcompView
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(DescTools)){install.packages("DescTools")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(rcompanion)){install.packages("rcompanion")}

Association tests for nominal variables example

Alexander Anderson runs the pesticide safety training course in four counties. Students must pass in
order to obtain their pesticide applicator’s license. He wishes to see if there is an association between
the county in which the course was held and the rate of passing the test. The following are his data.

County Pass Fail

Bloom County 21 5
Cobblestone County 6 11
Dougal County 7 8
Heimlich County 27 5

Reading the data as a matrix

Input =("
County Pass Fail
Bloom 21 5
Cobblestone 6 11
Dougal 7 8
Heimlich 27 5
")

Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix

Fisher Exact test of association

fisher.test(Matrix)

Fisher's Exact Test for Count Data

p-value = 0.000668
alternative hypothesis: two.sided

404
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Post-hoc analysis
Post-hoc analysis can be conducted with pairwise Fisher’s exact tests. The function
pairwiseNominalIndependence in the rcompanion package can be used to conduct this analysis.
### Order matrix

Matrix = Matrix[(c("Heimlich", "Bloom", "Dougal", "Cobblestone")),]

Matrix

### Pairwise tests of association

library(rcompanion)

PT = pairwiseNominalIndependence(Matrix,
compare = "row",
fisher = TRUE,
gtest = FALSE,
chisq = FALSE,
method = "fdr", # see ?p.adjust for options
digits = 3)

PT

Comparison p.Fisher p.adj.Fisher

1 Heimlich : Bloom 0.740000 0.74000
2 Heimlich : Dougal 0.013100 0.02620
3 Heimlich : Cobblestone 0.000994 0.00596
4 Bloom : Dougal 0.037600 0.05640
5 Bloom : Cobblestone 0.003960 0.01190
6 Dougal : Cobblestone 0.720000 0.74000

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adj.Fisher,
threshold = 0.05)

Group Letter MonoLetter

1 Heimlich a a
2 Bloom ab ab
3 Dougal bc bc
4 Cobblestone c c

The table of adjusted p-values can be summarized to a table of letters indicating which treatments are
not significantly different.
County Percent passing Letter
Heimlich County 84% a

405
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bloom County 81 ab
Dougal County 47 bc
Cobblestone County 35 c

Counties sharing a letter are not significantly different by Fisher exact test,
with p-values adjusted by FDR method for multiple comparisons (Benjamini–Hochberg
false discovery rate).

This table of letters can also be found using the pairwiseNominalMatrix function along with the
multcompLetters function in the multcompView package.
### Order matrix

Matrix = Matrix[(c("Heimlich", "Bloom", "Dougal", "Cobblestone")),]

Matrix

### Pairwise tests of association

library(rcompanion)

PM = pairwiseNominalMatrix(Matrix,
compare = "row",
fisher = TRUE,
gtest = FALSE,
chisq = FALSE,
method = "fdr", # see ?p.adjust for options
digits = 3)
PM

$Test
[1] "Fisher exact test"

$Method
[1] "fdr"

$Adjusted
Heimlich Bloom Dougal Cobblestone
Heimlich 1.00000 0.7400 0.0262 0.00596
Bloom 0.74000 1.0000 0.0564 0.01190
Dougal 0.02620 0.0564 1.0000 0.74000
Cobblestone 0.00596 0.0119 0.7400 1.00000

library(multcompView)

multcompLetters(PM$Adjusted,
compare="<",
threshold=0.05, ### p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

406
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Heimlich Bloom Dougal Cobblestone

"a" "ab" "bc" "c"

G-test of association
library(DescTools)

GTest(Matrix)

Log likelihood ratio (G-test) test of independence without correction

G = 17.14, X-squared df = 3, p-value = 0.0006615

Post-hoc analysis
### Order matrix

Matrix = Matrix[(c("Heimlich", "Bloom", "Dougal", "Cobblestone")),]

Matrix

### Pairwise tests of association

pairwiseNominalIndependence(Matrix,
compare = "row",
fisher = FALSE,
gtest = TRUE,
chisq = FALSE,
method = "fdr", # see ?p.adjust for options
digits = 3)

Comparison p.Gtest p.adj.Gtest

1 Heimlich : Bloom 0.718000 0.71800
2 Heimlich : Dougal 0.008300 0.01660
3 Heimlich : Cobblestone 0.000506 0.00304
4 Bloom : Dougal 0.024800 0.03720
5 Bloom : Cobblestone 0.002380 0.00714
6 Dougal : Cobblestone 0.513000 0.61600

The table of adjusted p-values can be summarized to a table of letters indicating which treatments are
not significantly different.
County Percent passing Letter
Heimlich County 84% a
Bloom County 81 a
Dougal County 47 b
Cobblestone County 35 b

Counties sharing a letter are not significantly different by G-test for

association, with p-values adjusted by FDR method for multiple comparisons

407
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

(Benjamini–Hochberg false discovery rate).

Chi-square test of association

chisq.test(Matrix)

Pearson's Chi-squared test

X-squared = 17.32, df = 3, p-value = 0.0006072

Post-hoc analysis
### Order matrix

Matrix = Matrix[(c("Heimlich", "Bloom", "Dougal", "Cobblestone")),]

Matrix

### Pairwise tests of association

library(rcompanion)

pairwiseNominalIndependence(Matrix,
compare = "row",
fisher = FALSE,
gtest = FALSE,
chisq = TRUE,
method = "fdr", # see ?p.adjust for options
digits = 3)

Comparison p.Chisq p.adj.Chisq

1 Heimlich : Bloom 0.99000 0.99000
2 Heimlich : Dougal 0.01910 0.03820
3 Heimlich : Cobblestone 0.00154 0.00924
4 Bloom : Dougal 0.05590 0.08380
5 Bloom : Cobblestone 0.00707 0.02120
6 Dougal : Cobblestone 0.77000 0.92400

The table of adjusted p-values can be summarized to a table of letters indicating which treatments are
not significantly different.
County Percent passing Letter
Heimlich County 84% a
Bloom County 81 ab
Dougal County 47 bc
Cobblestone County 35 c

Counties sharing a letter are not significantly different by chi-square test of

association, with p-values adjusted by FDR method for multiple comparisons
(Benjamini–Hochberg false discovery rate).

408
 ASSOCIATION TESTS FOR NOMINAL VARIABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional readings

“Small numbers in chi-square and G–tests” in McDonald, J.H. 2014. Handbook of Biological Statistics.
www.biostathandbook.com/small.html.

References

“Fisher’s Exact Test of Independence” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_07.html.

“G–test of Independence” in Mangiafico, S.S. 2015b. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/b_06.html.

“Chi-square Test of Independence” in Mangiafico, S.S. 2015c. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_05.html.

“Small Numbers in Chi-square and G–tests” in Mangiafico, S.S. 2015d. An R Companion for the
Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/b_08.html.

Exercises M

1. Considering Alexander Anderson’s data,

Numerically, which county had the greatest percentage of passing students?

Numerically, which county had the greatest number of total students?

Was there an association between county and student success?

Statistically, which counties performed the best? Which performed the worst? Be sure to
indicate which post-hoc test you are using.

Plot the data in an appropriate way. What does the plot suggest to you?

2. Ryuk and Rem held a workshop on planting habitat for pollinators like bees and butterflies. They
wish to know if there is an association between the profession of the attendees and their willingness to
undertake a conservation planting. The following are the data.

Will plant?
Profession Yes No
Homeowner 13 14
Landscaper 27 6
Farmer 7 19
NGO 6 6

409
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

Numerically, which profession had the greatest percentage of answering yes?

Numerically, which profession had the greatest number of total attendees?

Was there an association between profession and willingness to undertake a conservation

planting?

Statistically, which professions were most willing? Which were least willing? Be sure to
indicate which post-hoc test you are using.

Plot the data in an appropriate way. What does the plot suggest to you?

Tests for Paired Nominal Data

Tests of symmetry for nominal data are used when the counts on a contingency table represent values
that are paired or repeated in time.

As an example, consider a question on repeated on a pre-test and a post-test. We may want to know if
the number of correct responses changed from the pre-test to the post-test.
Did students have the correct answer to the question?

After
Before Correct Incorrect
Correct 2 0
Incorrect 21 7

Note that the row names and column names have the same levels, and that counts represent paired
responses. That is, for each observation you must know the individual’s response before and after.

Also note that the number of students included in the table are 30, or the sum of the cell counts.

In essence, those students with the same response before and after don’t affect the assessment of the
change in responses. We would focus on the “discordant” counts. That is, How many students had
incorrect answers before and correct answers after, in contrast to those who had the reverse trend.
Here, because 21 changed from incorrect to correct, and 0 changed from correct to incorrect, we might
suspect that there was a significant change in responses from incorrect to correct.

To grasp the difference between nominal tests of association and nominal tests of symmetry, be sure
to visit the coffee and tea example below in the section “An example without repeated measures,
comparing test of symmetry with test of association”.

410
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Appropriate data
• Two nominal variables with two or more levels each, and each with the same levels.
• Observations are paired or matched between the two variables.
• McNemar and McNemar–Bowker tests may not be appropriate if discordant cells have low
counts.

Hypotheses
• Null hypothesis: The contingency table is symmetric. That is, the probability of cell [i, j] is
equal to the probability of cell [j, i].
• Alternative hypothesis (two-sided): The contingency table is not symmetric.

Interpretation
Depending on the context, significant results can be reported as e.g. “There was a significant
change from answer A to answer B.” Or, “X was more popular than Y.”

Post-hoc analysis
Post hoc analysis for tests on a contingency table larger than 2 x 2 can be conducted by
conducting tests for the component 2 x 2 tables. A correction for multiple tests should be
applied.

Other notes and alternative tests

• For unpaired data, see the tests in the chapter Association Tests for Nominal Data.
• For multiple times or groups, Cochran’s Q test can be used.

Packages used in this chapter

The packages used in this chapter include:

• EMT
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(EMT)){install.packages("EMT")}
if(!require(rcompanion)){install.packages("rcompanion")}

McNemar and McNemar–Bowker tests

For a 2 x 2 table, the most common test for symmetry is McNemar’s test. For larger tables, McNemar’s
test is generalized as the McNemar–Bowker symmetry test. One drawback to the latter test is that it
may fail if there are 0’s in certain locations in the matrix.

McNemar’s test may not be reliable if there are low counts in the “discordant” cells. Authors
recommend that these cells to sum to at least 5 or 10 or 25.

Exact tests
Exact tests of symmetry reduce to exact tests for goodness-of-fit. A 2 x 2 table is analyzed with a
binomial exact test, and a larger table is analyzed with a multinomial exact test. Examples of these are
shown in this chapter in the “Optional analyses: conducting exact tests for symmetry” section.

411
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

I have written a function, nominalSymmetryTest to conduct these exact tests easily.

Example of tests for paired data nominal data

Alucard teaches a Master Gardener training on rain gardens for stormwater management and one on
rain barrels. He wishes to assess if people are more willing to install these green infrastructure
practices after attending the training. His data follow. Note that there are 46 attendees answering
each question.
Are you planning to install a rain barrel?

After
Before Yes No
Yes 9 5
No 17 15

Are you planning to install a rain garden?

After
Before Yes No Maybe
Yes 6 0 1
No 5 3 7
Maybe 11 1 12

Rain barrel
Input =("
Before After.yes After.no
Before.yes 9 5
Before.no 17 15
")

Matrix.1 = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix.1

sum(Matrix.1)

[1] 46

Rain garden
Input =("
Before Yes.after No.after Maybe.after
Yes.before 6 0 1
No.before 5 3 7
Maybe.before 11 1 12

412
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

")

Matrix.2 = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix.2

sum(Matrix.2)

[1] 46

Exact tests for symmetry

Rain barrel
library(rcompanion)

nominalSymmetryTest(Matrix.1,
digits = 3)

$Global.test.for.symmetry

Dimensions p.value
1 2 x 2 0.0169

Rain garden
library(rcompanion)

nominalSymmetryTest (Matrix.2,
method="fdr",
digits = 3)

### Note: This may take a long time

### Use MonteCarlo option for large matrices or counts

$Global.test.for.symmetry

Dimensions p.value
1 3 x 3 2e-04

$Pairwise.symmetry.tests
Comparison p.value p.adjust
1 Yes.before/Yes.after : No.before/No.after 0.0625 0.0703
2 Yes.before/Yes.after : Maybe.before/Maybe.after 0.00635 0.0190
3 No.before/No.after : Maybe.before/Maybe.after 0.0703 0.0703

$p.adjustment

Method
1 fdr

413
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Maybe to Yes, p.value = 0.0190

Before Yes.after No.after Maybe.after

Yes.before 6 0 1
No.before 5 3 7
Maybe.before 11 1 12

McNemar and McNemar–Bowker chi-square tests for symmetry

Rain barrel
mcnemar.test(Matrix.1)

McNemar's Chi-squared test with continuity correction

McNemar's chi-squared = 5.5, df = 1, p-value = 0.01902

Rain garden
mcnemar.test(Matrix.2)

McNemar's Chi-squared test

McNemar's chi-squared = 17.833, df = 3, p-value = 0.0004761

An example without repeated measures, comparing test of symmetry with test of association
As another example, consider a survey of tea and coffee drinking, in which each respondent is asked
both if they drink coffee, and if they drink tea.
Tea
Coffee Yes No
Yes 37 17
No 9 25

We would use a test of symmetry in this case if the question we wanted to answer was, Is coffee more
popular than tea? That is, is it more common for someone to drink coffee and not tea than to drink tea
and not coffee? (Those who drink both or drink neither are not relevant to this question.)

Note also that this is an example of using a test of symmetry to test the relative frequency of two
dichotomous variables when the same subjects are surveyed.
Input =("
Coffee Yes No
Yes 37 17
No 9 25
")

414
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Matrix.3 = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

mcnemar.test(Matrix.3)

McNemar's Chi-squared test with continuity correction

McNemar's chi-squared = 1.8846, df = 1, p-value = 0.1698

### Neither coffee nor tea is more popular, specifically because

### neither the 9 nor the 17 in the table are large relative to
### the other.

A test of association answers a very different question. Namely, Is coffee drinking associated with tea
drinking? That is, is someone more likely to drink tea if they drink coffee?
chisq.test(Matrix.3)

Pearson's Chi-squared test with Yates' continuity correction

X-squared = 13.148, df = 1, p-value = 0.0002878

### Coffee drinking and tea drinking are associated, in this case people who
### drink coffee are likely to drink tea. This is a positive association.
### A negative association could also be significant.

Optional analysis: a 4 x 4 example with several 0’s

As an additional example, imagine a religious caucusing event in which advocates try to sway
attendees to switch their religions.

Matrix row names are the attendees’ original religions, and the column names, with a “2” added, are
attendees’ new religions after the caucus. Note there are several 0 counts in the matrix.

Note first that the mcnemar.test function fails, namely because of the position of some 0 counts in the
matrix.

Second, note that the multinomial.test function used by the nominalSymmetryTest function would take
a long time to calculate an exact p-value for this matrix, so the MonteCarlo=TRUE option is used. The
number of samples used in the Monte Carlo approach can be adjusted with the ntrial option. Some of
the p-values in the post-hoc analysis cannot be produced because of the placement of 0 counts, but
these should also be considered non-significant results.
Input =("
Before Pastafarian2 Discordiant2 Dudist2 Jedi2
Pastafarian 7 0 23 0
Discordiant 0 7 0 33
Dudist 3 0 7 1
Jedi 0 1 0 7

415
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

")

Matrix.4 = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

Matrix.4

McNemar –Bowker test

mcnemar.test(Matrix.4)

McNemar's Chi-squared test

McNemar's chi-squared = NaN, df = 6, p-value = NA

Exact test
library(rcompanion)

nominalSymmetryTest(Matrix.4,
method="fdr",
digits = 3,
MonteCarlo = TRUE,
ntrial = 100000)

$Global.test.for.symmetry

Dimensions p.value
1 4 x 4 0

$Pairwise.symmetry.tests
Comparison p.value p.adjust
1 Pastafarian/Pastafarian2 : Discordiant/Discordiant2 <NA> NA
2 Pastafarian/Pastafarian2 : Dudist/Dudist2 8.8e-05 1.32e-04
3 Pastafarian/Pastafarian2 : Jedi/Jedi2 <NA> NA
4 Discordiant/Discordiant2 : Dudist/Dudist2 <NA> NA
5 Discordiant/Discordiant2 : Jedi/Jedi2 4.07e-09 1.22e-08
6 Dudist/Dudist2 : Jedi/Jedi2 1 1.00e+00

$p.adjustment

Method
1 fdr

A look at the significant results

Pastafarian to Dudist, p-value = 0.000132

Before Pastafarian2 Discordiant2 Dudist2 Jedi2

Pastafarian 7 0 23 0
Discordiant 0 7 0 33

416
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Dudist 3 0 7 1
Jedi 0 1 0 7

Discordiant to Jedi, p-value < 0.0001

Before Pastafarian2 Discordiant2 Dudist2 Jedi2

Pastafarian 7 0 23 0
Discordiant 0 7 0 33
Dudist 3 0 7 1
Jedi 0 1 0 7

Optional analyses: conducting exact tests for symmetry

The exact symmetry tests can be conducted directly with the binom.test and multinomial.test functions.
These results match the results of the nominalSymmetryTest function.
Rain barrel

After
Before Yes No
Yes 9 5
No 17 15

Rain garden

After
Before Yes No Maybe
Yes 6 0 1
No 5 3 7
Maybe 11 1 12

Rain barrel
For a 2 x 2 matrix, x = the count in one of the discordant cells, and n = the sum of the counts in
discordant cells. The expected proportion is 0.50.

You could also follow the method for an n x n matrix below.

x = 5
n = 5 + 17
expected = 0.50

binom.test(x, n, expected)

Exact binomial test

number of successes = 5, number of trials = 22, p-value = 0.0169

417
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Rain garden
For an n x n matrix, observed = a vector of the counts of discordant cells. Expected is a vector of length
(n * n – n), with each value equal to (1 / (n * n – n)).
observed = c(0, 1, 5, 7, 11, 1)
expected = c(1/6, 1/6, 1/6, 1/6, 1/6, 1/6)

library(EMT)

multinomial.test(observed, expected)

Exact Multinomial Test, distance measure: p

Events pObs p.value

142506 0 2e-04

Yes–No
For post-hoc testing, reduce the matrix to a 2 x 2 matrix.
x = 0
n = 0 + 5
expected = 0.50

binom.test(x, n, expected)

Exact binomial test

number of successes = 0, number of trials = 5, p-value = 0.0625

Yes–Maybe
x = 1
n = 1 + 11
expected = 0.50

binom.test(x, n, expected)

Exact binomial test

number of successes = 1, number of trials = 12, p-value = 0.006348

No–Maybe
x = 7
n = 7 + 1
expected = 0.50

binom.test(x, n, expected)

Exact binomial test

418
 TESTS FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

number of successes = 7, number of trials = 8, p-value = 0.07031

Exercises N

1. Considering Alucard’s data,

How many students responded to the rain barrel question?

How many students changed their answer on the rain barrel question from no to yes?

How would you interpret the results on the rain barrel question?

How would you interpret the result of the global test of the rain garden question?

How would you interpret the result of the post-hoc analysis of the rain garden question?

2. Considering the coffee and tea example, Do you understand the difference between the hypotheses
for tests of association and tests of symmetry? Would you be comfortable choosing the correct
approach?

3. Ryuk and Rem held a workshop on planting habitat for pollinators like bees and butterflies. They
wish to know if attendees were more likely to do a planting after the workshop than before.
Will plant?

After
Before Yes.after No.after Maybe.after
Yes 17 0 0
No 5 9 13
Maybe 15 0 7

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

How many students responded to this question?

How many students changed their answer from no to yes?

How would you interpret the result of the global test?

How would you interpret the result of the post-hoc analysis?

419
 COCHRAN–MANTEL–HAENSZEL TEST FOR 3-DIMENSIONAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cochran–Mantel–Haenszel Test for 3-

Dimensional Tables
The Cochran–Mantel–Haenszel test is an extension of the chi-square test of association. It is used for
multiple chi-square tests across multiple groups or times. The data are stratified so that each chi-
square table is within one group or time.

The following data investigate whether there is a link between listening to podcasts and using public
transportation to get to work, collected across three cities. This can be thought of as a 2 x 2
contingency table in each of the three cities.

Data can be arranged in a table of counts or can be arranged in long-format with or without counts. If
a table is used for input, it should follow R’s ftable format, as shown.

Table format
City Bikini.Bottom Frostbite.Falls New.New.York
Listen Transport
Podcast Drive 13 17 5
Public 27 25 27
No.podcast Drive 23 22 17
Public 44 31 22

Long-format with counts

City Listen Transport Count
Bikini.Bottom Podcast Drive 13
Bikini.Bottom Podcast Public 27
Bikini.Bottom No.podcast Drive 23
Bikini.Bottom No.podcast Public 44
Frostbite.Falls Podcast Drive 17
Frostbite.Falls Podcast Public 25
Frostbite.Falls No.podcast Drive 22
Frostbite.Falls No.podcast Public 31
New.New.York Podcast Drive 5
New.New.York Podcast Public 27
New.New.York No.podcast Drive 17
New.New.York No.podcast Public 22

The test can be conducted with the mantelhaen.test function in the native stats package.

One assumption of the test is that there are no three-way interactions in the data. This is confirmed
with a non-significant result from a test such as the Woolf test or Breslow–Day test.

Post-hoc analysis can include looking at the individual chi-square, Fisher exact, or G-test for
association for each time or group.

420
 COCHRAN–MANTEL–HAENSZEL TEST FOR 3-DIMENSIONAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The component n x n tables can be 2 x 2 or larger.

Appropriate data
• Three nominal variables with two or more levels each.
• Data can be stratified as n x n tables with the third time or grouping variable

Hypotheses
• Null hypothesis: There is no association between the two inner variables.
• Alternative hypothesis (two-sided): There is an association between the two inner variables.

Interpretation
Significant results can be reported as “There was a significant association between variable A
and variable B [across groups].”

Post-hoc analysis
Post-hoc analysis can include looking at the individual chi-square, Fisher exact, or g-test for
association for each time or group.

Packages used in this chapter

The packages used in this chapter include:

• psych
• vcd
• DescTools
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(vcd)){install.packages("vcd")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(rcompanion)){install.packages("rcompanion")}

C–M–H test example: long-format with counts

Alexander Anderson is concerned that there is a bias in his teaching methods for his pesticide
applicator’s course. He wants to know if there is an association between students’ sex and passing the
course across the four counties in which he teaches. The following are his data.
Input = ("
County Sex Result Count
Bloom Female Pass 9
Bloom Female Fail 5
Bloom Male Pass 7
Bloom Male Fail 17
Cobblestone Female Pass 11
Cobblestone Female Fail 4
Cobblestone Male Pass 9

421
 COCHRAN–MANTEL–HAENSZEL TEST FOR 3-DIMENSIONAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cobblestone Male Fail 21

Dougal Female Pass 9
Dougal Female Fail 7
Dougal Male Pass 19
Dougal Male Fail 9
Heimlich Female Pass 15
Heimlich Female Fail 8
Heimlich Male Pass 14
Heimlich Male Fail 17
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

Data$County = factor(Data$County,
levels=unique(Data$County))

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Data$Result = factor(Data$Result,
levels=unique(Data$Result))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Convert data to a table

Table = xtabs(Count ~ Sex + Result + County,
data=Data)

### Note that the grouping variable is last in the xtabs function

ftable(Table) # Display a flattened table

County Bloom Cobblestone Dougal Heimlich

Sex Result
Female Pass 9 11 9 15
Fail 5 4 7 8

422
 COCHRAN–MANTEL–HAENSZEL TEST FOR 3-DIMENSIONAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Male Pass 7 9 19 14
Fail 17 21 9 17

Cochran–Mantel–Haenszel test
mantelhaen.test(Table)

Mantel-Haenszel chi-squared test with continuity correction

Mantel-Haenszel X-squared = 6.7314, df = 1, p-value = 0.009473

alternative hypothesis: true common odds ratio is not equal to 1

Woolf test
library(vcd)

woolf_test(Table)

### Woolf test for homogeneity of odds ratios across strata.

### If significant, C-M-H test is not appropriate

Woolf-test on Homogeneity of Odds Ratios (no 3-Way assoc.)

X-squared = 7.1376, df = 3, p-value = 0.06764

Post-hoc analysis
The groupwiseCMH function will conduct analysis of the component n x n tables with Fisher exact, g-
test, or chi-square tests of association. It accepts only a 3-dimensional table. The group option
indicates which dimension should be considered the grouping variable (1, 2, or 3). It will conduct only
one type of test at a time. That is, if multiple of the options fisher, gtest, or chisq are set to TRUE, it will
conduct only one of them. As usual, method is the p-value adjustment method (see ?p.adjust for
options), and digits indicates the number of digits in the output. The correct option is used by the chi-
square test function.
library(rcompanion)

groupwiseCMH(Table,
group = 3,
fisher = TRUE,
gtest = FALSE,
chisq = FALSE,
method = "fdr",
correct = "none",
digits = 3)

Group Test p.value adj.p

1 Bloom Fisher 0.0468 0.0936
2 Cobblestone Fisher 0.0102 0.0408
3 Dougal Fisher 0.5230 0.5230

423
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

4 Heimlich Fisher 0.1750 0.2330

C–M–H test example: table format

The read.ftable function can be very fussy about the formatting of the input. 1) It seems to not like a
blank first line, so the double quote symbol in the input should be on the same line as the column
names. 2) It doesn’t like leading spaces on the input lines. These may appear when you paste the code
in to the RStudio Console or R Script area. One solution is to manually delete these spaces. R Script
files that are saved without these leading spaces should be able to be opened and run without further
modification.

The Cochran–Mantel–Haenszel test, Woolf test, and post-hoc analysis would be the same as those
conducted on Table above.
Input =(
" County Bloom Cobblestone Dougal Heimlich
Sex Result
Female Pass 9 11 9 15
Fail 5 4 7 8
Male Pass 7 9 19 14
Fail 17 21 9 17
")

Table = as.table(read.ftable(textConnection(Input)))

ftable(Table)

### Display a flattened table

Cochran’s Q Test for Paired Nominal Data

Cochran’s Q test is an extension of the McNemar test, when the response variable is dichotomous and
there are either multiple times for a repeated measure or multiple categories with paired responses. A
dichotomous variable is a nominal variable with only two levels.

One case would be extending the rain barrel question from the previous chapter to multiple times. A
more common case would be extending the coffee and tea example to multiple beverages. In this case,
we would test among several beverages which is more popular that the others.

Data are not typically arranged in a contingency table, but are either organized in long-format, with
each row representing a single observation, or organized in a short-format matrix with each row
representing an experimental unit with the repeated measures across columns.
Long-format

Student Beverage Response

424
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

a Coffee Yes
a Tea Yes
a Cola No
b Coffee No
b Tea No
b Cola Yes
c Coffee Yes
c Tea No
c Cola No

Short-format

Student Coffee Tea Cola

a Yes Yes No
b No No Yes
c Yes No No

Data need to be arranged in an unreplicated complete block design. That is, for each experimental unit
(Student in this case) there needs to be one and only one response per question or time.

The package RVAideMemoire has a function cochran.qtest which will conduct Cochran’s Q test on long-
format data. The function symmetry_test in the coin package will also conduct the test using
permutation, which a type of resampling technique.

A word of caution about the symmetry_test function: be sure not to use it in cases where the response
variable is multinomial; that is, where there are more than two levels in the response. The function
will return a result, but it is not the same kind of symmetry result as the McNemar–Bowker test. I
suspect that for a multinomial response variable, coin treats it like an ordered factor, not like a
nominal variable.

When there are two questions or times, Cochran’s Q test is equivalent to the McNemar test.

Post-hoc analysis can be conducted with pairwise McNemar or exact tests on 2 x 2 tables. The function
pairwiseMcnemar in the rcompanion package will conduct pairwise McNemar, exact, or permutation
tests.

Appropriate data
• The response variable is a dichotomous nominal variable.
• The responses are paired by experimental unit.
• The responses are measured across two or more times or factors.
• The data follow an unreplicated complete block design, with each experimental unit treated
as the block.

Hypotheses
• Null hypothesis: The marginal probability of a [positive] response is unchanged across the
times or factors. That is, a positive response is equally likely across times or factors.
• Alternative hypothesis: Positive responses are not equally likely across times or factors.
425
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interpretation
Significant results can be reported as “There was a significant difference in the proportion of
positive responses across times or factors.” Or, “A positive response for X was more likely
than a positive response for Y.”

Post-hoc analysis
Post hoc analysis can be conducted by conducting tests for the component 2 x 2 tables. A
correction for multiple tests should be applied.

Packages used in this chapter

The packages used in this chapter include:

• psych
• RVAideMemoire
• coin
• reshape2
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(coin)){install.packages("coin")}
if(!require(reshape2)){install.packages("reshape2")}
if(!require(rcompanion)){install.packages("rcompanion")}

Cochran’s Q test example: long-format

Hayley Smith teaches an extension course on environmentally-friendly lawn care. After the course,
she surveys her students about their willingness to adopt good practices: Soil testing, measuring the
amount of irrigation, raising the mowing height, and returning the grass clippings to the lawn. She
wants to know which practice students were more willing to adopt.
Input = ("
Practice Student Response
SoilTest a Yes
SoilTest b No
SoilTest c Yes
SoilTest d Yes
SoilTest e No
SoilTest f Yes
SoilTest g Yes
SoilTest h Yes
SoilTest i No
SoilTest j Yes
SoilTest k Yes
SoilTest l Yes
SoilTest m Yes

426
COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

SoilTest n Yes
Irrigation a Yes
Irrigation b No
Irrigation c No
Irrigation d No
Irrigation e No
Irrigation f No
Irrigation g No
Irrigation h No
Irrigation i Yes
Irrigation j No
Irrigation k No
Irrigation l No
Irrigation m No
Irrigation n No
MowHeight a Yes
MowHeight b No
MowHeight c Yes
MowHeight d Yes
MowHeight e Yes
MowHeight f Yes
MowHeight g Yes
MowHeight h Yes
MowHeight i No
MowHeight j Yes
MowHeight k Yes
MowHeight l Yes
MowHeight m Yes
MowHeight n Yes
Clippings a Yes
Clippings b No
Clippings c No
Clippings d Yes
Clippings e Yes
Clippings f No
Clippings g No
Clippings h Yes
Clippings i Yes
Clippings j Yes
Clippings k Yes
Clippings l No
Clippings m Yes
Clippings n No
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

Data$Practice = factor(Data$Practice,
levels=unique(Data$Practice))

Data$Response = factor(Data$Response,
levels=c("Yes", "No"))

427
COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

### View data as a table

Data$Response.n = as.numeric(Data$Response) - 1 ### Creates a new numeric variable

### that is Response as a 0 or 1

Table = xtabs(Response.n ~ Student + Practice,

data=Data)

Table

Practice
Student SoilTest Irrigation MowHeight Clippings
a 0 0 0 0
b 1 1 1 1
c 0 1 0 1
d 0 1 0 0
e 1 1 0 0
f 0 1 0 1
g 0 1 0 1
h 0 1 0 0
i 1 0 1 0
j 0 1 0 0
k 0 1 0 0
l 0 1 0 1
m 0 1 0 0
n 0 1 0 1

### View counts of responses

xtabs( ~ Practice + Response,

data=Data)

Response
Practice Yes No
SoilTest 11 3
Irrigation 2 12

428
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

MowHeight 12 2
Clippings 8 6

### Create bar plot

Table = xtabs( ~ Response + Practice,

data=Data)

Table

barplot(Table,
beside = TRUE,
legend = TRUE,
ylim = c(0, 12), ### y-axis: used to prevent legend overlapping bars
cex.names = 0.8, ### Text size for bars
cex.axis = 0.8, ### Text size for axis
args.legend = list(x = "topright", ### Legend location
cex = 0.8, ### Legend text size
bty = "n")) ### Remove legend box

Cochran’s Q test
library(RVAideMemoire)

cochran.qtest(Response ~ Practice | Student,

data = Data)

Cochran's Q test

Q = 16.9535, df = 3, p-value = 0.0007225

### Note that the function also gives you proportion of responses for the
### positive response, which is always the second of the nominal response,
### which in this case is "no", or the response coded as "1".

429
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### I wouldn’t use the post-hoc analysis included with the output of the
### function in RVAideMemoire

library(coin)

symmetry_test(Response ~ Practice | Student,

data = Data,
teststat = "quad")

Asymptotic General Symmetry Test

chi-squared = 16.953, df = 3, p-value = 0.0007225

Post-hoc analysis for Cochran’s Q test

The pairwiseMcnemar function will conduct pairwise McNemar, binomial exact, or permutation tests
analogous to uncorrected McNemar tests. The permutation tests require the coin package. As usual,
method is the p-value adjustment method (see ?p.adjust for options), and digits indicates the number of
digits in the output. The correct option is used by the chi-square test function.
### Order groups

Data$Practice = factor(Data$Practice,
levels = c("MowHeight", "SoilTest",
"Clippings", "Irrigation"))

### Pairwise McNemar tests

library(rcompanion)

PT = pairwiseMcnemar(x = Data$Response,
g = Data$Practice,
block = Data$Student,
test = "permutation",
method = "fdr",
digits = 3)

PT

$Pairwise
Comparison p.value p.adjust
1 MowHeight - SoilTest = 0 0.317 0.3170
2 MowHeight - Clippings = 0 0.102 0.1530
3 MowHeight - Irrigation = 0 0.00389 0.0200
4 SoilTest - Clippings = 0 0.257 0.3080
5 SoilTest - Irrigation = 0 0.00666 0.0200
6 Clippings - Irrigation = 0 0.0143 0.0286

### Compact letter display

430
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

PT = PT$Pairwise

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.adjust,
threshold = 0.05)

Group Letter MonoLetter

1 MowHeight a a
2 SoilTest a a
3 Clippings a a
4 Irrigation b b

Table of results

Proportion "Yes" Grouping letter

MowHeight 0.86 a
SoilTest 0.79 a
Clippings 0.57 a
Irrigation 0.14 b

Optional analysis: Converting a matrix of p-values to a compact letter display

This compact letter display could also be generated with the multcompView package. To perform this
well, it is helpful to first order the factors by the numeric order of their results, in this case by
proportion of “yes” responses. Then the data need to be arranged into a matrix of pairwise p-values.
Input =("
Column MowHeight SoilTest Clippings Irrigation
MowHeight NA 0.317 0.153 0.0200
SoilTest NA NA 0.308 0.0200
Clippings NA NA NA 0.0286
Irrigation NA NA NA NA
")

PT = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

PT

### Transform the upper matrix to a full matrix

diag(PT) = 1

PT1 = t(PT)

PT[lower.tri(PT)] = PT1[lower.tri(PT1)]

PT

431
COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Produce compact letter display

library(multcompView)

multcompLetters(PT,
compare="<",
threshold=0.05, ### p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

MowHeight SoilTest Clippings Irrigation

"a" "a" "a" "b"

Cochran’s Q test example: short-format in matrix

Input =("
Student SoilTest Irrigation MowHeight Clippings
a 0 0 0 0
b 1 1 1 1
c 0 1 0 1
d 0 1 0 0
e 1 1 0 0
f 0 1 0 1
g 0 1 0 1
h 0 1 0 0
i 1 0 1 0
j 0 1 0 0
k 0 1 0 0
l 0 1 0 1
m 0 1 0 0
n 0 1 0 1
")

Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))

### Add names to matrix dimensions that will be names of variables in long-format

names(dimnames(Matrix))[1] = "Student"
names(dimnames(Matrix))[2] = "Practice"

Matrix

### Convert matrix to long-format

library(reshape2)

Data = melt(Matrix)

432
 COCHRAN’S Q TEST FOR PAIRED NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### View data frame

library(psych)

headTail(Data)

Student Practice value

1 a SoilTest 0
2 b SoilTest 1
3 c SoilTest 0
4 d SoilTest 0
... <NA> <NA> ...
53 k Clippings 0
54 l Clippings 1
55 m Clippings 0
56 n Clippings 1

### Note response variable is called "value"

Cochran’s Q test
library(RVAideMemoire)

cochran.qtest(value ~ Practice | Student,

data = Data)

Cochran's Q test

Q = 16.9535, df = 3, p-value = 0.0007225

library(coin)

symmetry_test(value ~ Practice | Student,

data = Data,
teststat = "quad")

Asymptotic General Symmetry Test

chi-squared = 16.953, df = 3, p-value = 0.0007225

Post-hoc analysis for Cochran’s Q test

### Order groups

Data$Practice = factor(Data$Practice,
levels = c("MowHeight", "SoilTest",
"Clippings", "Irrigation"))

### Pairwise McNemar tests

433
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

pairwiseMcnemar(x = Data$value,
g = Data$Practice,
block = Data$Student,
test = "permutation",
method = "fdr",
digits = 3)

### test = "exact", "mcnemar", "permutation"

### method: p-value adjustment, see ?p.adjust
### correct: TRUE, to apply continuity correction for McNemar test

$Pairwise
Comparison p.value p.adjust
1 MowHeight - SoilTest = 0 0.317 0.3170
2 MowHeight - Clippings = 0 0.102 0.1530
3 MowHeight - Irrigation = 0 0.00389 0.0200
4 SoilTest - Clippings = 0 0.257 0.3080
5 SoilTest - Irrigation = 0 0.00666 0.0200
6 Clippings - Irrigation = 0 0.0143 0.0286

Association Tests for Ordinal Tables

The linear-by-linear test can be used to test the association among variables in a contingency table
with ordered categories (Agresti, 2007). This test or a test with a similar function is sometimes called
“ordinal chi-square” test.

In Agresti, the method used is called the linear-by-linear association model. In R, the test can be
performed by permutation test with the coin package.

An association test can also be performed on a contingency table with one ordered nominal variable
and one non-ordered nominal variable. The Cochran–Armitage test is a special case of this when the
non-ordered variable has only two variables.

Most of the examples in this chapter use two-dimensional tables, although the coin package can handle
three-dimensional tables. For three-dimensional analyses, it may be easier to use data in the long
format, as is shown in the final example in this chapter.

Packages used in this chapter

The packages used in this chapter include:

• coin
• rcompanion

The following commands will install these packages if they are not already installed:

434
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(coin)){install.packages("coin")}
if(!require(rcompanion)){install.packages("rcompanion")}

Linear-by-linear test for ordered contingency tables

The lbl_test function in the coin package with automatically treat the variables as ordered, with the
levels in the table ordered from smallest to largest. By default, the levels are equally spaced, but the
scores option can be used to specify the distance between the levels of each variable.

The null hypothesis for the linear-by-linear test is that there is no association among the variables in
the table. A significant p-value suggests that there is an association. This is similar to a chi-square test,
except that the categories are ordered in nature.

Example of linear-by-linear test 1

For this hypothetical example, farmers were surveyed about how often they use some best
management practice. Responses are organized according to the size of the operation. Both variables
in the contingency table are ordered categories.

Note the placement of the initial quote mark in the Input function.
Input =(
"Adopt Always Sometimes Never
Size
Hobbiest 0 1 5
Mom-and-pop 2 3 4
Small 4 4 4
Medium 3 2 0
Large 2 0 0
")

Tabla = as.table(read.ftable(textConnection(Input)))

Tabla

sum (Tabla)

prop.table(Tabla,
margin = NULL) ### proportion in the table

Size Always Sometimes Never

Hobbiest 0.00000000 0.02941176 0.14705882
Mom-and-pop 0.05882353 0.08823529 0.11764706
Small 0.11764706 0.11764706 0.11764706
Medium 0.08823529 0.05882353 0.00000000
Large 0.05882353 0.00000000 0.00000000

library(coin)

spineplot(Tabla)

435
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Spine plot for each Size showing the proportion of Always (dark gray), Sometimes
(medium gray), and Never (light gray).

library(coin)

LxL = lbl_test(Tabla)

LxL

Asymptotic Linear-by-Linear Association Test

data: Adopt (ordered) by Size (Hobbiest < Mom-and-pop < Small < Medium < Large)

Z = -3.3276, p-value = 0.0008761

statistic(LxL)^2

11.07262

Compare to chi-square test without ordered categories

ChiSq = chisq_test(Tabla)

ChiSq

Asymptotic Pearson Chi-Squared Test

data: Adopt by Size (Hobbiest, Mom-and-pop, Small, Medium, Large)

436
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

chi-squared = 13.495, df = 8, p-value = 0.09593

Example of linear-by-linear test 2

The following example revisits the data from Converting Numeric Data to Categories chapter. It tests if
there is an association between Tired and Happy. Both variables in the contingency table are ordered
categories, with the levels of each being 1, 2, 3, 4, and 5.
Input =(
"Tired 1 2 3 5
Happy
1 0 0 0 3
2 0 0 0 2
3 0 0 3 0
4 2 0 0 0
5 3 2 0 5
")

Tabla = as.table(read.ftable(textConnection(Input)))

Tabla

sum (Tabla)

prop.table(Tabla,
margin = NULL) ### proportion in the table

Tired
Happy 1 2 3 5
1 0.00 0.00 0.00 0.15
2 0.00 0.00 0.00 0.10
3 0.00 0.00 0.15 0.00
4 0.10 0.00 0.00 0.00
5 0.15 0.10 0.00 0.25

library(coin)

LxL = lbl_test(Tabla)

LxL

Asymptotic Linear-by-Linear Association Test

data: Tired (ordered) by Happy (1 < 2 < 3 < 4 < 5)

Z = -1.8878, p-value = 0.05906

Extended Cochran–Armitage test

437
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The chisq_test function in the coin package can be used to conduct a test of association for a
contingency table with one ordered nominal variable and one non-ordered nominal variable. The
Cochran–Armitage test is a special case of this where the non-ordered variable has only two
categories.

The scores option is used to indicate which variable should be treated as ordered, and the spacing of
the levels of this variable.

Example of extended Cochran–Armitage test 1

For this hypothetical example, students were surveyed about how often they eat breakfast and how
they travel to school. Breakfast is treated as ordered, and Travel is treated as nominal.

Note the placement of the initial quote mark in the Input function.
Input =(
"Breakfast Never Rarely Sometimes Often Always
Travel
Walk 6 9 6 5 2
Bus 2 5 8 5 3
Drive 2 4 6 8 8
")

Tabla = as.table(read.ftable(textConnection(Input)))

Tabla

sum (Tabla)

prop.table(Tabla,
margin = 1) ### proportion in each row

Breakfast
Travel Never Rarely Sometimes Often Always
Walk 0.21428571 0.32142857 0.21428571 0.17857143 0.07142857
Bus 0.08695652 0.21739130 0.34782609 0.21739130 0.13043478
Drive 0.07142857 0.14285714 0.21428571 0.28571429 0.28571429

library(coin)

spineplot(Tabla)

438
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(coin)

Test = chisq_test(Tabla,
scores = list("Breakfast" = c(-2, -1, 0, 1, 2)))

Test

Asymptotic Generalized Pearson Chi-Squared Test

data: Breakfast (ordered) by Travel (Walk, Bus, Drive)

chi-squared = 8.6739, df = 2, p-value = 0.01308

statistic(Test)^2

[1] 75.23709

Post-hoc analysis
For a contingency table with one ordered variable and one non-ordered variable, it makes sense to
analyze the component tables with pairwise comparisons of the levels of the non-ordered variable.

Results of the compact letter display will be easier to interpret if the table is ordered so that the first
row, in this case, ranks highest or lowest in the ordered variable, and so on.

439
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

PT = pairwiseOrdinalIndependence(Tabla,
compare="row")

PT

Comparison p.value p.adjust

1 Walk : Bus 0.12800 0.1570
2 Walk : Drive 0.00462 0.0139
3 Bus : Drive 0.15700 0.1570

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.value,
threshold = 0.05)

Group Letter MonoLetter

1 Walk a a
2 Bus ab ab
3 Drive b b

Example of extended Cochran–Armitage test 2

This example revisits the example from Kruskal–Wallis chapter. Likert is treated as ordered, and
Speaker is treated as non-ordered.

Note the placement of the initial quote mark in the Input function.
Input =(
"Likert 1 2 3 4 5
Speaker
Pooh 0 0 1 6 3
Piglet 1 6 2 1 0
Tigger 0 0 2 6 2
")

Tabla = as.table(read.ftable(textConnection(Input)))

Tabla

sum(Tabla)

prop.table(Tabla,
margin = 1) ### proportion in each row

Likert
Speaker 1 2 3 4 5
Pooh 0.0 0.0 0.1 0.6 0.3
Piglet 0.1 0.6 0.2 0.1 0.0

440
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Tigger 0.0 0.0 0.2 0.6 0.2

library(coin)

Test = chisq_test(Tabla,
scores = list("Likert" = c(1, 2, 3, 4, 5)))

Test

Asymptotic Generalized Pearson Chi-Squared Test

data: Likert (ordered) by Speaker (Pooh, Piglet, Tigger)

chi-squared = 18.423, df = 2, p-value = 9.991e-05

library(rcompanion)

PT = pairwiseOrdinalIndependence(Tabla,
compare="row")

PT

Comparison p.value p.adjust

1 Pooh : Piglet 0.000310 0.000902
2 Pooh : Tigger 0.474000 0.474000
3 Piglet : Tigger 0.000601 0.000902

library(rcompanion)

cldList(comparison = PT$Comparison,
p.value = PT$p.value,
threshold = 0.05)

Group Letter MonoLetter

1 Pooh a a
2 Piglet b b
3 Tigger a a

Long-form data and three-dimensional contingency tables

The tests in this chapter can also be performed on data in long format rather than in table format.
Each row of data can represent a single observation, or one variable can hold counts of each category,
as the Count variable does below.

For the independence_test function in the coin package to handle ordered and non-ordered variables
properly, the user need only specify which variables are to be considered ordered.

The independence_test function can also handle a stratification variable. That is, a two-dimensional
table within each level of the stratification variable.
441
ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Input =("
Crop Size Adopt Count
Nursery Hobbiest Always 0
Nursery Hobbiest Sometimes 1
Nursery Hobbiest Never 3
Nursery Mom-and-pop Always 1
Nursery Mom-and-pop Sometimes 1
Nursery Mom-and-pop Never 2
Nursery Small Always 2
Nursery Small Sometimes 3
Nursery Small Never 2
Nursery Medium Always 2
Nursery Medium Sometimes 1
Nursery Medium Never 0
Nursery Large Always 1
Nursery Large Sometimes 0
Nursery Large Never 0
Vegetable Hobbiest Always 0
Vegetable Hobbiest Sometimes 0
Vegetable Hobbiest Never 2
Vegetable Mom-and-pop Always 1
Vegetable Mom-and-pop Sometimes 2
Vegetable Mom-and-pop Never 2
Vegetable Small Always 2
Vegetable Small Sometimes 1
Vegetable Small Never 2
Vegetable Medium Always 1
Vegetable Medium Sometimes 1
Vegetable Medium Never 0
Vegetable Large Always 1
Vegetable Large Sometimes 0
Vegetable Large Never 0
")

Data = read.table(textConnection(Input),header=TRUE)

### Tell R which variables are ordered

Data$Size = factor(Data$Size,
ordered = TRUE,
levels=unique(Data$Size))

Data$Adopt = factor(Data$Adopt,
ordered = TRUE,
levels=unique(Data$Adopt))

### Make sure the ordered levels are in the proper order

str(Data$Adopt)

Ord.factor w/ 3 levels "Always"<"Sometimes"<..: 1 2 3 1 2 3 1 2 3 1 ...

442
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

levels(Data$Adopt)

[1] "Always" "Sometimes" "Never"

str(Data$Size)

Ord.factor w/ 5 levels "Hobbiest"<"Mom-and-pop"<..: 1 1 1 2 2 2 3 3 3 4 ...

levels(Data$Size)

[1] "Hobbiest" "Mom-and-pop" "Small" "Medium" "Large"

Two-dimensional table example

XT = xtabs(Count ~ Size + Adopt,
data = Data)

XT

Adopt
Size Always Sometimes Never
Hobbiest 0 1 5
Mom-and-pop 2 3 4
Small 4 4 4
Medium 3 2 0
Large 2 0 0

spineplot(XT)

library(coin)

independence_test(Adopt ~ Size,
data = Data,
weights = ~ Count)

Asymptotic General Independence Test

data: Adopt (ordered) by Size (Hobbiest < Mom-and-pop < Small < Medium < Large)

Z = -3.3276, p-value = 0.0008761

Three-dimensional table example

The following example adds Crop as a stratification variable.
ftable(xtabs(Count ~ Crop + Size + Adopt,
data = Data))

Adopt Always Sometimes Never

Crop Size

443
 ASSOCIATION TESTS FOR ORDINAL TABLES SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Nursery Hobbiest 0 1 3
Mom-and-pop 1 1 2
Small 2 3 2
Medium 2 1 0
Large 1 0 0
Vegetable Hobbiest 0 0 2
Mom-and-pop 1 2 2
Small 2 1 2
Medium 1 1 0
Large 1 0 0

library(coin)

independence_test(Adopt ~ Size | Crop,

data = Data,
weights = ~ Count)

Asymptotic General Independence Test

data: Adopt (ordered) by Size (Hobbiest < Mom-and-pop < Small < Medium < Large)
stratified by Crop

Z = -3.281, p-value = 0.001034

Spine plot for two-dimensional tables

XT1 = xtabs(Count ~ Size + Adopt,
data = Data,
subset = Data$Crop=="Nursery")

XT1

spineplot(XT1,
main = "Nursery")

XT2 = xtabs(Count ~ Size + Adopt,

data = Data,
subset = Data$Crop=="Vegetable")

XT2

spineplot(XT2,
main = "Vegetable")

References

Agresti, A. 2007. An Introduction to Categorical Data Analysis 2nd Edition. Wiley-Interscience.

444
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Models for Nominal Data

Packages used in this chapter

The packages used in this chapter include:

• car
• multcompView
• lsmeans
• VGAM
• lmtest
• psych
• lme4
• MASS
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(VGAM)){install.packages("VGAM")}
if(!require(lmtest)){install.packages("lmtest")}
if(!require(psych)){install.packages("psych")}
if(!require(lme4)){install.packages("lme4")}
if(!require(MASS)){install.packages("MASS")}
if(!require(rcompanion)){install.packages("rcompanion")}

Log-linear models for tests of association

Analysis of contingency tables with two or more dimensions can be accomplished with log-linear
models. These will test for association. When there are multiple dimensions, this approach is
sometimes called multiway frequency analysis. It is analogous to chi-square tests for two-way tables or
Cochran–Mantel–Haenszel test for three way tables.

One advantage of log-linear models is that they handle tables with a higher number of dimensions.

Another advantage is that the models can be specified to test for more complicated patterns of
association. When each term in the model is included, and there are no interactions, the model tests
for mutual independence of the terms. Models can also test for conditional independence and partial
independence.

For more information on specifying models for conditional independence or partial independence, see:

Quick-R. 2014. Frequencies and Crosstabs. www.statmethods.net/stats/frequencies.html.

Agresti, A. 2007. An Introduction to Categorical Data Analysis 2nd Edition. Wiley-Interscience.

445
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For a more complete discussion of log-linear models for multiway frequency analysis see:

Tabachnick, B.G. and S.F. Fidell. 2001. Using multivariate statistics. 4th Edition. Allyn and Bacon,
Boston, MA.

Zero frequencies in cells may cause the maximum likelihood estimation to fail, or may cause bias in the
results.

Structural zeros can be handled with the start argument in the loglm function. See library(MASS);
?loglm , library(MASS); ?loglm1 , and ?loglin .

Log-linear model example

The following example revisits Alexander Anderson’s data of passing grades by sex within counties, for
which we had used the Cochran–Mantel–Haenszel test. Here we use a log-linear model with the loglm
function in the MASS package.
Input = ("
County Sex Result Count
Bloom Female Pass 9
Bloom Female Fail 5
Bloom Male Pass 7
Bloom Male Fail 17
Cobblestone Female Pass 11
Cobblestone Female Fail 4
Cobblestone Male Pass 9
Cobblestone Male Fail 21
Dougal Female Pass 9
Dougal Female Fail 7
Dougal Male Pass 19
Dougal Male Fail 9
Heimlich Female Pass 15
Heimlich Female Fail 8
Heimlich Male Pass 14
Heimlich Male Fail 17
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

Data$County = factor(Data$County,
levels=unique(Data$County))

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Data$Result = factor(Data$Result,
levels=unique(Data$Result))

446
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Create table
Table = xtabs(Count ~ Sex + Result + County,
data=Data)

ftable(Table) # Display a flattened table

County Bloom Cobblestone Dougal Heimlich

Sex Result
Female Pass 9 11 9 15
Fail 5 4 7 8
Male Pass 7 9 19 14
Fail 17 21 9 17

Log-linear model
The model here tests for mutual independence of the three variables.
library(MASS)

loglm( ~ Sex + Result + County,

Table)

Statistics:
X^2 df P(> X^2)
Likelihood Ratio 20.96662 10 0.0213275
Pearson 20.79050 10 0.0226027

Post-hoc analysis
For a post-hoc analysis, you could slice up a multi-dimensional table in any way that makes sense for
the hypotheses you want to test. Here, we’ll look for an association of Sex and Result within each
county.

Third dimension in our table is County, so Table[,,1] yields the piece of the table where the third
dimension is equal to 1, that is, where County = Bloom. And so on.

447
MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Bloom = Table[,,1]

loglm( ~ Sex + Result,

Bloom)

X^2 df P(> X^2)

Likelihood Ratio 4.504069 1 0.03381429
Pearson 4.473688 1 0.03442062

Cobblestone = Table[,,2]

loglm( ~ Sex + Result,

Cobblestone)

X^2 df P(> X^2)

Likelihood Ratio 7.777229 1 0.005290890
Pearson 7.605000 1 0.005820666

Dougal = Table[,,3]

loglm( ~ Sex + Result,

Dougal)

X^2 df P(> X^2)

Likelihood Ratio 0.5876125 1 0.4433438
Pearson 0.5927934 1 0.4413410

Heimlich = Table[,,4]

loglm( ~ Sex + Result,

Heimlich)

X^2 df P(> X^2)

Likelihood Ratio 2.158817 1 0.1417538
Pearson 2.136182 1 0.1438595

Summary of analysis from Cochran–Mantel–Haenszel tests and log-linear models

County C-H-M log-linear
p-value p-value (lr)
Bloom 0.0468 0.03381
Cobblestone 0.0102 0.00529
Dougal 0.5230 0.4433
Heimlich 0.1750 0.14175

Logistic regression

448
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Logistic regression is a common option for building models with a nominal dependent variable. For
standard logistic regression, the dependent variable must have only two levels. That is, it must be
dichotomous.

The following references should be useful for conducting logistic regression.

“Simple Logistic Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_06.html.

“Multiple Logistic Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_07.html.

“Simple logistic regression” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/simplelogistic.html.

“Multiple logistic regression” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/multiplelogistic.html.

Logistic regression example

Note that model assumptions and pitfalls of logistic regression are not discussed here. The reader is
urged to read the preceding references or other appropriate materials before proceeding.

The following example revisits Alexander Anderson pesticide safety training course data across sex
and four counties, for which we had used the Cochran–Mantel–Haenszel Test.
Input = ("
County Sex Result Count
Bloom Female Pass 9
Bloom Female Fail 5
Bloom Male Pass 7
Bloom Male Fail 17
Cobblestone Female Pass 11
Cobblestone Female Fail 4
Cobblestone Male Pass 9
Cobblestone Male Fail 21
Dougal Female Pass 9
Dougal Female Fail 7
Dougal Male Pass 19
Dougal Male Fail 9
Heimlich Female Pass 15
Heimlich Female Fail 8
Heimlich Male Pass 14
Heimlich Male Fail 17
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

449
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$County = factor(Data$County,
levels=unique(Data$County))

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Data$Result = factor(Data$Result,
levels=unique(Data$Result))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Logistic regression
Logistic regression can be conducted with the glm function in the native stats package. The summary
function will return coefficients for the model. A p-value and a pseudo R-squared value for the model
can be produced with the nagelkerke function. The Anova function in the car package will produce an
analysis of deviance table with either likelihood ratio, Wald, or F tests.
model = glm(Result ~ County + Sex + County:Sex,
weight = Count,
data = Data,
family = binomial(link="logit")
)

summary(model)

library(rcompanion)

nagelkerke(model)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.0797857
Cox and Snell (ML) 0.7136520
Nagelkerke (Cragg and Uhler) 0.7136520

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value

450
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

-7 -10.004 20.009 0.0055508

library(car)

Anova(model,
type="II",
test="LR")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
County 4.9627 3 0.174548
Sex 7.8108 1 0.005193 **
County:Sex 7.2169 3 0.065296 .

Post-hoc analysis
Post-hoc analysis can be conducted with the lsmeans package. Note that estimates for lsmeans,
differences, or standard errors are on a log or logit scale. For notes on using lsmeans with different
model types, see ?lsmeans::models.

For this example, no adjustment in p-values was made for multiple comparisons. Typically, an
adjustment such as Tukey would be applied. See ?lsmeans::summary for p-value adjustment options in
lsmeans. Note that the adjustment should be applied for each output; that is, for each of lsmeans and
cld.

Also note that normally, because the County x Sex interaction was not significant, we wouldn’t want to
explore the mean separations of the County x Sex interaction.

The results are truncated to emphasize the differences between sexes within counties.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ County:Sex,
adjust="none") ### No adjustment for multiple comparisons

leastsquare

$contrasts
contrast estimate SE df z.ratio p.value

Bloom,Female - Bloom,Male -1.47508986 0.7160935 NA -2.060 0.0394

Cobblestone,Female - Cobblestone,Male -1.85889877 0.7068488 NA -2.630 0.0085

Dougal,Female - Dougal,Male 0.49589997 0.6463052 NA 0.767 0.4429

Heimlich,Female - Heimlich,Male -0.82276467 0.5673787 NA -1.450 0.1470

451
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Results are given on the log (not the response) scale.

Tests are performed on the log scale

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="none") ### No adjustment for multiple comparisons

County Sex lsmean SE df asymp.LCL asymp.UCL .group

Cobblestone Female -1.0116009 0.5838720 NA -2.155968949 0.13276713 a
Dougal Male -0.7472144 0.4046506 NA -1.540315071 0.04588627 a
Heimlich Female -0.6286087 0.4377974 NA -1.486675732 0.22945841 a
Bloom Female -0.5877867 0.5577733 NA -1.681002268 0.50542894 a
Dougal Female -0.2513144 0.5039526 NA -1.239043434 0.73641458 ab
Heimlich Male 0.1941560 0.3609046 NA -0.513203923 0.90151595 ab
Cobblestone Male 0.8472979 0.3984077 NA 0.066433148 1.62816257 b
Bloom Male 0.8873032 0.4490867 NA 0.007109497 1.76749689 b

Results are given on the logit (not the response) scale.

Confidence level used: 0.95
Results are given on the log (not the response) scale.
Tests are performed on the log scale
significance level used: alpha = 0.05

Summary table of results

Comparison p-value
Bloom–Female - Bloom–Male 0.039
Cobblestone–Female - Cobblestone–Male 0.0085
Dougal–Female - Dougal–Male 0.44
Heimlich–Female - Heimlich–Male 0.15

p.value = c(0.039, 0.0085, 0.44, 0.15)

p.adj = p.adjust(p.value,
method = "fdr")

p.adj = signif(p.adj,
2)

p.adj

[1] 0.078 0.034 0.440 0.200

Comparison p-value p.adj

Bloom–Female - Bloom–Male 0.039 0.078
Cobblestone–Female - Cobblestone–Male 0.0085 0.034
Dougal–Female - Dougal–Male 0.44 0.44
Heimlich–Female - Heimlich–Male 0.15 0.20

452
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Multinomial logistic regression

If the nominal dependent variable has more than two levels, multinomial logistic regression can be
used.

VGAM package
The VGAM package provides a flexible framework for building models with categorical data. The
vignette for the package provides an overview of the package, as well as brief overview of categorical
analysis in R.

Yee, T.W. The VGAM package for Categorical Data Analysis. The Comprehensive R Archive
Network. cran.r-project.org/web/packages/VGAM/vignettes/categoricalVGAM.pdf.

The following article may also be helpful for those interested in using the VGAM package.

Yee, T.W. 2008. The VGAM Package. R News, 8(2), 28–39. URL. cran.r-
project.org/doc/Rnews/Rnews_2008-2.pdf.

The VGAM package can be explored through the help files for the package.
if(!require(VGAM)){install.packages("VGAM")}

help(package="VGAM")

And multinomial regression is discussed specifically in the entry for multinomial.

library(VGAM)

?multinomial

mlogit and nnet packages

Two other packages that can perform multinomial regression are mlogit and nnet. The packages can
be explored with the following vignette, article, and the package help files.

Croissant, Y. Estimation of multinomial logit models in R: The mlogit Packages. cran.r-

project.org/web/packages/mlogit/vignettes/mlogit.pdf.
if(!require(mlogit)){install.packages("mlogit")}
help(package="mlogit")

[IDRE] Institute for Digital Research and Education. 2015. “R Data Analysis Examples:
Multinomial Logistic Regression”. UCLA. www.ats.ucla.edu/stat/r/dae/mlogit.htm.
if(!require(nnet)){install.packages("nnet")}
help(package="nnet")

453
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Multinomial regression example

The following example revisits Alexander Anderson’s data of passing grades by sex within counties, for
which we had used the Cochran–Mantel–Haenszel Test.

Because the dependent variable, Result, has only two levels, it could be modeled with standard
binomial regression. However, a multinomial approach with VGAM will be used here as an example.
Note that in multinomial regression, a reference level for the dependent variable must be specified.
Coefficients of the model will change if a different reference level is specified.

Note that model assumptions and pitfalls of this approach are not discussed here. The reader is urged
to understand the assumptions of this kind of modeling before proceeding.

The summary function will provide estimates of coefficients and standard errors for the coefficients. A
p-value for the overall model as well as pseudo R-squared value is provided by the nagelkerke function.

The lrtest function in VGAM can be used to compare two nested models with likelihood ratio test. If
only one model is given, it provides a p-value for the overall model, compared with a null model. The
Anova function in the car package will provide chi-square tests for individual factors in the model. The
user might prefer to use likelihood ratio tests with nested models instead of chi-square tests. This
could be accomplished with the lrtest function in the lmtest package, or by specifying the second model
with the null= option in the nagelkerke function.
Input = ("
County Sex Result Count
Bloom Female Pass 9
Bloom Female Fail 5
Bloom Male Pass 7
Bloom Male Fail 17
Cobblestone Female Pass 11
Cobblestone Female Fail 4
Cobblestone Male Pass 9
Cobblestone Male Fail 21
Dougal Female Pass 9
Dougal Female Fail 7
Dougal Male Pass 19
Dougal Male Fail 9
Heimlich Female Pass 15
Heimlich Female Fail 8
Heimlich Male Pass 14
Heimlich Male Fail 17
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

Data$County = factor(Data$County,
levels=unique(Data$County))

454
MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Sex = factor(Data$Sex,
levels=unique(Data$Sex))

Data$Result = factor(Data$Result,
levels=unique(Data$Result))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Multinomial regression
library(VGAM)

model = vglm(Result ~ Sex + County + Sex:County,

family=multinomial(refLevel=1),
weights = Count,
data = Data)

summary(model)

library(car)

Anova(model,
type="II",
test="Chisq")

Analysis of Deviance Table (Type II tests)

Response: Result
Df Chisq Pr(>Chisq)
Sex 1 6.7132 0.00957 **
County 3 4.1947 0.24120
Sex:County 3 7.1376 0.06764 .

Library(rcompanion)

nagelkerke(model)

455
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.0797857
Cox and Snell (ML) 0.7136520
Nagelkerke (Cragg and Uhler) 0.7136520

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
7 -10.004 20.009 0.0055508

library(lmtest)

lrtest(model)

Likelihood ratio test

Model 1: Result ~ Sex + County + Sex:County

Model 2: Result ~ 1

#Df LogLik Df Chisq Pr(>Chisq)

1 8 -115.39
2 15 -125.39 7 20.009 0.005551 **

Post-hoc analysis
At the time of writing, the lsmeans package cannot be used with vglm models.

One option for post-hoc analysis would be to conduct analyses on reduced models, including only two
levels of a factor. For example, if the variable County x Sex term had been significant, the following
code could be used to create a reduced dataset with only Bloom–Female and Bloom–Male, and analyze
this data with vglm.
Data.b = Data[Data$County=="Bloom" &
(Data$Sex=="Female"| Data$Sex=="Male") , ]

Data.b$County = factor(Data.b$County)
Data.b$Sex = factor(Data.b$Sex)

summary(Data.b)

County Sex Result Count

Bloom:4 Female:2 Pass:2 Min. : 5.0
Male :2 Fail:2 1st Qu.: 6.5
Median : 8.0
Mean : 9.5
3rd Qu.:11.0
Max. :17.0

library(VGAM)

456
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

model.b = vglm(Result ~ Sex,

family=multinomial(refLevel=1),
weights = Count,
data = Data.b)

lrtest(model.b)

Likelihood ratio test

#Df LogLik Df Chisq Pr(>Chisq)

1 2 -23.612
2 3 -25.864 1 4.5041 0.03381 *

Summary table of results

Comparison p-value
Bloom–Female - Bloom–Male 0.034
Cobblestone–Female - Cobblestone–Male 0.0052
Dougal–Female - Dougal–Male 0.44
Heimlich–Female - Heimlich–Male 0.14

p.value = c(0.034, 0.0052, 0.44, 0.14)

p.adj = p.adjust(p.value,
method = "fdr")

p.adj = signif(p.adj,
2)

p.adj

[1] 0.068 0.021 0.440 0.190

Comparison p-value p.adj

Bloom–Female - Bloom–Male 0.034 0.068
Cobblestone–Female - Cobblestone–Male 0.0052 0.021
Dougal–Female - Dougal–Male 0.44 0.44
Heimlich–Female - Heimlich–Male 0.14 0.19

Mixed-effects logistic regression example

A mixed-effects generalized linear model, as in the case of logistic regression with random effects, can
be specified. This example revisits Hayley Smith’s friendly lawn care course, for which we had used
Cochran’s Q test.

This example uses the glmer function in the package mle4, which can fit binomial dependent variables,
with the binomial family of models, or other families of models. As far as I know, it will not fit
multinomial regression. For more information on families of models, see ?family and ?glm. For more

457
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

information on glmer, see ?glmer. For a list of methods used to extract information from the model
object, see methods(class="merMod").

The Anova function in the car package will provide chi-square tests for individual factors in the model.
The user might prefer to use likelihood ratio tests with nested models instead of chi-square tests. This
could be accomplished with the anova function, or by specifying the second model with the null=
option in the nagelkerke function.
Input = ("
Practice Student Response
SoilTest a Yes
SoilTest b No
SoilTest c Yes
SoilTest d Yes
SoilTest e No
SoilTest f Yes
SoilTest g Yes
SoilTest h Yes
SoilTest i No
SoilTest j Yes
SoilTest k Yes
SoilTest l Yes
SoilTest m Yes
SoilTest n Yes
Irrigation a Yes
Irrigation b No
Irrigation c No
Irrigation d No
Irrigation e No
Irrigation f No
Irrigation g No
Irrigation h No
Irrigation i Yes
Irrigation j No
Irrigation k No
Irrigation l No
Irrigation m No
Irrigation n No
MowHeight a Yes
MowHeight b No
MowHeight c Yes
MowHeight d Yes
MowHeight e Yes
MowHeight f Yes
MowHeight g Yes
MowHeight h Yes
MowHeight i No
MowHeight j Yes
MowHeight k Yes
MowHeight l Yes
MowHeight m Yes
MowHeight n Yes
Clippings a Yes
Clippings b No

458
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Clippings c No
Clippings d Yes
Clippings e Yes
Clippings f No
Clippings g No
Clippings h Yes
Clippings i Yes
Clippings j Yes
Clippings k Yes
Clippings l No
Clippings m Yes
Clippings n No
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors otherwise R will alphabetize them

Data$Practice = factor(Data$Practice,
levels=unique(Data$Practice))

Data$Response = factor(Data$Response,
levels=c("Yes", "No"))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Mixed-effects logistic regression

library(lme4)

model = glmer(Response ~ Practice + (1 | Student),

data = Data,
family = binomial,
nAGQ = 1) ### Must be 1 to compare with glm model objects

summary(model)

library(car)

459
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Anova(model,
type="II",
test="Chisq")

Analysis of Deviance Table (Type II Wald chisquare tests)

Chisq Df Pr(>Chisq)
Practice 10.713 3 0.01338 *

### p-value for model and pseudo R-squared

model.null = glm(Response ~ 1,
data = Data,
family = binomial)

library(rcompanion)

nagelkerke(model,
model.null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.258315
Cox and Snell (ML) 0.295184
Nagelkerke (Cragg and Uhler) 0.397900

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-4 -9.7949 19.59 0.00060164

### Test the random effects

model.fixed = glm(Response ~ Practice,

data = Data,
family = binomial)

anova(model, model.fixed)

model.fixed: Response ~ Practice

model: Response ~ Practice + (1 | Student)

Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)

model.fixed 4 64.636 72.738 -28.318 56.636
model 5 66.247 76.374 -28.124 56.247 0.3889 1 0.5329

Post-hoc analysis
Post-hoc analysis can be conducted with the lsmeans package. Note that estimates for lsmeans,
differences, or standard errors are on a log or logit scale. For notes on using lsmeans with different
model types, see ?lsmeans::models.

460
 MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Practice,
adjust="tukey")
### Tukey adjustment for multiple comparisons

leastsquare

$contrasts
contrast estimate SE df z.ratio p.value
SoilTest - Irrigation -3.4004979 1.1888138 NA -2.860 0.0220
SoilTest - MowHeight 0.5256967 1.0384143 NA 0.506 0.9576
SoilTest - Clippings -1.1165227 0.9087443 NA -1.229 0.6086
Irrigation - MowHeight 3.9261946 1.2853129 NA 3.055 0.0121
Irrigation - Clippings 2.2839752 1.0431115 NA 2.190 0.1261
MowHeight - Clippings -1.6422194 1.0071891 NA -1.630 0.3614

Results are given on the log (not the response) scale.

P value adjustment: tukey method for comparing a family of 4 estimates
Tests are performed on the log scale

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey adjustment for multiple comparisons

MowHeight -1.9640077 0.8712261 NA -4.1341580 0.2061426 a

SoilTest -1.4383110 0.7498776 NA -3.3061924 0.4295705 a
Clippings -0.3217883 0.6055520 NA -1.8301670 1.1865905 ab
Irrigation 1.9621870 0.8672121 NA -0.1979648 4.1223388 b

Results are given on the logit (not the response) scale.

Confidence level used: 0.95

P value adjustment: tukey method for comparing a family of 4 estimates

Tests are performed on the log scale
significance level used: alpha = 0.05

Other tools for categorical analysis

The vcd package has several function that are useful when working with categorical data. The package
can be explored with the following vignette and the package help files.

Friendly, M. Working with categorical data with R and the vcd and vcdExtra packages. The
Comprehensive R Archive Network. cran.r-
project.org/web/packages/vcdExtra/vignettes/vcd-tutorial.pdf.

461
MODELS FOR NOMINAL DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

if(!require(vcd)){install.packages("vcd")}
help(package="vcd")

462
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Parametric Tests

Introduction to Parametric Tests

Packages used in this chapter

The packages used in this chapter include:

• psych
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(rcompanion)){install.packages("rcompanion")}

When to use parametric tests

Parametric statistical tests are among the most common you’ll encounter. They include t-test, analysis
of variance, and linear regression.

They are used when the dependent variable is a measured interval/ratio data variable. This might
include variables measured in science such as fish length, child height, crop yield weight, or pollutant
concentration in water.

One advantage of using parametric statistical tests is that your audience will likely be familiar with the
techniques and interpretation of the results. These tests are also often more flexible and more
powerful than their nonparametric analogues.

Their major drawback is that all parametric tests assume something about the distribution of the
underlying data. If these assumptions are violated, the resultant test statistics will not be valid, and
the tests will not be as powerful as for cases when assumptions are met.

Count data may not be appropriate for common parametric tests

A frequent error is to use common parametric models and tests with count data for the dependent
variable. Instead, count data could be analyzed either by using tests for nominal data or by using
regression methods appropriate for count data. These include Poisson regression, negative binomial
regression, and zero-inflated Poisson regression. The Hermite and Poisson Regression for Count Data
chapter uses Hermite and Poisson regression for analysis of count data.

Commonly this is the case when subjects are being counted. An example of this might be counting the
number of students who pass or fail a test given one of two instruction methods.

When to use parametric tests for count data

It is sometimes permissible to use common parametric tests for count data or other discrete data.
They can be used in cases where counts are used as a type of measurement of some property of
subjects, provided that 1) the distribution of data or residuals from the analysis meet test
assumptions; and 2) there are few or no counts at or close to zero, or close to a maximum, if one exists.

463
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Permissible examples might include test scores, age, or number of steps taken during the day.
Technically, each of these measurements is bound by zero, and are discrete rather than continuous
measurements. However, if other conditions are met, it is reasonable to handle them as if they were
continuous measurement variables.

This kind of count data will sometimes need to be transformed to meet the assumptions of parametric
analysis. Square root and logarithmic transformations are common. However, if there are many
counts at or near zero, transformation is unlikely to help. With the use of computers, we have
appropriate methods for count data available to us. It is therefore usually not worth the effort to
attempt to force count data to meet the assumptions of parametric analysis with transformations.

Percentage and proportion data

Percentage and proportion data are often inappropriate for parametric statistics for the same reasons
given for count data. They will often not meet the assumptions of the tests, and are particularly
problematic if there are some or many observations close to 0 or 1. The arcsine transformation was
used traditionally, but logistic regression or beta regression may be more appropriate.

Assumptions in parametric statistics

All parametric analyses have assumptions about the underlying data, and these assumptions should
always be confirmed when using these tests. If these assumptions are violated, the resulting statistics
and conclusions will not be valid, and the tests may lack power relative to alternative tests.

The exact assumptions vary among tests, but a general discussion follows.

For examples in this section, we’ll revisit the Catbus data.

Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8
n female Satsuki 8000 9
o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7

464
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

v female Totoro 6000 7

w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

Data

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Random sampling
All statistical tests assume that the data captured in the sample are randomly chosen from the
population as a whole. Selection bias will obviously affect the validity of the outcome of the analysis.

Independent observations
Tests will also assume that observations are independent of one another, except when the analysis
takes non-independence into account. One common case of non-independent observations is in
repeated measures experiments, in which the same subject is observed over time. If you were
measuring test scores of students over time, you might expect students with a high test score on one
date to have a high test score on subsequent dates. In this case the observation on one date would not
be independent of observations on other dates.

The independence of observation is often assumed from good experimental design. Also, data or
residuals can be plotted, for example to see if observations from one date are correlated to those for
another date.

Normal distribution of data or residuals

Parametric tests assume that the data come from a population of known distribution. The tests
discussed in this section assume that the data or errors are normally distributed.

A select number of tests will require that data itself be normally distributed. This will be limited to
one-sample t-test, two-sample t-test, and paired t-test. For other tests, the distribution of the residuals
will be investigated.

465
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Residuals from an analysis are also commonly called errors. They are the difference between the
observations and the value predicted by the model. For example, if the calculated mean of a sample is
10, and one observation is 12, the residual for this observation is 2. If another observation is 7, the
residual for this observation is –3.

The parametric tests discussed here require that the residuals be normally distributed. This will
usually be assessed with a histogram of residuals, a density plot as shown below, or with quantile–
quantile plot.

Be careful not to get confused about this assumption. You may see discussion about how “data” should
be normally distributed for parametric tests. This is usually wrong-headed. The t-test assumes that
the observations for each group are normally distributed, but if there is a difference in the groups, we
might expect a bi-modal distribution, not a simple normal distribution, for the combined data. This is
why in most cases we look at the distribution of the residuals, not the raw data.

### Density plot by sex for Catbus data set

ggplot(Data,
aes(Steps, fill = Sex)) +
geom_density(position="dodge",
alpha = 0.6)

The following code creates a variable Mean for Steps for each Sex in the Catbus data set, and then
subtracts Steps from each mean, calling the result Residual. This is for illustrative purposes only; you
won’t normally need to manipulate data in this manner.
M1 = mean(Data$Steps[Data$Sex=="female"])
M2 = mean(Data$Steps[Data$Sex=="male"])

Data$Mean[Data$Sex=="female"] = M1

466
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Mean[Data$Sex=="male"] = M2

Data$Residual = Data$Steps - Data$Mean

### Density plot of residuals from mean for each sex for Catbus data set

ggplot(Data,
aes(Residual)) +
geom_density(fill = "gray")

Homogeneity of variance
Parametric analyses will also assume a homogeneity of variance among groups. That is, for a t-test
comparing two groups, each group should have the same variance.

A good approach to assess this assumption is to plot residuals vs. predicted values. The residuals
should have approximately the same spread across all predicted values.

Homogeneity of variance is also called homoscedasticity. The opposite is heteroscedasticity.

### Residuals from mean for each sex vs. mean for Catbus data set

plot(jitter(Residual) ~ Mean,
data = Data)

467
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Additivity of treatment effects

Models for two-way analysis of variance and similar analyses are constructed as linear models in
which the dependent variable is predicted as a linear combination of the independent variables.

A violation of this assumption is sometimes indicated when a plot of residuals versus predicted values
exhibits a curved pattern.

Outliers
Outliers are observations whose value is far outside what is expected. They can play havoc with
parametric analyses since they affect the distribution of the data and strongly influence the mean.

There are a variety of formal tests for detecting outliers, but they will not be discussed here. The best
approach is one that looks at residuals after an analysis. Good tools are the “Residuals vs. leverage”
plot and other plots in the “Other diagnostic plots” section below.

It’s my opinion that outliers should not be removed from data unless there is a good reason, usually
when a value is impossible or a measurement error of some kind is suspected.

Parametric tests are somewhat robust

Some parametric tests are somewhat robust to violations of certain assumptions. For example, the t-
test is reasonably robust to violations of normality for symmetric distributions, but not to samples
having unequal variances. A one-way analysis of variance is likewise reasonably robust to violations
in normality.

The upshot is that model assumptions should always be checked, but you may be able to tolerate small
violations in the distribution of residuals or homoscedasticity. Large violations will make the test
invalid, though. It is important to be honest with your assessments when checking model
assumptions. It is better to transform data, change your model, use a robust method, or use a
nonparametric test than to not have confidence in your analysis.

468
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Assessing model assumptions

For this example, we’ll revisit the Catbus data. We’ll then define a linear model where Steps is the
dependent variable and Sex and Teacher are the independent variables.
Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8
n female Satsuki 8000 9
o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7
v female Totoro 6000 7
w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

469
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Define a linear model

model = lm(Steps ~ Sex + Teacher,
data = Data)

Using formal tests to assess normality of residuals

There are formal tests to assess the normality of residuals. Common tests include Shapiro-Wilk,
Anderson–Darling, Kolmogorov–Smirnov, and D’Agostino–Pearson. These are presented in the
“Optional analyses: formal tests for normality” section.

In general, I don’t recommend using these tests because their results are dependent on sample size.
When the sample size is large, the tests may indicate a statistically significant departure from
normality, even if that departure is small. And when sample sizes are small, they won’t detect
departures from normality.

The article from the Fells Stats blog, in the “References” section has pretty convincing examples of
these problems.

Skew and kurtosis

There are no definitive guidelines as to what range of skew or kurtosis are acceptable for considering
residuals to be normally distributed.

In general, I would not recommend relying on skew and kurtosis calculations, but instead use
histograms and other plots. In the chapter on Normality in the “Optional readings” section, John
McDonald recommends not calculating skew or kurtosis at all.

If I were forced to give advice for skewness calculations, I might say, be cautious if the absolute value is
> 0.5, and consider it not normally distributed if the absolute value is > 1.0. Some authors use 2.0 as a
cutoff for normality, and others use a higher limit for kurtosis.
x = residuals(model)

library(psych)

describe(x,
type=2) # Type of skew and kurtosis

vars n mean sd median trimmed mad min max range skew kurtosis
1 1 26 0 1132.26 -39.58 11.07 1114.18 -2202.4 2123.25 4325.65 -0.13 -0.11

Using visual inspection to assess the normality of residuals

Usually, the best method to see if model residuals meet the assumptions of normal distribution and
homoscedasticity are to plot them and inspect the plots visually.

470
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histogram with normal curve

A histogram of the residuals should be approximately normal, without excessive skew or kurtosis.
Adding a normal curve with the same mean and standard deviation as the data helps to assess the
histogram.
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

Kernel density plot with normal curve

A kernel density plot is similar to a histogram, but is smoothed into a curve. Sometimes a density plot
gives a better representation of the distribution of data, because the appearance of the histogram
depends upon how many bins are used.

The plotNormalDensity function will produce this plot. Options include those for the plot function, as
well as adjust, bw, and kernel which are passed to the density function. col1, col2, and col3 change plot
colors, and lwd changes line thickness.
x = residuals(model)

library(rcompanion)

plotNormalDensity(x,
adjust = 1) ### Decrease this number
### to make line less smooth

471
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plot of residuals vs. fitted values

Patterns in the plot of residuals versus fitted values can indicate a lack of homoscedasticity or that
errors are not independent of fitted values.
plot(fitted(model),
residuals(model))

Examples of residual plots and histograms

In the four plots below, A) Residuals.a show normally distributed and homoscedastic residuals,
suggesting model assumptions were met. B) Residuals.b show a non-normal distribution of residuals.
C) Residuals.c show that the residuals are not independent of the fitted values. In this case, the model
needs to be modified in order to describe the data well. D) Residuals.d show heteroscedasticity, since
variability in the residuals is greater for large fitted values than for small fitted values. (Adapted from
similar plots in Tabachnick, 2001).

472
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The following plots are histograms of the same residuals shown in the previous plots. A) Residuals.a
are reasonably-close to normally distributed. B) Residuals.b are highly skewed (left, or negative). C)
Residuals.c are moderately negatively skewed. This distribution would probably not cause too much
havoc with most parametric tests, but, depending on the circumstances, I would probably try to
transform a variable or find a better-fitting model. D) Residuals.d are symmetric, but leptokurtic. I
probably wouldn’t be too concerned with the distribution.

Results from the describe function in the psych package are shown below the histograms.

473
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### describe(Residuals.a)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 200 0 6.82 0.63 0.07 6.91 -20 18.51 38.51 -0.14 0.03 0.48

### describe(Residuals.b)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 200 0 3.77 1.38 0.86 1.62 -20 2.62 22.62 -2.53 7.2 0.27

### describe(Residuals.c)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 200 0 6.79 1.25 0.48 7.75 -20 12.88 32.88 -0.54 -0.42 0.48

### describe(Residuals.d)

vars n mean sd median trimmed mad min max range skew kurtosis se
1 1 200 0 5.53 0.11 0.21 3.83 -20 16.57 36.57 -0.48 1.94 0.39

Other diagnostic plots

Using the plot function for most common types of models will show diagnostic plots or other useful
plots for the model. For linear models, the plot function produces plots of 1) residuals vs. fitted values;
2) normal quantile–quantile plot; 3) square root of standardized residuals vs. fitted values; and 4)
standardized residuals vs. leverage.

While these are useful diagnostic plots, their interpretation will not be discussed here.
plot(model)

474
INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

475
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Steps to handle violations of assumptions

If residuals show heteroscedasticity or a non-normal distribution, the analysis must be modified in
some way to get valid results. Some options include the following.

1. Select the correct parametric test and check assumptions

2. Change your model

Problems with distribution of residuals is sometimes related to the model not describing the data well.
Adding or removing terms from the model may correct this. As an example, imagine you had plotted
bivariate data and they formed a shape like a parabola. If you fit a straight line to this data, you might
get residuals that look like Residuals.c above. Adding an x2 term, or other term to add curvature to the
fitted model, might correct this. A similar situation can be encountered in analysis of variance and
similar tests; it is sometimes helpful to add other terms or covariates.

3. Transform data and repeat your original analysis

The dependent variable, or other variables, can be transformed so that model assumptions are met.
The tests are then performed on these transformed data. See the Transforming Data chapter for
details.

4. Use classic nonparametric tests

Classic nonparametric tests include those in the Ordinal Tests section of this book. These tests are
mostly limited to analogues for t-test, paired t-test, one-way anova, and anova for unreplicated
complete block design.

5. Use robust methods.

There are other statistical methods which are robust to violations of parametric assumptions or are
nonparametric. One example is the cumulative link models discussed in the Ordinal Tests with
Cumulative Link Models section of this book. Other examples include permutation tests and robust
estimation. Permutation tests are discussed in the Introduction to Permutation Tests chapter. See also
Mangiafico (2015a) and (2015b) in the “References” section for examples.

476
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Descriptive Statistics for Parametric Statistics

Descriptive statistics for interval/ratio data are discussed in the “Descriptive statistics for
interval/ratio data” section in the Descriptive Statistics chapter.

Descriptive plots for interval/ratio data are discussed in the “Examples of basic plots for interval/ratio
and ordinal data” section in the Basic Plots chapter.

Optional readings

“Normality” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/normality.html.

“Independence” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/independence.html.

“Homoscedasticity and heteroscedasticity” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/homoscedasticity.html.

References

“One-way Analysis with Permutation Test” in Mangiafico, S.S. 2015a. An R Companion for the Handbook
of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_06a.html.

“Two-way Anova with Robust Estimation” in Mangiafico, S.S. 2015b. An R Companion for the Handbook
of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_08a.html.

Tabachnick, B.G. and S.F. Fidell. 2001. Using multivariate statistics. 4th Edition. Allyn and
Bacon, Boston, MA.

“Normality tests don't do what you think they do.” 2012. Fells Stats. blog.fellstat.com/?p=61.

Optional analyses: formal tests for normality

Code for conducting formal tests of normality are included here. I don’t recommend using them, but
they may be instructive in training the eye to interpret histograms and Q-Q plots, or to respond to a
less-savvy reviewer.

In each case, the null hypothesis is that the data distribution is not different from normal. That is, a
significant p-value (p < 0.05) suggests that data are not normally distributed.

As mentioned previously, a limitation to using these tests is that as the number of observations are
increased, the ability of the test to return a significant p-value increases, even for small deviations
from a normal distribution.

477
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8
n female Satsuki 8000 9
o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7
v female Totoro 6000 7
w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Define a linear model

model = lm(Steps ~ Sex + Teacher,
data = Data)

478
INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Shapiro–Wilk normality test

x = residuals(model)

shapiro.test(x)

Shapiro-Wilk normality test

W = 0.96256, p-value = 0.4444

Anderson-Darling normality test

if(!require(nortest)){install.packages("nortest")}

library(nortest)

x = residuals(model)

ad.test(x)

Anderson-Darling normality test

A = 0.39351, p-value = 0.3506

One-sample Kolmogorov-Smirnov test

x = residuals(model)

ks.test(x,
"pnorm",
mean = mean(x),
sd = sd(x))

One-sample Kolmogorov-Smirnov test

D = 0.1399, p-value = 0.6889

D'Agostino Normality Test

if(!require(fBasics)){install.packages("fBasics")}

library(fBasics)

x = residuals(model)

dagoTest(x)

Title:
D'Agostino Normality Test

479
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Test Results:
STATISTIC:
Chi2 | Omnibus: 0.1071
Z3 | Skewness: -0.3132
Z4 | Kurtosis: 0.0948
P VALUE:
Omnibus Test: 0.9479
Skewness Test: 0.7541
Kurtosis Test: 0.9245

Optional analyses: formal tests for homogeneity of variance

Code for formal tests of homogeneity of variance among groups is presented here. I don’t recommend
using them, but instead recommend using diagnostic plots.

In each case, the null hypothesis is that the variance among groups is not different. That is, a
significant p-value (p < 0.05) suggests that the variance among groups is different.
Input = ("
Student Sex Teacher Steps Rating
a female Catbus 8000 7
b female Catbus 9000 10
c female Catbus 10000 9
d female Catbus 7000 5
e female Catbus 6000 4
f female Catbus 8000 8
g male Catbus 7000 6
h male Catbus 5000 5
i male Catbus 9000 10
j male Catbus 7000 8
k female Satsuki 8000 7
l female Satsuki 9000 8
m female Satsuki 9000 8
n female Satsuki 8000 9
o male Satsuki 6000 5
p male Satsuki 8000 9
q male Satsuki 7000 6
r female Totoro 10000 10
s female Totoro 9000 10
t female Totoro 8000 8
u female Totoro 8000 7
v female Totoro 6000 7
w male Totoro 6000 8
x male Totoro 8000 10
y male Totoro 7000 7
z male Totoro 7000 7
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

480
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Define a linear model

model = lm(Steps ~ Sex + Teacher,
data = Data)

Bartlett’s test for homogeneity of variance

Bartlett’s test is known to be sensitive to non-normality in samples. That is, non-normal samples can
result in a significant test due to the non-normality.
x = residuals(model)

bartlett.test(x ~ interaction(Sex, Teacher),

data=Data)

Bartlett test of homogeneity of variances

Bartlett's K-squared = 3.7499, df = 5, p-value = 0.586

Levene’s test for homogeneity of variance

Levene’s test is an alternative to Bartlett’s that is supposedly less sensitive to departures from
normality in the data.
if(!require(car)){install.packages("car")}

library(car)

x = residuals(model)

leveneTest(x ~ Sex * Teacher

data=Data,
center=mean) ### Use the original Levene’s test

Levene's Test for Homogeneity of Variance (center = mean)

Df F value Pr(>F)
group 5 0.4457 0.8113

481
 INTRODUCTION TO PARAMETRIC TESTS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

20

Brown–Forsythe or robust Levene’s test

The Brown–Forsythe modification of Levene’s test makes it more robust to departures in normality of
the data.
if(!require(car)){install.packages("car")}

library(car)

x = residuals(model)

leveneTest(x ~ Sex * Teacher, data=Data)

Levene's Test for Homogeneity of Variance (center = median)

Df F value Pr(>F)
group 5 0.4015 0.842
20

if(!require(lawstat)){install.packages("lawstat")}

library(lawstat)

x = residuals(model)

levene.test(x, interaction(Data$Sex, Data$Teacher))

modified robust Brown-Forsythe Levene-type test based on the absolute deviations

from the median

Test Statistic = 0.4015, p-value = 0.842

Fligner-Killeen test
The Fligner-Killeen test is another test for homogeneity of variances that is robust to departures in
normality of the data.
x = residuals(model)

fligner.test(x ~ interaction(Sex, Teacher), data=Data)

Fligner-Killeen test of homogeneity of variances

Fligner-Killeen:med chi-squared = 2.685, df = 5, p-value = 0.7484

482
 ONE-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One-sample t-test
One-sample tests are not used too often, but are useful to compare a set of values to a given default
value. For example, one might ask if a set of student scores are significantly different from a “default”
or “neutral” score of 75.

Appropriate data
• One-sample data
• Data are interval/ratio, and are continuous
• Data are normally distributed
• Moderate skewness is permissible if the data distribution is unimodal without outliers

Hypotheses
• Null hypothesis: The mean is equal to the default value.
• Alternative hypothesis (two-sided): The mean is not equal to the default value.

Interpretation
Reporting significant results as, e.g., “Mean score for Variable A was significantly different from a
default value of 75” is acceptable.

Other notes and alternative tests

• The nonparametric analogue for this test is the one-sample Wilcoxon signed-rank test.
• Power analysis for the one-sample t-test can be found at Mangiafico (2015) in the References
section.

Packages used in this chapter

The packages used in this chapter include:

• psych
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-sample t-test example

In the following example, Brendon Small has his SNAP-Ed students keep diaries of what they eat for a
week, and then calculate the daily sodium intake in milligrams. As a first step in the analysis, he wants
to compare mean sodium intake by his students to the American Heart Association recommendation of
1500 mg.

A one-sample t-test can be conducted with the t.test function in the native stats package. Conveniently
the output includes the mean of the sample, a confidence interval for that mean, and a p-value for the t-
test.
483
 ONE-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

We will use a histogram with an imposed normal curve to confirm data are approximately normal.
Input = ("
Instructor Student Sodium
'Brendon Small' a 1200
'Brendon Small' b 1400
'Brendon Small' c 1350
'Brendon Small' d 950
'Brendon Small' e 1400
'Brendon Small' f 1150
'Brendon Small' g 1300
'Brendon Small' h 1325
'Brendon Small' i 1425
'Brendon Small' j 1500
'Brendon Small' k 1250
'Brendon Small' l 1150
'Brendon Small' m 950
'Brendon Small' n 1150
'Brendon Small' o 1600
'Brendon Small' p 1300
'Brendon Small' q 1050
'Brendon Small' r 1300
'Brendon Small' s 1700
'Brendon Small' t 1300
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Histogram of data
x = Data$Sodium

library(rcompanion)

484
ONE-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plotNormalHistogram(x)

Normal quantile plot of data

x = Data$Sodium

qqnorm(x)
qqline(x, col="red")

One-sample t-test
t.test(Data$Sodium,
mu = 1500,
conf.int = 0.95)

One Sample t-test

t = -4.9053, df = 19, p-value = 9.825e-05

485
 ONE-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

alternative hypothesis: true mean is not equal to 1500

95 percent confidence interval:

1196.83 1378.17

sample estimates:
mean of x
1287.5

Reference’s

“Student’s t–test for One Sample” in Mangiafico, S.S. 2015. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_01.html.

Exercises O

1. Considering Brendon Small’s data,

What was the mean sodium intake?

What is the 95% confidence interval for sodium intake?

Is the data distribution reasonably normal?

Was the mean sodium intake significantly different from the American Heart Association
recommendation or 1500 mg per day?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency. Dr.
Katz wants to test his students’ mean typing speed against a nominal speed of 40 words per minute.
Instructor Student Words.per.minute
'Dr. Katz Professional Therapist' a 35
'Dr. Katz Professional Therapist' b 50
'Dr. Katz Professional Therapist' c 55
'Dr. Katz Professional Therapist' d 60
'Dr. Katz Professional Therapist' e 65
'Dr. Katz Professional Therapist' f 60
'Dr. Katz Professional Therapist' g 70
'Dr. Katz Professional Therapist' h 55
'Dr. Katz Professional Therapist' i 45
'Dr. Katz Professional Therapist' j 55
'Dr. Katz Professional Therapist' k 60
'Dr. Katz Professional Therapist' l 45
'Dr. Katz Professional Therapist' m 65
'Dr. Katz Professional Therapist' n 55
'Dr. Katz Professional Therapist' o 50
'Dr. Katz Professional Therapist' p 60

486
 TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the mean typing speed?

What is the 95% confidence interval for typing speed?

Is the data distribution reasonably normal?

Was the mean typing speed significantly different from the nominal rate of 40 words per
minute?

Two-sample t-test
The two-sample unpaired t-test is a commonly used test that compares the means of two samples.

Appropriate data
• Two-sample data. That is, one measurement variable in two groups or samples
• Dependent variable is interval/ratio, and is continuous
• Independent variable is a factor with two levels. That is, two groups
• Data for each population are normally distributed
• For the traditional test, the two samples need to have the same variance. However, Welch’s t-
test, which is used by default in R, does not assume equal variances.
• Observations between groups are independent. That is, not paired or repeated measures
data
• Moderate skewness is permissible if the data distribution is unimodal without outliers

Hypotheses
• Null hypothesis: The means of the measurement variable for each sample are equal.
• Alternative hypothesis (two-sided): The means of the measurement variable for each sample
are not equal.

Interpretation
Reporting significant results as “Mean of variable Y for group A was different than that for group B.” is
acceptable.

Other notes and alternative tests

• The nonparametric analogue for this test is the two-sample Mann–Whitney U Test. Another
option is to use a permutation test. See Mangiafico (2015b) in the “References” section.
• Power analysis for the two-sample t-test can be found at Mangiafico (2015a) in the
“References” section.

487
 TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• lattice

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}

Two-sample t-test example

In the following example, Brendon Small and Coach McGuirk have their SNAP-Ed students keep diaries
of what they eat for a week, and then calculate the daily sodium intake in milligrams. Since the classes
have received different nutrition education programs, they want to see if the mean sodium intake is
the same for both classes.

A two-sample t-test can be conducted with the t.test function in the native stats package. The default is
to use Welch’s t-test, which doesn’t require equal variance between groups. Conveniently the output
includes the mean of each sample, a confidence interval for the difference in means, and a p-value for
the t-test.

We will use a histograms with imposed normal curves to confirm data are approximately normal.

Input = ("
Instructor Student Sodium
'Brendon Small' a 1200
'Brendon Small' b 1400
'Brendon Small' c 1350
'Brendon Small' d 950
'Brendon Small' e 1400
'Brendon Small' f 1150
'Brendon Small' g 1300
'Brendon Small' h 1325
'Brendon Small' i 1425
'Brendon Small' j 1500
'Brendon Small' k 1250
'Brendon Small' l 1150
'Brendon Small' m 950
'Brendon Small' n 1150
'Brendon Small' o 1600
'Brendon Small' p 1300
'Brendon Small' q 1050
'Brendon Small' r 1300
'Brendon Small' s 1700
'Brendon Small' t 1300
'Coach McGuirk' u 1100

488
TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Coach McGuirk' v 1200

'Coach McGuirk' w 1250
'Coach McGuirk' x 1050
'Coach McGuirk' y 1200
'Coach McGuirk' z 1250
'Coach McGuirk' aa 1350
'Coach McGuirk' ab 1350
'Coach McGuirk' ac 1325
'Coach McGuirk' ad 1525
'Coach McGuirk' ae 1225
'Coach McGuirk' af 1125
'Coach McGuirk' ag 1000
'Coach McGuirk' ah 1125
'Coach McGuirk' ai 1400
'Coach McGuirk' aj 1200
'Coach McGuirk' ak 1150
'Coach McGuirk' al 1400
'Coach McGuirk' am 1500
'Coach McGuirk' an 1200
")

Data = read.table(textConnection(Input),header=TRUE)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data by group

library(FSA)

Summarize(Sodium ~ Instructor,
data=Data,
digits=3)

Instructor n nvalid mean sd min Q1 median Q3 max percZero

1 Brendon Small 20 20 1287.50 193.734 950 1150 1300 1400 1700 0
2 Coach McGuirk 20 20 1246.25 142.412 1000 1144 1212 1350 1525 0

489
 TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histograms for data by group

Adding normal curves to histograms by groups requires some extra code.
library(lattice)

histogram(~ Sodium | Instructor,

data = Data,
type = "density",
layout = c(1,2), ### columns and rows of individual plots
panel=function(x, ...) {
panel.histogram(x, ...)

panel.mathdensity(dmath = dnorm,
col = "blue",
lwd = 2,
args = list(mean=mean(x),
sd=sd(x)), ...)})

Box plots for data by group

boxplot(Sodium ~ Instructor,
data = Data)

490
 TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Two-sample unpaired t-test

t.test(Sodium ~ Instructor,
data = Data,
conf.int = 0.95)

Welch Two Sample t-test

data: Sodium by Instructor

t = 0.76722, df = 34.893, p-value = 0.4481

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

-67.91132 150.41132

sample estimates:
mean in group Brandon Small mean in group Coach McGuirk
1287.50 1246.25

Optional readings
“Student's t–test for two samples” in McDonald, J.H. 2014. Handbook of Biological Statistics.
www.biostathandbook.com/twosamplettest.html.

References
“Student’s t–test for Two Samples” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_02.html.

“Mann–Whitney and Two-sample Permutation Test” in Mangiafico, S.S. 2015a. An R Companion for
the Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_02a.html.

Exercises P

1. Considering Brendon and McGuirk’s data,

491
 TWO-SAMPLE T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

What was the mean sodium intake for each class?

What is the 95% confidence interval for the difference in sodium intake?

Are the data distributions for each sample reasonably normal?

Was the mean sodium intake significantly different between classes?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency. Dr.
Katz and Laura want to compare their students’ mean typing speed between their classes.
Instructor Student Words.per.minute
'Dr. Katz Professional Therapist' a 35
'Dr. Katz Professional Therapist' b 50
'Dr. Katz Professional Therapist' c 55
'Dr. Katz Professional Therapist' d 60
'Dr. Katz Professional Therapist' e 65
'Dr. Katz Professional Therapist' f 60
'Dr. Katz Professional Therapist' g 70
'Dr. Katz Professional Therapist' h 55
'Dr. Katz Professional Therapist' i 45
'Dr. Katz Professional Therapist' j 55
'Dr. Katz Professional Therapist' k 60
'Dr. Katz Professional Therapist' l 45
'Dr. Katz Professional Therapist' m 65
'Dr. Katz Professional Therapist' n 55
'Dr. Katz Professional Therapist' o 50
'Dr. Katz Professional Therapist' p 60
'Laura the Receptionist' q 55
'Laura the Receptionist' r 60
'Laura the Receptionist' s 75
'Laura the Receptionist' t 65
'Laura the Receptionist' u 60
'Laura the Receptionist' v 70
'Laura the Receptionist' w 75
'Laura the Receptionist' x 70
'Laura the Receptionist' y 65
'Laura the Receptionist' z 72
'Laura the Receptionist' aa 73
'Laura the Receptionist' ab 65
'Laura the Receptionist' ac 80
'Laura the Receptionist' ad 50
'Laura the Receptionist' ae 55
'Laura the Receptionist' af 70

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the mean typing speed for each class?

Are the data distributions for each sample reasonably normal?

492
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Was the mean typing speed significantly different between the classes?

Paired t-test
The paired t-test is commonly used. It compares the means of two populations of paired observations
by testing if the difference between pairs is statistically different from zero or another number.

Appropriate data
• Two-sample data. That is, one measurement variable in two groups or samples
• Dependent variable is interval/ratio, and is continuous
• Independent variable is a factor with two levels. That is, two groups
• Data are paired. That is, the measurement for each observation in one group can be paired
logically or by subject to a measurement in the other group
• The distribution of the difference of paired measurements is normally distributed
• Moderate skewness is permissible if the data distribution is unimodal without outliers

Hypotheses
• Null hypothesis: The difference between paired observations is equal to zero.
• Alternative hypothesis (two-sided): The difference between paired observations is not equal
to zero.

Interpretation
Reporting significant results as “Mean of variable Y for group A was different than that for group B.” is
acceptable.

Other notes and alternative tests

• The nonparametric analogue for this test is the two-sample paired rank-sum test.
• Power analysis for the paired t-test can be found at Mangiafico (2015) in the “References”
section.

Packages used in this chapter

The packages used in this chapter include:

• psych
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(rcompanion)){install.packages("rcompanion")}

Paired t-test example

493
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In the following example, Dumbland Extension had adult students fill out a financial literacy
knowledge questionnaire both before and after completing a home financial management workshop.
Each student’s score before and after was paired by student.

Note in the following data that the students’ names are repeated, so that there is a before score for
student a and an after score for student a.

Since the data is in long form, we’ll order by Time, then Student to be sure the first observation for
Before is student a and the first observation for After is student a, and so on.

Input = ("
Time Student Score
Before a 65
Before b 75
Before c 86
Before d 69
Before e 60
Before f 81
Before g 88
Before h 53
Before i 75
Before j 73
After a 77
After b 98
After c 92
After d 77
After e 65
After f 77
After g 100
After h 73
After i 93
After j 75
")

Data = read.table(textConnection(Input),header=TRUE)

### Order data by Time and Student

Data = Data[order(Time, Student),]

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

494
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Remove unnecessary objects

rm(Input)

Histogram of difference data

A histogram with a normal curve imposed will be used to check if the paired differences between the
two populations is approximately normal in distribution.

First, two new variables, Before and After, are created by extracting the values of Score for
observations with the Time variable equal to Before or After, respectively.
Before = Data$Score[Data$Time=="Before"]

After = Data$Score[Data$Time=="After"]

Difference = After - Before

x = Difference

library(rcompanion)

plotNormalHistogram(x,
xlab="Difference (After - Before)")

Plot the paired data

Scatter plot with one-to-one line

Paired data can visualized with a scatter plot of the paired cases. In the plot below, points that fall
above and to the left of the blue line indicate cases for which the value for After was greater than for
Before.

Note that the points in the plot are jittered slightly so that points that would fall directly on top of one
another can be seen.

495
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

First, two new variables, Before and After, are created by extracting the values of Score for
observations with the Time variable equal to Before or After, respectively.

A variable Names is also created for point labels.

Before = Data$Score[Data$Time=="Before"]

After = Data$Score[Data$Time=="After"]

Names = Data$Student[Data$Time=="Before"]

plot(Before, jitter(After), # jitter offsets points so you can see them all
pch = 16, # shape of points
cex = 1.0, # size of points
xlim=c(50, 110), # limits of x-axis
ylim=c(50, 110), # limits of y-axis
xlab="Before", # label for x-axis
ylab="After" # label for y-axis
)

text(Before, After, labels=Names, # Label location and text

pos=3, cex=1.0) # Label text position and size

abline(0,1, col="blue", lwd=2) # line with intercept of 0 and slope of 1

Bar plot of differences

Paired data can also be visualized with a bar chart of differences. In the plot below, bars with a value
greater than zero indicate cases for which the value for After was greater than for Before.

New variables are first created for Before, After, and their Difference.

A variable Names is also created for bar labels.

496
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Before = Data$Score[Data$Time=="Before"]

After = Data$Score[Data$Time=="After"]

Difference = After – Before – 9

Names = Data$Student[Data$Time=="Before"]

barplot(Difference, # variable to plot

col="dark gray", # color of bars
xlab="Observation", # x-axis label
ylab="Difference (After – Before)", # y-axis label
names.arg=Names # labels for bars
)

Paired t-test

t.test(Score ~ Time,
data=Data,
paired = TRUE,
conf.level = 0.95)

Paired t-test

t = 3.8084, df = 9, p-value = 0.004163

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

4.141247 16.258753

sample estimates:
mean of the differences
10.2

497
 PAIRED T-TEST SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional readings

“Paired t–test” in McDonald, J.H. 2014. Handbook of Biological Statistics.

www.biostathandbook.com/pairedttest.html.

References

“Paired t–test” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological Statistics,
version 1.09. rcompanion.org/rcompanion/d_09.html.

Exercises Q

1. Considering the Dumbland Extension data,

What was the mean difference in score before and after the training?

Was this an increase or a decrease?

What is the 95% confidence interval for this difference?

Is the data distribution for the paired differences reasonably normal?

Was the mean score significantly different before and after the training?

2. Residential properties in Dougal County rarely need phosphorus for good turfgrass growth. As part
of an extension education program, Early and Rusty Cuyler asked homeowners to report their
phosphorus fertilizer use, in pounds of P2O5 per acre, before the program and then one year later.
Date Homeowner P2O5
'2014-01-01' a 0.81
'2014-01-01' b 0.86
'2014-01-01' c 0.79
'2014-01-01' d 0.59
'2014-01-01' e 0.71
'2014-01-01' f 0.88
'2014-01-01' g 0.63
'2014-01-01' h 0.72
'2014-01-01' i 0.76
'2014-01-01' j 0.58
'2015-01-01' a 0.67
'2015-01-01' b 0.83
'2015-01-01' c 0.81
'2015-01-01' d 0.50
'2015-01-01' e 0.71
'2015-01-01' f 0.72
'2015-01-01' g 0.67

498
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'2015-01-01' h 0.67
'2015-01-01' i 0.48
'2015-01-01' j 0.68

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the mean difference in P2O5 before and after the training?

Is this an increase or a decrease?

What is the 95% confidence interval for this difference?

Is the data distribution for the paired differences reasonably normal?

Was the mean P2O5 use significantly different before and after the training?

One-way ANOVA
A one-way analysis of variance (ANOVA) is similar to a t-test, except that it is capable of comparing
more than two groups.

We will conduct the ANOVA by constructing a general linear model with the lm function in the native
stats package. The general linear model is the basis for more advanced parametric models that can
include multiple independent variables that can be continuous or factor variables.

Appropriate data
• One-way data. That is, one measurement variable in two or more groups
• Dependent variable is interval/ratio, and is continuous
• Independent variable is a factor with two or more levels. That is, two or more groups
• In the general linear model approach, residuals are normally distributed
• In the general linear model approach, groups have the same variance. That is,
homoscedasticity
• Observations among groups are independent. That is, not paired or repeated measures data
• Moderate deviation from normally-distributed residuals is permissible

Hypotheses
• Null hypothesis: The means of the measurement variable for each group are equal.
• Alternative hypothesis (two-sided): The means of the measurement variable for among
groups are not equal.

499
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interpretation
• Reporting significant results for the omnibus test as “Significant differences were found
among means for groups.” is acceptable. Alternatively, “A significant effect for Independent
Variable on Dependent Variable was found.”
• Reporting significant results for mean separation post-hoc tests as “Mean of variable Y for
group A was different than that for group B.” is acceptable.

Other notes and alternative tests

• The nonparametric analogue for this test is the Kruskal–Wallis test. Another nonparametric
approach is to use a permutation test. See Mangiafico (2015b) in the “References” section.
• Power analysis for the two-sample t-test can be found at Mangiafico (2015a) in the
“References” section.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• Rmisc
• ggplot2
• car
• multcompView
• lsmeans
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(Rmisc)){install.packages("Rmisc")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-way ANOVA example

In the following example, Brendon Small, Coach McGuirk, and Melissa Robbins have their SNAP-Ed
students keep diaries of what they eat for a week, and then calculate the daily sodium intake in
milligrams. Since the classes have received different nutrition education programs, they want to see if
the mean sodium intake is the same among classes.

The analysis will be conducted by constructing a general linear model with the lm function in the
native stats package, and conducting the analysis of variance with the Anova function in the car
package.

500
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plots will be used to check model assumptions for normality of residuals and homoscedasticity.

Finally, if the effect of Instructor is significant, a mean comparison test will be conducted to determine
which means differ from which others.
Input = ("
Instructor Student Sodium
'Brendon Small' a 1200
'Brendon Small' b 1400
'Brendon Small' c 1350
'Brendon Small' d 950
'Brendon Small' e 1400
'Brendon Small' f 1150
'Brendon Small' g 1300
'Brendon Small' h 1325
'Brendon Small' i 1425
'Brendon Small' j 1500
'Brendon Small' k 1250
'Brendon Small' l 1150
'Brendon Small' m 950
'Brendon Small' n 1150
'Brendon Small' o 1600
'Brendon Small' p 1300
'Brendon Small' q 1050
'Brendon Small' r 1300
'Brendon Small' s 1700
'Brendon Small' t 1300
'Coach McGuirk' u 1100
'Coach McGuirk' v 1200
'Coach McGuirk' w 1250
'Coach McGuirk' x 1050
'Coach McGuirk' y 1200
'Coach McGuirk' z 1250
'Coach McGuirk' aa 1350
'Coach McGuirk' ab 1350
'Coach McGuirk' ac 1325
'Coach McGuirk' ad 1525
'Coach McGuirk' ae 1225
'Coach McGuirk' af 1125
'Coach McGuirk' ag 1000
'Coach McGuirk' ah 1125
'Coach McGuirk' ai 1400
'Coach McGuirk' aj 1200
'Coach McGuirk' ak 1150
'Coach McGuirk' al 1400
'Coach McGuirk' am 1500
'Coach McGuirk' an 1200
'Melissa Robins' ao 900
'Melissa Robins' ap 1100
'Melissa Robins' aq 1150
'Melissa Robins' ar 950
'Melissa Robins' as 1100
'Melissa Robins' at 1150
'Melissa Robins' au 1250

501
ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Melissa Robins' av 1250

'Melissa Robins' aw 1225
'Melissa Robins' ax 1325
'Melissa Robins' ay 1125
'Melissa Robins' az 1025
'Melissa Robins' ba 950
'Melissa Robins' bc 925
'Melissa Robins' bd 1200
'Melissa Robins' be 1100
'Melissa Robins' bf 950
'Melissa Robins' bg 1300
'Melissa Robins' bh 1400
'Melissa Robins' bi 1100
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Summarize data by group

library(FSA)

Summarize(Sodium ~ Instructor,
data=Data,
digits=3)

Instructor n nvalid mean sd min Q1 median Q3 max percZero

1 Brendon Small 20 20 1287.50 193.734 950 1150 1300 1400 1700 0
2 Coach McGuirk 20 20 1246.25 142.412 1000 1144 1212 1350 1525 0
3 Melissa Robins 20 20 1123.75 143.149 900 1006 1112 1231 1400 0

502
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Box plots for data by group

boxplot(Sodium ~ Instructor,
data = Data)

Plot of means and confidence intervals

For this plot, a data frame called Sum will be created with the mean sodium intake and confidence
interval for this mean for each instructor.
library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Sodium",
group = "Instructor",
conf = 0.95,
digits = 3,
traditional = FALSE,
percentile = TRUE)

Sum

Instructor n Mean Conf.level Percentile.lower Percentile.upper

1 Brendon Small 20 1290 0.95 1200 1370
2 Coach McGuirk 20 1250 0.95 1190 1310
3 Melissa Robins 20 1120 0.95 1060 1180

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Instructor,
y = Mean)) +
geom_errorbar(aes(ymin = Percentile.lower,
ymax = Percentile.upper),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Mean sodium, mg")

503
ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Define linear model

model = lm(Sodium ~ Instructor,
data = Data)

summary(model) ### Will show overall p-value and r-square

Multiple R-squared: 0.1632, Adjusted R-squared: 0.1338

F-statistic: 5.558 on 2 and 57 DF, p-value: 0.006235

Conduct analysis of variance

library(car)

Anova(model,
type = "II") ### Type II sum of squares

Anova Table (Type II tests)

Response: Sodium

Sum Sq Df F value Pr(>F)

Instructor 290146 2 5.5579 0.006235 **
Residuals 1487812 57

504
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histogram of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model),
residuals(model))

plot(model) ### Additional model checking plots

Post-hoc analysis: mean separation tests

For an explanation of using least square means for multiple comparisons, review the following
chapters:
• What are Least Square Means?
• Least Square Means for Multiple Comparisons

505
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The code defines an object leastsquare which will hold the output of the lsmeans function. The lsmeans
function here calls for all pairwise comparisons for the independent variable Instructor in the
previously defined model object.

There are two ways to view the lsmeans output. Calling the leastsquare object to be printed produces
two sections of output: $lsmeans, which shows estimates for the LS means along with their standard
errors and confidence intervals; and $contrasts, which indicate the pairwise comparisons with p-
values for the compared LS means being different.

Using the cld function produces a compact letter display. LS means sharing a letter are not
significantly different from one another at the alpha level indicated. A Tukey adjustment is made for
multiple comparisons with the adjust="tukey" option. And Letters indicates the symbols to use for
groups.

To adjust the confidence intervals for the LS means, use e.g. summary(leastsquare, level=0.99).

Note that because Instructor is the only independent variable in the model, the LS means are equal to
the arithmetic means produced by the Summarize function in this case.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust = "tukey")

leastsquare

$lsmeans
Instructor lsmean SE df lower.CL upper.CL
Brendon Small 1287.50 36.12615 57 1215.159 1359.841
Coach McGuirk 1246.25 36.12615 57 1173.909 1318.591
Melissa Robins 1123.75 36.12615 57 1051.409 1196.091

Confidence level used: 0.95

$contrasts
contrast estimate SE df t.ratio p.value
Brendon Small - Coach McGuirk 41.25 51.09009 57 0.807 0.7000
Brendon Small - Melissa Robins 163.75 51.09009 57 3.205 0.0062
Coach McGuirk - Melissa Robins 122.50 51.09009 57 2.398 0.0510

P value adjustment: tukey method for comparing a family of 3 estimates

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

506
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instructor lsmean SE df lower.CL upper.CL .group

Melissa Robins 1123.75 36.12615 57 1034.882 1212.618 a
Coach McGuirk 1246.25 36.12615 57 1157.382 1335.118 ab
Brendon Small 1287.50 36.12615 57 1198.632 1376.368 b

Confidence level used: 0.95

P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

References

“One-way Anova” in Mangiafico, S.S. 2015a. An R Companion for the Handbook of Biological Statistics,
version 1.09. rcompanion.org/rcompanion/d_05.html.

“One-way Analysis with Permutation Test” in Mangiafico, S.S. 2015b. An R Companion for the
Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_06a.html.

Exercises R

1. Considering Brendon, Melissa, and McGuirk’s data,

What was the mean sodium intake for each instructor?

Does Instructor have a significant effect on sodium intake?

Are the residuals reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean sodium intake from which
others?

Does this result correspond to what you would have thought from looking at the plot of the
means and standard errors or the box plots?

How does the mean separation change if you change the alpha level to 0.06?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency. Dr.
Katz, Laura, and Ben want to compare their students’ mean typing speed between their classes.
Instructor Student Words.per.minute
'Dr. Katz Professional Therapist' a 35
'Dr. Katz Professional Therapist' b 50
'Dr. Katz Professional Therapist' c 55
'Dr. Katz Professional Therapist' d 60
'Dr. Katz Professional Therapist' e 65
'Dr. Katz Professional Therapist' f 60
'Dr. Katz Professional Therapist' g 70
'Dr. Katz Professional Therapist' h 55
'Dr. Katz Professional Therapist' i 45

507
 ONE-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Dr. Katz Professional Therapist' j 55

'Dr. Katz Professional Therapist' k 60
'Dr. Katz Professional Therapist' l 45
'Dr. Katz Professional Therapist' m 65
'Dr. Katz Professional Therapist' n 55
'Dr. Katz Professional Therapist' o 50
'Dr. Katz Professional Therapist' p 60
'Laura the Receptionist' q 55
'Laura the Receptionist' r 60
'Laura the Receptionist' s 75
'Laura the Receptionist' t 65
'Laura the Receptionist' u 60
'Laura the Receptionist' v 70
'Laura the Receptionist' w 75
'Laura the Receptionist' x 70
'Laura the Receptionist' y 65
'Laura the Receptionist' z 72
'Laura the Receptionist' aa 73
'Laura the Receptionist' ab 65
'Laura the Receptionist' ac 80
'Laura the Receptionist' ad 50
'Laura the Receptionist' ae 60
'Laura the Receptionist' af 70
'Ben Katz' ag 55
'Ben Katz' ah 55
'Ben Katz' ai 70
'Ben Katz' aj 55
'Ben Katz' ak 65
'Ben Katz' al 60
'Ben Katz' am 70
'Ben Katz' an 60
'Ben Katz' ao 60
'Ben Katz' ap 62
'Ben Katz' aq 63
'Ben Katz' ar 65
'Ben Katz' as 75
'Ben Katz' at 50
'Ben Katz' au 50
'Ben Katz' av 65

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the mean typing speed for each instructor?

Does Instructor have a significant effect on typing speed?

Are the residuals reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean typing speed from which
others?

508
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Does this result correspond to what you would have thought from looking at the plot of the
means and standard errors or box plots?

One-way ANOVA with Blocks

Blocks are used in an analysis of variance or similar models in order to account for suspected variation
from factors other than the treatments or main independent variables being investigated.

Traditionally, in agricultural experiments, plots would be arranged into blocks according to factors in
the field that could not be controlled. For example, if one side of the field were close to the tree line,
those plots might get less sun, or be more protected from wind. So the experimenter would consider
those plots near the tree line as one block, and another set in a more open spot another block. In this
way, all kinds of variation in the soils and locations could be accounted for somewhat. The experiment
might be designed in a randomized complete block design in which each block had a plot with each
treatment.

When using blocks, the experimenter isn’t concerned necessarily with the effect of the blocks or even
the factors behind assigning those blocks. That is, the east end of the field might be by the tree line
and may have soils with slightly poorer drainage. If the experimenter were testing the effects of
different fertilizers, she isn’t going to try to report the effect of “next to the tree line” or “in poorly
drained soil,” but she does want to account for these effects statistically.

Blocks in extension education for measurements on students might include school or class or grade.
There might be differences according to these factors, but we might not care to investigate if one
particular school has a different result than another particular one. Still, we want to take to these
differences into account statistically.

Appropriate data
• One-way data, with blocks. That is, one measurement variable in two or more groups, where
each group is also distributed among at least two blocks
• Dependent variable is interval/ratio, and is continuous
• Independent variable is a factor with two or more levels. That is, two or more groups
• A second independent variable is a blocking factor variable with two or more levels
• In the general linear model approach, residuals are normally distributed
• In the general linear model approach, groups have the same variance. That is,
homoscedasticity
• Observations among groups are independent. That is, not paired or repeated measures data,
except that observations are within blocks
• Moderate deviation from normally-distributed residuals is permissible

Hypotheses
• Null hypothesis: The means of the measurement variable for each group are equal

509
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• Alternative hypothesis (two-sided): The means of the measurement variable for among
groups are not equal
• An additional null hypothesis is tested for the effect of blocks: The means of the measurement
variable for each block are equal

Interpretation
• Reporting significant results for the omnibus test as “Significant differences were found
among means for groups.” is acceptable. Alternatively, “A significant effect for independent
variable on dependent variable was found.”
• Reporting significant results for mean separation post-hoc tests as “Mean of variable Y for
group A was different than that for group B.” is acceptable.

Other notes and alternative tests

• Only for unreplicated complete block designs, the nonparametric analogues for this test are
the Friedman and Quade tests.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• ggplot2
• car
• multcompView
• lsmeans
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-way ANOVA with blocks example

This example will revisit the sodium intake data set with Brendon Small and the other instructors.
This time, though, they have recorded the town each student is from, and they would like to use this as
a blocking variable. They suspect that the town of residence may have some effect on sodium intake
since each town has varying income, ethnic makeup, and other demographic factors.

The instructors are focused on the effect of their different nutrition education programs. They are not
concerned about sodium intake in one specific town or another per se, but they want to take account
this effect into account statistically.

510
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note that when the blocking variable is added to the model, the calculated p-value for the effect of
Instructor is different than it was for the case of the one-way ANOVA without the blocking variable.
Input = ("
Instructor Town Sodium
'Brendon Small' Squiggleville 1200
'Brendon Small' Squiggleville 1400
'Brendon Small' Squiggleville 1350
'Brendon Small' Metalocalypse 950
'Brendon Small' Squiggleville 1400
'Brendon Small' Squiggleville 1150
'Brendon Small' Squiggleville 1300
'Brendon Small' Metalocalypse 1325
'Brendon Small' Metalocalypse 1425
'Brendon Small' Squiggleville 1500
'Brendon Small' Squiggleville 1250
'Brendon Small' Metalocalypse 1150
'Brendon Small' Metalocalypse 950
'Brendon Small' Squiggleville 1150
'Brendon Small' Metalocalypse 1600
'Brendon Small' Metalocalypse 1300
'Brendon Small' Metalocalypse 1050
'Brendon Small' Metalocalypse 1300
'Brendon Small' Squiggleville 1700
'Brendon Small' Squiggleville 1300
'Coach McGuirk' Squiggleville 1100
'Coach McGuirk' Squiggleville 1200
'Coach McGuirk' Squiggleville 1250
'Coach McGuirk' Metalocalypse 1050
'Coach McGuirk' Metalocalypse 1200
'Coach McGuirk' Metalocalypse 1250
'Coach McGuirk' Squiggleville 1350
'Coach McGuirk' Squiggleville 1350
'Coach McGuirk' Squiggleville 1325
'Coach McGuirk' Squiggleville 1525
'Coach McGuirk' Squiggleville 1225
'Coach McGuirk' Squiggleville 1125
'Coach McGuirk' Metalocalypse 1000
'Coach McGuirk' Metalocalypse 1125
'Coach McGuirk' Squiggleville 1400
'Coach McGuirk' Metalocalypse 1200
'Coach McGuirk' Squiggleville 1150
'Coach McGuirk' Squiggleville 1400
'Coach McGuirk' Squiggleville 1500
'Coach McGuirk' Squiggleville 1200
'Melissa Robins' Metalocalypse 900
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 1150
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 1150
'Melissa Robins' Squiggleville 1250
'Melissa Robins' Squiggleville 1250

511
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Melissa Robins' Squiggleville 1225

'Melissa Robins' Squiggleville 1325
'Melissa Robins' Metalocalypse 1125
'Melissa Robins' Metalocalypse 1025
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 925
'Melissa Robins' Squiggleville 1200
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 1300
'Melissa Robins' Squiggleville 1400
'Melissa Robins' Metalocalypse 1100
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

Data$Town = factor(Data$Town,
levels=unique(Data$Town))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Use Summarize to check if cell sizes are balanced

We can use the n or nvalid columns in the Summarize output to check if cell sizes are balanced. That is,
if each treatment and block combination have the same number of observations.
library(FSA)

Summarize(Sodium ~ Instructor + Town,

data=Data,
digits=3)

512
ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Define linear model

model = lm(Sodium ~ Instructor + Town,
data = Data)

summary(model) ### Will show overall p-value and r-squared

Multiple R-squared: 0.3485, Adjusted R-squared: 0.3136

F-statistic: 9.987 on 3 and 56 DF, p-value: 2.265e-05

Conduct analysis of variance

library(car)

Anova(model,
type = "II") ### Type II sum of squares

Anova Table (Type II tests)

Response: Sodium

Sum Sq Df F value Pr(>F)

Instructor 148944 2 3.6006 0.0338033 *
Town 329551 1 15.9332 0.0001928 ***
Residuals 1158261 56

Histogram and plot of residuals

x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

513
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plot(fitted(model),
residuals(model))

plot(model) ### Additional model checking plots

Post-hoc analysis: mean separation tests

For an explanation of using least square means for multiple comparisons, see the section “Post-hoc
analysis: mean separation tests” in the One-way ANOVA chapter.

In this example, you’ll note that the LS means are different from the arithmetic means calculated for
Instructor in the last chapter. This is a result of the fact that instructors have different numbers of
students from each town. That is, the design is unbalanced.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust = "tukey")

leastsquare

$lsmeans
Instructor lsmean SE df lower.CL upper.CL
Brendon Small 1279.644 32.21856 56 1215.103 1344.186
Coach McGuirk 1214.827 33.10792 56 1148.504 1281.150
Melissa Robins 1155.173 33.10792 56 1088.850 1221.496

Results are averaged over the levels of: Town

Confidence level used: 0.95

$contrasts
contrast estimate SE df t.ratio p.value
Brendon Small - Coach McGuirk 64.81742 45.86048 56 1.413 0.3410
Brendon Small - Melissa Robins 124.47097 46.53123 56 2.675 0.0261
Coach McGuirk - Melissa Robins 59.65356 48.12705 56 1.240 0.4352

514
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Results are averaged over the levels of: Town

P value adjustment: tukey method for comparing a family of 3 estimates

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lower-case letters for .group
adjust = "tukey") ### Tukey-adjusted p-values

Instructor lsmean SE df lower.CL upper.CL .group

Melissa Robins 1155.173 33.10792 56 1088.850 1221.496 a
Coach McGuirk 1214.827 33.10792 56 1148.504 1281.150 ab
Brendon Small 1279.644 32.21856 56 1215.103 1344.186 b

Results are averaged over the levels of: Town

Confidence level used: 0.95
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

Plot of LS means and confidence intervals

Extracting the LS means from the output of the lsmeans function takes a few extra steps. We’ll create a
data frame called Sum, and pass that data frame to ggplot to plot the LS means and confidence
intervals.

When a design is unbalanced, it is usually desirable to present LS means. For further discussion on
this, see the chapter What are Least Square Means?
### Extract variables from leastsquare and create new data frame called Sum

Instructor = summary(leastsquare$lsmeans)["Instructor"]
lsmean = summary(leastsquare$lsmeans)["lsmean"]
lower.CL = summary(leastsquare$lsmeans)["lower.CL"]
upper.CL = summary(leastsquare$lsmeans)["upper.CL"]

Sum = data.frame(Instructor, lsmean, lower.CL, upper.CL)

Sum

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Instructor,
y = lsmean)) +
geom_errorbar(aes(ymin = lower.CL,
ymax = upper.CL),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("LS mean sodium, mg")

515
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Least square mean daily sodium intake for students in each of three classes
following different nutrition education programs. Error bars represent confidence
intervals of the least square mean.

Exercises S

1. Considering Brendon, Melissa, and McGuirk’s data with means adjusted for students’ towns,

What was the LS mean sodium intake for students for each instructor?

Does Instructor have a significant effect on sodium intake?

Does Town have a significant effect on sodium intake?

Are the residuals reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean sodium intake from which
others?

Does this result correspond to what you would have thought from looking at the plot of the LS
means and confidence intervals?

Is this design balanced or unbalanced?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency. Dr.
Katz, and Laura, and Ben want to compare their students’ mean typing speed between their classes.
They have also recorded the year or grade of each student. They are not interested in the effect of Year
per se, but they want to account for the effect statistically.

516
 ONE-WAY ANOVA WITH BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instructor Year Words.per.minute

'Dr. Katz Professional Therapist' 7 35
'Dr. Katz Professional Therapist' 7 50
'Dr. Katz Professional Therapist' 7 55
'Dr. Katz Professional Therapist' 7 60
'Dr. Katz Professional Therapist' 8 65
'Dr. Katz Professional Therapist' 8 60
'Dr. Katz Professional Therapist' 8 70
'Dr. Katz Professional Therapist' 8 55
'Dr. Katz Professional Therapist' 9 45
'Dr. Katz Professional Therapist' 9 55
'Dr. Katz Professional Therapist' 9 60
'Dr. Katz Professional Therapist' 9 45
'Dr. Katz Professional Therapist' 10 65
'Dr. Katz Professional Therapist' 10 55
'Dr. Katz Professional Therapist' 11 50
'Dr. Katz Professional Therapist' 12 60
'Laura the Receptionist' 7 55
'Laura the Receptionist' 7 60
'Laura the Receptionist' 7 75
'Laura the Receptionist' 7 65
'Laura the Receptionist' 8 60
'Laura the Receptionist' 8 70
'Laura the Receptionist' 8 75
'Laura the Receptionist' 8 70
'Laura the Receptionist' 9 65
'Laura the Receptionist' 9 72
'Laura the Receptionist' 9 73
'Laura the Receptionist' 9 65
'Laura the Receptionist' 10 80
'Laura the Receptionist' 10 50
'Laura the Receptionist' 10 60
'Laura the Receptionist' 10 70
'Ben Katz' 7 55
'Ben Katz' 7 55
'Ben Katz' 7 70
'Ben Katz' 7 55
'Ben Katz' 8 65
'Ben Katz' 8 60
'Ben Katz' 8 70
'Ben Katz' 8 60
'Ben Katz' 9 60
'Ben Katz' 9 62
'Ben Katz' 9 63
'Ben Katz' 9 65
'Ben Katz' 10 75
'Ben Katz' 10 50
'Ben Katz' 10 50
'Ben Katz' 10 65

Note that the following code is necessary for R to recognize Year as a factor variable.
### Convert Year from an integer variable to a factor variable

517
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data$Year = factor(Data$Year,
levels=c("7", "8", "9", "10"))

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the LS mean typing speed for each instructor’s class?

Does Instructor have a significant effect on typing speed?

Does Year have a significant effect on typing speed?

Are the residuals reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean typing speed from which
others?

Does this result correspond to what you would have thought from looking at the plot of the LS
means and standard errors?

Is this design balanced or unbalanced?

One-way ANOVA with Random Blocks

This chapter reproduces the example from the previous chapter. The reader should consult that
chapter for an explanation of one-way analysis of variance with blocks. Here, the analysis is done with
a mixed effects model, with the treatments treated as a fixed effect and the blocks treated as a random
effect.

In analysis of variance, blocking variables are often treated as random variables. This is because the
blocking variable represents a random selection of levels of that variable. The analyst wants to take
the effects of the blocking variable into account, but the identity of the specific levels of the blocks are
not of interest.

In the example in this chapter, the instructors are focused on the effect of their different nutrition
education programs, which are the treatments; they are not concerned about the effect of one specific
town or another per se, but do want to want to take into account any differences due to the different
towns.

For a more complete discussion of random effects, see the Using Random Effects in Models chapter.

518
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Packages used in this chapter

The packages used in this chapter include:

• psych
• lme4
• lmerTest
• multcompView
• lsmeans
• nlme
• car
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(lme4)){install.packages("lme4")}
if(!require(lmerTest)){install.packages("lmerTest")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(nlme)){install.packages("nlme")}
if(!require(car)){install.packages("car")}
if(!require(rcompanion)){install.packages("rcompanion")}

One-way ANOVA with random blocks example

Input = ("
Instructor Town Sodium
'Brendon Small' Squiggleville 1200
'Brendon Small' Squiggleville 1400
'Brendon Small' Squiggleville 1350
'Brendon Small' Metalocalypse 950
'Brendon Small' Squiggleville 1400
'Brendon Small' Squiggleville 1150
'Brendon Small' Squiggleville 1300
'Brendon Small' Metalocalypse 1325
'Brendon Small' Metalocalypse 1425
'Brendon Small' Squiggleville 1500
'Brendon Small' Squiggleville 1250
'Brendon Small' Metalocalypse 1150
'Brendon Small' Metalocalypse 950
'Brendon Small' Squiggleville 1150
'Brendon Small' Metalocalypse 1600
'Brendon Small' Metalocalypse 1300
'Brendon Small' Metalocalypse 1050
'Brendon Small' Metalocalypse 1300
'Brendon Small' Squiggleville 1700
'Brendon Small' Squiggleville 1300
'Coach McGuirk' Squiggleville 1100
'Coach McGuirk' Squiggleville 1200
'Coach McGuirk' Squiggleville 1250
'Coach McGuirk' Metalocalypse 1050

519
ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Coach McGuirk' Metalocalypse 1200

'Coach McGuirk' Metalocalypse 1250
'Coach McGuirk' Squiggleville 1350
'Coach McGuirk' Squiggleville 1350
'Coach McGuirk' Squiggleville 1325
'Coach McGuirk' Squiggleville 1525
'Coach McGuirk' Squiggleville 1225
'Coach McGuirk' Squiggleville 1125
'Coach McGuirk' Metalocalypse 1000
'Coach McGuirk' Metalocalypse 1125
'Coach McGuirk' Squiggleville 1400
'Coach McGuirk' Metalocalypse 1200
'Coach McGuirk' Squiggleville 1150
'Coach McGuirk' Squiggleville 1400
'Coach McGuirk' Squiggleville 1500
'Coach McGuirk' Squiggleville 1200
'Melissa Robins' Metalocalypse 900
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 1150
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 1150
'Melissa Robins' Squiggleville 1250
'Melissa Robins' Squiggleville 1250
'Melissa Robins' Squiggleville 1225
'Melissa Robins' Squiggleville 1325
'Melissa Robins' Metalocalypse 1125
'Melissa Robins' Metalocalypse 1025
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 925
'Melissa Robins' Squiggleville 1200
'Melissa Robins' Metalocalypse 1100
'Melissa Robins' Metalocalypse 950
'Melissa Robins' Metalocalypse 1300
'Melissa Robins' Squiggleville 1400
'Melissa Robins' Metalocalypse 1100
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

Data$Town = factor(Data$Town,
levels=unique(Data$Town))

### Check the data frame

library(psych)

headTail(Data)

520
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Mixed-effects model with lmer

In this first example, the model will be specified with the lmer function in the package lme4. The term
(1|Town) in the model formula indicates that Town should be treated as a random variable, with each
level having its own intercept in the model. The anova function in the package lmerTest is used to
produce p-values for the fixed effects. The rand function in the package lmerTest produces p-values for
the random effects.

Technical note
lmerTest::anova by default returns p-values for Type III sum of squares with a Satterthwaite
approximation for the degrees of freedom.
library(lme4)

library(lmerTest)

model = lmer(Sodium ~ Instructor + (1|Town),

data=Data,
REML=TRUE)

anova(model)

Analysis of Variance Table of type III with Satterthwaite

approximation for degrees of freedom

Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)

Instructor 154766 77383 2 56.29 3.7413 0.02982 *

rand(model)

Analysis of Random effects Table:

Chi.sq Chi.DF p.value

Town 10.5 1 0.001 **

p-value and pseudo R-squared for model

To use the nagelkerke function with an lmer object, the null model with which to compare the lmer
model has to be specified explicitly.

521
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One approach is to define the null model as one with no fixed effects except for an intercept, indicated
with a 1 on the right side of the ~. And to also include the random effects, in this case (1|Town).
model.null = lmer(Sodium ~ 1 + (1|Town),
data = Data,
REML = TRUE)

anova(model,
model.null)

Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)

..1 3 782.34 788.63 -388.17 776.34
object 5 778.82 789.29 -384.41 768.82 7.5255 2 0.02322 *

library(rcompanion)

nagelkerke(fit = model,
null = model.null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.00971559
Cox and Snell (ML) 0.11818400
Nagelkerke (Cragg and Uhler) 0.11818400

Another approach to determining the p-value and pseudo R-squared for an lmer model is to compare
the model to a null model with only an intercept and neither the fixed nor the random effects. To
accomplish this, the null model has to be specified with lm and not lmer. However, it may not be
permissible to compare models specified with different functions in this way. The anova function in
lmerTest does allow these comparisons. The nagelkerke function will also allow these comparisons,
although I do not know if the results are valid.
model.null.2 = lm(Sodium ~ 1,
data = Data)

anova(model,
model.null.2)

Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)

..1 2 792.07 796.26 -394.04 788.07
object 5 778.82 789.29 -384.41 768.82 19.253 3 0.0002424 ***

library(rcompanion)

nagelkerke(fit = model,
null = model.null.2)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.0239772

522
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cox and Snell (ML) 0.2701590

Nagelkerke (Cragg and Uhler) 0.2701600

Post-hoc analysis
The lsmeans package is able to handle lmer objects. For further details, see ?lsmeans::models. For a
review of mean separation tests and least square means, see the chapters What are Least Square
Means?; the chapter Least Square Means for Multiple Comparisons; and the “Post-hoc analysis: mean
separation tests” section in the One-way ANOVA chapter.

The lmerTest package has a function Difflsmeans which also compares leastsquare means for
treatments, but doesn’t include output for a compact letter display or adjusted p-values.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust="tukey") ### Tukey-adjusted comparisons

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey-adjusted comparisons

Instructor lsmean SE df lower.CL upper.CL .group

Melissa Robins 1153.201 83.01880 1.24 476.7519 1829.650 a
Coach McGuirk 1216.799 83.01880 1.24 540.3498 1893.248 ab
Brendon Small 1280.137 82.60038 1.22 588.9293 1971.345 b

Confidence level used: 0.95

P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

leastsquare

$contrasts
contrast estimate SE df t.ratio p.value
Brendon Small - Coach McGuirk 63.33828 45.93368 56.11 1.379 0.3587
Brendon Small - Melissa Robins 126.93620 46.73138 56.29 2.716 0.0234
Coach McGuirk - Melissa Robins 63.59792 48.62097 56.62 1.308 0.3967

P value adjustment: tukey method for comparing a family of 3 estimate

A similar analysis can be conducted with the difflsmeans function in the lmerTest package.
library(lmerTest)

difflsmeans(model,
test.effs="Instructor")

523
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Differences of LSMEANS:
Estimate Standard Error DF t-value Lower CI Upper CI p-value
Instructor Brendon Small - Coach McGuirk 63.3 45.8 56.1 1.38 -28.5 155 0.172
Instructor Brendon Small - Melissa Robins 126.9 46.5 56.3 2.73 33.9 220 0.008 **
Instructor Coach McGuirk - Melissa Robins 63.6 48.0 56.6 1.33 -32.5 160 0.190

The following code extracts a data frame from the difflsmeans output, and then adds a column of p-
values adjusted by the fdr method. See ?p.adjust for other adjustment options.
comparison = difflsmeans(model,
test.effs="Instructor")[[1]]

p.value = comparison$"p-value"

comparison$p.adj = p.adjust(p.value,
method = "fdr")

comparison
Estimate Standard Error DF t-value Lower CI Upper CI p-value p.adj
Instructor Brendon Small - Coach McGuirk 63.3383 45.8366 56.1 1.38 -28.4807 155.1573 0.1725 0.1902
Instructor Brendon Small - Melissa Robins 126.9362 46.4659 56.3 2.73 33.8631 220.0093 0.0084 0.0252
Instructor Coach McGuirk - Melissa Robins 63.5979 47.9651 56.6 1.33 -32.4661 159.6619 0.1902 0.1902

Mixed-effects model with nlme

Mixed models can also be specified with the nlme package. The formula notation for this package is
slightly different than for the lme4 package. The expression random=~1|Town indicates that Town
should be treated as a random variable, with each level having its own intercept in the model.

The fixed effects in the model can be tested with the Anova function in the car package.
library(nlme)

model = lme(Sodium ~ Instructor, random=~1|Town,

data=Data,
method="REML")

library(car)

Anova(model)

Analysis of Deviance Table (Type II tests)

Chisq Df Pr(>Chisq)
Instructor 7.4827 2 0.02372 *

The random effects in the model can be tested by specifying a null model with only fixed effects and
comparing it to the full model with anova. The nlme package has a function gls that creates model
objects without random effects in a manner analogous to those specified with lme.
model.null = gls(Sodium ~ Instructor,
data=Data,
method="REML")

524
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

anova(model,
model.null)

Model df AIC BIC logLik Test L.Ratio p-value

model 1 5 749.9281 760.1434 -369.9641
model.null 2 4 758.4229 766.5951 -375.2114 1 vs 2 10.49476 0.0012

p-value and pseudo R-squared for model

By default, the nagelkerke function will compare an lme object to a null model with the fixed effects
removed, but the random effects retained.
library(rcompanion)

nagelkerke(model)

$Models

Model: "lme.formula, Sodium ~ Instructor, Data, ~1 | Town, REML"

Null: "lme.formula, Sodium ~ 1, Data, ~1 | Town, REML"

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.00971559
Cox and Snell (ML) 0.11818400
Nagelkerke (Cragg and Uhler) 0.11818400

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -3.7732 7.5463 0.02298

$Messages
[1] "Note: For models fit with REML, these statistics are based on refitting with
ML"

Another approach to determining the p-value and pseudo R-squared for an lme model is to compare
the model to a null model with only an intercept and neither the fixed nor the random effects. To
accomplish this, the null model has to be specified with the gls function.
model.null.2 = gls(Sodium ~ 1,
data=Data,
method="REML")

library(rcompanion)

nagelkerke(fit = model,
null = model.null.2)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.0239772
Cox and Snell (ML) 0.2701590
Nagelkerke (Cragg and Uhler) 0.2701600

525
 ONE-WAY ANOVA WITH RANDOM BLOCKS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-3 -9.4479 18.896 0.00028731

Post-hoc analysis
The lsmeans package is able to handle lme objects. For futher details, see ?lsmeans::models. For a
review of mean separation tests and least square means, see the chapters What are Least Square
Means?; the chapter Least Square Means for Multiple Comparisons; and the “Post-hoc analysis: mean
separation tests” section in the One-way ANOVA chapter.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust="tukey") ### Tukey-adjusted comparisons

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lower-case letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Instructor lsmean SE df lower.CL upper.CL .group

Melissa Robins 1153.201 82.92335 1 99.55992 2206.842 a
Coach McGuirk 1216.799 82.92335 1 163.15784 2270.440 ab
Brendon Small 1280.137 82.59437 1 230.67625 2329.598 b

Confidence level used: 0.95

P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

leastsquare

$contrasts
contrast estimate SE df t.ratio p.value
Brendon Small - Coach McGuirk 63.33828 45.83662 56 1.382 0.3572
Brendon Small - Melissa Robins 126.93620 46.46588 56 2.732 0.0225
Coach McGuirk - Melissa Robins 63.59792 47.96514 56 1.326 0.3869

P value adjustment: tukey method for comparing a family of 3 estimate

Comparison of results from one-way ANOVA with blocks

Just for interest, a comparison of results from this chapter and the previous chapter are presented
here. There is some variation in the values obtained for these statistics.
Fixed lmer lme
p-value for Instructor 0.034 0.030 0.024

526
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

p-value for Town 0.00019 0.001 0.0012

Mean separation a a a
for Instructor ab ab ab
b b b

Overall p-value < 0.0001 0.00024 0.00029

R-square or 0.348 0.270 0.270

Pseudo R-square
(Nagelkerke)

Two-way ANOVA
A two-way anova can investigate the main effects of each of two independent factor variables, as well
as the effect of the interaction of these variables.

For an overview of the concepts in multi-way analysis of variance, review the chapter Factorial
ANOVA: Main Effects, Interaction Effects, and Interaction Plots.

For a review of mean separation tests and least square means, see the chapters What are Least Square
Means?; the chapter Least Square Means for Multiple Comparisons; and the “Post-hoc analysis: mean
separation tests” section in the One-way ANOVA chapter.

Appropriate data
• Two-way data. That is, one dependent variable measured across two independent factor
variables
• Dependent variable is interval/ratio, and is continuous
• Independent variables are a factor with two or more levels. That is, two or more groups for
each independent variable
• In the general linear model approach, residuals are normally distributed
• In the general linear model approach, groups have the same variance. That is,
homoscedasticity
• Observations among groups are independent. That is, not paired or repeated measures data
• Moderate deviation from normally-distributed residuals is permissible

Hypotheses
• Null hypothesis 1: The means of the dependent variable for each group in the first
independent variable are equal
• Alternative hypothesis 1 (two-sided): The means of the dependent variable for each group in
the first independent variable are not equal
• Similar hypotheses are considered for the other independent variable
• Similar hypotheses are considered for the interaction effect

527
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Interpretation
• Reporting significant results for the effect of one independent variable as “Significant
differences were found among means for groups.” is acceptable. Alternatively, “A significant
effect for independent variable on dependent variable was found.”
• Reporting significant results for the interaction effect as “A significant effect for the
interaction of independent variable 1 and independent variable 2 on dependent variable was
found.” Is acceptable.
• Reporting significant results for mean separation post-hoc tests as “Mean of variable Y for
group A was different than that for group B.” is acceptable.

Packages used in this chapter

The packages used in this chapter include:

• psych
• FSA
• car
• multcompView
• lsmeans
• Rmisc
• ggplot2
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(Rmisc)){install.packages("Rmisc")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(rcompanion)){install.packages("rcompanion")}

Two-way ANOVA example

This example will revisit the sodium intake data set with Brendon Small and the other instructors.
This time, though, they want to test not only the different nutrition education programs (indicated by
Instructor), but also four supplements to the program, each of which they have used with some
students.
Input = ("
Instructor Supplement Sodium
'Brendon Small' A 1200
'Brendon Small' A 1400
'Brendon Small' A 1350
'Brendon Small' A 950
'Brendon Small' A 1400
'Brendon Small' B 1150

528
TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Brendon Small' B 1300

'Brendon Small' B 1325
'Brendon Small' B 1425
'Brendon Small' B 1500
'Brendon Small' C 1250
'Brendon Small' C 1150
'Brendon Small' C 950
'Brendon Small' C 1150
'Brendon Small' C 1600
'Brendon Small' D 1300
'Brendon Small' D 1050
'Brendon Small' D 1300
'Brendon Small' D 1700
'Brendon Small' D 1300
'Coach McGuirk' A 1100
'Coach McGuirk' A 1200
'Coach McGuirk' A 1250
'Coach McGuirk' A 1050
'Coach McGuirk' A 1200
'Coach McGuirk' B 1250
'Coach McGuirk' B 1350
'Coach McGuirk' B 1350
'Coach McGuirk' B 1325
'Coach McGuirk' B 1525
'Coach McGuirk' C 1225
'Coach McGuirk' C 1125
'Coach McGuirk' C 1000
'Coach McGuirk' C 1125
'Coach McGuirk' C 1400
'Coach McGuirk' D 1200
'Coach McGuirk' D 1150
'Coach McGuirk' D 1400
'Coach McGuirk' D 1500
'Coach McGuirk' D 1200
'Melissa Robins' A 900
'Melissa Robins' A 1100
'Melissa Robins' A 1150
'Melissa Robins' A 950
'Melissa Robins' A 1100
'Melissa Robins' B 1150
'Melissa Robins' B 1250
'Melissa Robins' B 1250
'Melissa Robins' B 1225
'Melissa Robins' B 1325
'Melissa Robins' C 1125
'Melissa Robins' C 1025
'Melissa Robins' C 950
'Melissa Robins' C 925
'Melissa Robins' C 1200
'Melissa Robins' D 1100
'Melissa Robins' D 950
'Melissa Robins' D 1300
'Melissa Robins' D 1400
'Melissa Robins' D 1100
")

529
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

Data$Supplement = factor(Data$Supplement,
levels=unique(Data$Supplement))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Use Summarize to check if cell sizes are balanced

We can use the n or nvalid columns in the Summarize output to check if cell sizes are balanced. That is,
if each combination of treatments have the same number of observations.
library(FSA)

Summarize(Sodium ~ Supplement + Instructor,

data=Data,
digits=3)

Define linear model

Note that the interaction of Instruction and Supplement is indicated in the model definition by
Instructor:Supplement.

model = lm(Sodium ~ Instructor + Supplement + Instructor:Supplement,

data = Data)

summary(model) ### Will show overall p-value and r-square

Multiple R-squared: 0.3491, Adjusted R-squared: 0.2

F-statistic: 2.341 on 11 and 48 DF, p-value: 0.02121

530
TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Conduct analysis of variance

library(car)

Anova(model,
type = "II") ### Type II sum of squares

Anova Table (Type II tests)

Response: Sodium
Sum Sq Df F value Pr(>F)
Instructor 290146 2 6.0173 0.004658 **
Supplement 306125 3 4.2324 0.009841 **
Instructor:Supplement 24438 6 0.1689 0.983880
Residuals 1157250 48

Histogram of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model),
residuals(model))

531
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plot(model) ### Additional model checking plots

Post-hoc analysis: mean separation tests

For an explanation of using least square means for multiple comparisons, see the section “Post-hoc
analysis: mean separation tests” in the One-way ANOVA chapter.

LS means for main effects

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust = "tukey")

leastsquare

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lowercase letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Instructor lsmean SE df lower.CL upper.CL .group

Melissa Robins 1123.75 34.71986 48 1053.941 1193.559 a
Coach McGuirk 1246.25 34.71986 48 1176.441 1316.059 b
Brendon Small 1287.50 34.71986 48 1217.691 1357.309 b

Results are averaged over the levels of: Supplement

Confidence level used: 0.95
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

532
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Supplement,
adjust = "tukey")

leastsquare

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lowercase letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Supplement lsmean SE df lower.CL upper.CL .group

C 1146.667 40.09104 48 1066.058 1227.275 a
A 1153.333 40.09104 48 1072.725 1233.942 a
D 1263.333 40.09104 48 1182.725 1343.942 ab
B 1313.333 40.09104 48 1232.725 1393.942 b

Results are averaged over the levels of: Instructor

Confidence level used: 0.95
P value adjustment: tukey method for comparing a family of 4 estimates
significance level used: alpha = 0.05

Plot of means and confidence intervals for main effects

Because the interaction term was not significant, we would probably want to present means for the
main effects only. Some authors would suggest re-doing the analysis without the interaction included.

Here we’ll use the summarySE function in Rmisc to get the traditional confidence intervals for the
means, and then use ggplot to plot the means.

Note that the last few lines of code add mean separation letters to the plot. This code will need to be
changed manually in other instances.

library(Rmisc)

Sum = summarySE(data = Data,

measurevar = "Sodium",
groupvars = "Instructor",
conf.interval = 0.95)

Sum

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Instructor,
y = Sodium)) +

533
TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

geom_errorbar(aes(ymin = Sodium - ci,

ymax = Sodium + ci),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Mean sodium, mg") +

annotate("text",
x = 1:3,
y = Sum$Sodium + Sum$ci + 20,
label = c("b", "b", "a"))

library(Rmisc)

Sum = summarySE(data = Data,

measurevar = "Sodium",
groupvars = "Supplement",
conf.interval = 0.95)

Sum

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Supplement,
y = Sodium)) +
geom_errorbar(aes(ymin = Sodium - ci,

534
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

ymax = Sodium + ci),

width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Mean sodium, mg") +

annotate("text",
x = 1:4,
y = Sum$Sodium + Sum$ci + 20,
label = c("a", "b", "a", "ab"))

Plot of LS means and confidence intervals for interaction

If the interaction effect were significant, it would make sense to present an interaction plot of the
means for both main effects together on one plot. We can indicate mean separations with letters on
the plot.

Extracting the LS means from the output of the lsmeans function takes a few extra steps. We’ll create a
data frame called Sum, and pass that data frame to ggplot to plot the LS means and confidence
intervals.

leastsquare = lsmeans(model,
pairwise ~ Supplement:Instructor,
adjust = "tukey")

### Extract variables from leastsquare and create new data frame called Sum

535
TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instructor = summary(leastsquare$lsmeans)["Instructor"]
Supplement = summary(leastsquare$lsmeans)["Supplement"]
lsmean = summary(leastsquare$lsmeans)["lsmean"]
lower.CL = summary(leastsquare$lsmeans)["lower.CL"]
upper.CL = summary(leastsquare$lsmeans)["upper.CL"]

Sum = data.frame(Instructor, Supplement, lsmean, lower.CL, upper.CL)

Sum

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x=Instructor,
y=lsmean,
color=Supplement)) +
geom_errorbar(aes(ymin=lower.CL,
ymax=upper.CL),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold"))+
ylab("LS mean sodium, mg")

Least square mean daily sodium intake for students in each of three classes with
four different supplemental programs. Error bars indicate confidence intervals
for the least square means.

536
 TWO-WAY ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Exercises T

1. Considering Brendon, Melissa, and McGuirk’s data with for instructional programs and supplements,

Is the experimental design balanced?

Does Instructor have a significant effect on sodium intake?

Does Supplement have a significant effect on sodium intake?

Does the interaction of Instructor and Supplement have a significant effect on sodium intake?

Are the residuals from the analysis reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean sodium intake from which
others?

Does this result correspond to what you would have thought from looking at the plot of the LS
means and confidence intervals?

How would you summarize the effect of the supplements?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency. Dr.
Katz, and Laura, and Ben want to compare their students’ mean typing speed between their classes, as
well as for the software they used to teach typing.
Instructor Software Words.per.minute
'Dr. Katz Professional Therapist' A 35
'Dr. Katz Professional Therapist' A 50
'Dr. Katz Professional Therapist' A 55
'Dr. Katz Professional Therapist' A 60
'Dr. Katz Professional Therapist' B 65
'Dr. Katz Professional Therapist' B 60
'Dr. Katz Professional Therapist' B 70
'Dr. Katz Professional Therapist' B 55
'Dr. Katz Professional Therapist' C 45
'Dr. Katz Professional Therapist' C 55
'Dr. Katz Professional Therapist' C 60
'Dr. Katz Professional Therapist' C 45
'Laura the Receptionist' A 55
'Laura the Receptionist' A 60
'Laura the Receptionist' A 75
'Laura the Receptionist' A 65
'Laura the Receptionist' B 60
'Laura the Receptionist' B 70
'Laura the Receptionist' B 75
'Laura the Receptionist' B 70

537
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Laura the Receptionist' C 65

'Laura the Receptionist' C 72
'Laura the Receptionist' C 73
'Laura the Receptionist' C 65
'Ben Katz' A 55
'Ben Katz' A 55
'Ben Katz' A 70
'Ben Katz' A 55
'Ben Katz' B 65
'Ben Katz' B 60
'Ben Katz' B 70
'Ben Katz' B 60
'Ben Katz' C 60
'Ben Katz' C 62
'Ben Katz' C 63
'Ben Katz' C 65

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

What was the LS mean typing speed for each instructor’s class?

Does Instructor have a significant effect on typing speed?

Does Software have a significant effect on typing speed?

Are the residuals reasonably normal and homoscedastic?

Which instructors had classes with significantly different mean typing speed from which
others?

Does this result correspond to what you would have thought from looking at the plot of the LS
means and standard errors?

Is this design balanced or unbalanced?

Repeated Measures ANOVA

When an experimental design takes measurements on the same experimental unit over time, the
analysis of the data must take into account the probability that measurements for a given
experimental unit will be correlated in some way. For example, if we were measuring calorie intake
for students, we would expect that if one student had a higher intake at Time 1, that that student would
have a higher intake others at Time 2, and so on.

In previous chapters, our approach to deal with non-independent observations was to treat Student as
a blocking variable or as a random (blocking) variable.

538
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The approach in this chapter is to include an autocorrelation structure in the model using the nmle
package.

Packages used in this chapter

The packages used in this chapter include:

• psych
• nlme
• car
• multcompView
• lsmeans
• ggplot2
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(nlme)){install.packages("nlme")}
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(rcompanion)){install.packages("rcompanion")}

Repeated measures ANOVA example

In this example, students were asked to document their daily caloric intake once a month for six
months. Students were divided into three groups with each receiving instruction in nutrition
education using one of three curricula.

There are different ways we might approach this problem. If we simply wanted to see if one
curriculum was better at decreasing caloric intake in students, we might do a simple analysis of
variance on the difference between each student’s final and initial intake.

In the approach here we will use a repeated measures analysis with all the measurements, treating
Student as a random variable to take into account native differences among students, and including an
autocorrelation structure.
Input = ("
Instruction Student Month Calories.per.day
'Curriculum A' a 1 2000
'Curriculum A' a 2 1978
'Curriculum A' a 3 1962
'Curriculum A' a 4 1873
'Curriculum A' a 5 1782
'Curriculum A' a 6 1737
'Curriculum A' b 1 1900

539
REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Curriculum A' b 2 1826

'Curriculum A' b 3 1782
'Curriculum A' b 4 1718
'Curriculum A' b 5 1639
'Curriculum A' b 6 1644
'Curriculum A' c 1 2100
'Curriculum A' c 2 2067
'Curriculum A' c 3 2065
'Curriculum A' c 4 2015
'Curriculum A' c 5 1994
'Curriculum A' c 6 1919
'Curriculum A' d 1 2000
'Curriculum A' d 2 1981
'Curriculum A' d 3 1987
'Curriculum A' d 4 2016
'Curriculum A' d 5 2010
'Curriculum A' d 6 1946
'Curriculum B' e 1 2100
'Curriculum B' e 2 2004
'Curriculum B' e 3 2027
'Curriculum B' e 4 2109
'Curriculum B' e 5 2197
'Curriculum B' e 6 2294
'Curriculum B' f 1 2000
'Curriculum B' f 2 2011
'Curriculum B' f 3 2089
'Curriculum B' f 4 2124
'Curriculum B' f 5 2199
'Curriculum B' f 6 2234
'Curriculum B' g 1 2000
'Curriculum B' g 2 2074
'Curriculum B' g 3 2141
'Curriculum B' g 4 2199
'Curriculum B' g 5 2265
'Curriculum B' g 6 2254
'Curriculum B' h 1 2000
'Curriculum B' h 2 1970
'Curriculum B' h 3 1951
'Curriculum B' h 4 1981
'Curriculum B' h 5 1987
'Curriculum B' h 6 1969
'Curriculum C' i 1 1950
'Curriculum C' i 2 2007
'Curriculum C' i 3 1978
'Curriculum C' i 4 1965
'Curriculum C' i 5 1984
'Curriculum C' i 6 2020
'Curriculum C' j 1 2000
'Curriculum C' j 2 2029
'Curriculum C' j 3 2033
'Curriculum C' j 4 2050
'Curriculum C' j 5 2001
'Curriculum C' j 6 1988
'Curriculum C' k 1 2000
'Curriculum C' k 2 1976

540
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Curriculum C' k 3 2025

'Curriculum C' k 4 2047
'Curriculum C' k 5 2033
'Curriculum C' k 6 1984
'Curriculum C' l 1 2000
'Curriculum C' l 2 2020
'Curriculum C' l 3 2009
'Curriculum C' l 4 2017
'Curriculum C' l 5 1989
'Curriculum C' l 6 2020
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instruction = factor(Data$Instruction,
levels=unique(Data$Instruction))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Define model and conduct analysis of deviance

This example will use a mixed effects model to describe the repeated measures analysis, using the lme
function in the nlme package. Student is treated as a random variable in the model.

The autocorrelation structure is described with the correlation statement. In this case, corAR1 is used
to indicate a temporal autocorrelation structure of order one, often abbreviated as AR(1). This
statement takes the form:
correlation = Structure(form = ~ time | subjvar)

where:
• Structure is the autocorrelation structure. Options are listed in library(nlme); ?corClasses
• time is the variable indicating time. In this case, Month. For the corAR1 structure, the time
variable must be an integer variable.

541
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• subjvar indicates the variable for experimental units, in this case Student. Autocorrelation is
modeled within levels of the subjvar, and not between them.

For the corAR1 structure, a value for the first order correlation can be specified. In this case, the value
of 0.429 is found using the ACF function in the “Optional analysis: determining autocorrelation in
residuals” section below.

Autocorrelation structures can be chosen by either of two methods. The first is to choose a structure
based on theoretical expectations of how the data should be correlated. The second is to try various
autocorrelation structures and compare the resulting models with a criterion like AIC, AICc, or BIC to
choose the structure that best models the data.

Optional technical note:

In this case, it isn’t necessary to use a mixed effects model. That is, it’s not necessary to include
Student as a random variable. A similar analysis could be conducted by using the gls function in the
nlme package, and including the correlation option, but excluding the random option, as follows in
black.

library(nlme)

model = gls(Calories.per.day ~ Instruction + Month + Instruction*Month,

correlation = corAR1(form = ~ Month | Student,
value = 0.8990),
data=Data,
method="REML")

library(car)

Anova(model)

Code for analysis

library(nlme)

model = lme(Calories.per.day ~ Instruction + Month + Instruction*Month,

random = ~1|Student,
correlation = corAR1(form = ~ Month | Student,
value = 0.4287),
data=Data,
method="REML")

library(car)

Anova(model)

Analysis of Deviance Table (Type II tests)

Response: Calories.per.day
Chisq Df Pr(>Chisq)
Instruction 10.4221 2 0.005456 **
Month 0.0198 1 0.888045

542
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instruction:Month 38.6045 2 4.141e-09 ***

Test the random effects in the model

The random effects in the model can be tested by comparing the model to a model fitted with just the
fixed effects and excluding the random effects. Because there are not random effects in this second
model, the gls function in the nlme package is used to fit this model.
model.fixed = gls(Calories.per.day ~ Instruction + Month + Instruction*Month,
data=Data,
method="REML")

anova(model,
model.fixed)

Model df AIC BIC logLik Test L.Ratio p-value

model 1 9 716.9693 736.6762 -349.4847
model.fixed 2 7 813.6213 828.9489 -399.8106 1 vs 2 100.652 <.0001

p-value and pseudo R-squared for model

The nagelkerke function can be used to calculate a p-value and pseudo R-squared value for the model.

One approach is to define the null model as one with no fixed effects except for an intercept, indicated
with a 1 on the right side of the ~. And to also include the random effects, in this case 1|Student.
library(rcompanion)

model.null = lme(Calories.per.day ~ 1,
random = ~1|Student,
data = Data)

nagelkerke(model,
model.null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.123969
Cox and Snell (ML) 0.768652
Nagelkerke (Cragg and Uhler) 0.768658

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-6 -52.698 105.4 1.8733e-20

Another approach to determining the p-value and pseudo R-squared for an lme model is to compare
the model to a null model with only an intercept and neither the fixed nor the random effects. To
accomplish this, the null model has to be specified with the gls function.
library(rcompanion)

model.null.2 = gls(Calories.per.day ~ 1,

543
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

data = Data)

nagelkerke(model,
model.null.2)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.168971
Cox and Snell (ML) 0.877943
Nagelkerke (Cragg and Uhler) 0.877946

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-7 -75.717 151.43 2.0282e-29

Post-hoc analysis
The lsmeans package is able to handle lme objects. For further details, see ?lsmeans::models. For a
review of mean separation tests and least square means, see the chapters What are Least Square
Means?; the chapter Least Square Means for Multiple Comparisons; and the “Post-hoc analysis: mean
separation tests” section in the One-way ANOVA chapter.

Because Month is an integer variable, not a factor variable, it is listed in the lsmeans cld table as its
average only.
library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instruction:Month,
adjust="tukey") ### Tukey-adjusted comparisons

cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lower-case letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Instruction Month lsmean SE df lower.CL upper.CL .group

Curriculum A 3.5 1907.601 42.83711 11 1787.206 2027.996 a
Curriculum C 3.5 1997.429 42.83711 9 1872.222 2122.637 ab
Curriculum B 3.5 2102.965 42.83711 9 1977.758 2228.172 b

Confidence level used: 0.95

Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05.05

Interaction plot
For this plot, we will use the groupwiseMean function to calculate the natural mean of each Instruction
x Month combination, along with the confidence interval of each mean with the percentile method.

544
REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

Sum = groupwiseMean(data = Data,

var = "Calories.per.day",
group = c("Instruction", "Month" ),
conf = 0.95,
digits = 3,
traditional = FALSE,
percentile = TRUE)

Sum

Instruction Month n Mean Conf.level Percentile.lower Percentile.upper

1 Curriculum A 1 4 2000 0.95 1920 2080
2 Curriculum A 2 4 1960 0.95 1860 2040
3 Curriculum A 3 4 1950 0.95 1830 2040
4 Curriculum A 4 4 1910 0.95 1760 2020
5 Curriculum A 5 4 1860 0.95 1710 2000
6 Curriculum A 6 4 1810 0.95 1690 1930
7 Curriculum B 1 4 2020 0.95 2000 2080
8 Curriculum B 2 4 2010 0.95 1980 2060
9 Curriculum B 3 4 2050 0.95 1990 2120
10 Curriculum B 4 4 2100 0.95 2020 2180
11 Curriculum B 5 4 2160 0.95 2040 2250
12 Curriculum B 6 4 2190 0.95 2040 2280
13 Curriculum C 1 4 1990 0.95 1960 2000
14 Curriculum C 2 4 2010 0.95 1990 2020
15 Curriculum C 3 4 2010 0.95 1990 2030
16 Curriculum C 4 4 2020 0.95 1990 2050
17 Curriculum C 5 4 2000 0.95 1990 2020
18 Curriculum C 6 4 2000 0.95 1990 2020

library(ggplot2)

pd = position_dodge(.2)

ggplot(Sum, aes(x = Month,

y = Mean,
color = Instruction)) +
geom_errorbar(aes(ymin=Percentile.lower,
ymax=Percentile.upper),
width=.2, size=0.7, position=pd) +
geom_point(shape=15, size=4, position=pd) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Mean calories per day")

545
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Histogram of residuals
Residuals from a mixed model fit with nlme should be normally distributed. Plotting residuals vs.
fitted values, to check for homoscedasticity and independence, is probably also advisable.
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model),
residuals(model))

546
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Optional analysis: determining autocorrelation in residuals

The ACF function in the nlme package will indicate the autocorrelation for lags in the time variable.
Note that for a gls model, the form of the autocorrelation structure can be specified. For an lme model,
the function uses the innermost group level and assumes equally spaced intervals.

library(nlme)

model.a = gls(Calories.per.day ~ Instruction + Month + Instruction*Month,

data=Data)

ACF(model.a,
form = ~ Month | Student)

lag ACF
1 0 1.0000000
2 1 0.8989822
3 2 0.7462712
4 3 0.6249217
5 4 0.5365430
6 5 0.4564673

library(nlme)

model.b = lme(Calories.per.day ~ Instruction + Month + Instruction*Month,

random = ~1|Student,
data=Data)

ACF(model.b)

lag ACF
1 0 1.0000000
2 1 0.4286522
3 2 -0.1750233

547
 REPEATED MEASURES ANOVA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

4 3 -0.6283754
5 4 -0.8651088
6 5 -0.6957774

Optional discussion on specifying formulae for repeated measures analysis

Specifying random effects in models

The general schema for specifying random effects in R is:
Formula.for.random.effects | Unit.for.which.these.apply

So that, in the nlme package:

• random = ~ 1 | subject
Indicates that each subject will have its own intercept
• random = ~ rep | subject
Indicates that each subject will have its own intercept and its own slope for rep
• random = ~ 1 | rep/subject
Indicates that each subject-within-rep unit will have its own intercept

Indicating time and subject variables

A simple repeated analysis statement in proc mixed in SAS could be specified with:
repeated date / subject = id type = AR(1)

A similar specification in with the gls function in nlme package in R would be:
correlation = corAR1(form = ~ date | id)

Likewise, a simple mixed effects repeated analysis statement in proc mixed in SAS could be specified
with:
random id
repeated date / subject = id type = AR(1)

A similar specification in with the lme function in nlme package in R would be:
random = ~1 | id,
correlation = corAR1(form = ~ date | id)

Specifying nested effects

In repeated measures analysis, it is common to used nested effects. For example, if our subject
variable is treatment within block,

548
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In SAS:
treatment(block)

is equivalent to
block block*treatment

In R:
block/treatment

is equivalent to
block + block:treatment

Correlation and Linear Regression

Correlation and linear regression each explore the relationship between two quantitative variables.
Both are very common analyses.

Correlation determines if one variable varies systematically as another variable changes. It does not
specify that one variable is the dependent variable and the other is the independent variable. The
three forms of correlation presented here are Pearson, Kendall, and Spearman. Pearson is a
parametric test that assumes that data are bivariate normal. Kendall and Spearman are
nonparametric tests. It is often useful to look at which variables are correlated to others in a data set,
and it is especially useful to see which variables correlate to a particular variable of interest.

In contrast, linear regression specifies one variable as the independent variable and another as the
dependent variable. The resultant model relates the variables with a linear relationship. Linear
regression is a parametric test that assumes normality, homoscedasticity, and independence of
residuals, as well as a linear relationship between the two variables.

Appropriate data
• For Pearson correlation, two interval/ratio variables. Together the data in the variables are
bivariate normal. The relationship between the two variables is linear. Outliers can
detrimentally affect results.

• For Kendall correlation, two variables of interval/ratio or ordinal type.

• For Spearman correlation, two variables of interval/ratio or ordinal type.

549
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• For linear regression, two interval/ratio variables. The relationship between the two
variables is linear. Residuals are normal, independent, and homoscedastic. Outliers can
affect the results unless robust methods are used.

Hypotheses
• For correlation, null hypothesis: The correlation coefficient (r, tau, or rho) is zero. Or, there
is no correlation between the two variables.

• For linear regression, null hypothesis: The slope of the fit line is zero. Or, there is no linear
relationship between the two variables.

Interpretation
• For correlation, reporting significant results as “Variable A was significantly correlated to
Variable B” is acceptable. Alternatively, “A significant correlation between Variable A and
Variable B was found.” Or, “Variables A, B, and C were significantly correlated.”

• For linear regression, reporting significant results as “Variable A was significantly linearly
related to Variable B” is acceptable. Alternatively, “A significant linear regression between
Variable A and Variable B was found.”

Packages used in this chapter

The packages used in this chapter include:

• psych
• PerformanceAnalytics
• ggplot2
• ggExtra
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(PerformanceAnalytics)){install.packages("PerformanceAnalytics")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(ggExtra)){install.packages("ggExtra")}
if(!require(rcompanion)){install.packages("rcompanion")}

Examples for correlation and linear regression

Brendon Small and company recorded several measurements for students in their classes related to
their nutrition education program: Grade, Weight in kilograms, intake of Calories per day, daily Sodium
intake in milligrams, and Score on the assessment of knowledge gain.

Input = ("
Instructor Grade Weight Calories Sodium Score
'Brendon Small' 6 43 2069 1287 77
'Brendon Small' 6 41 1990 1164 76

550
CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Brendon Small' 6 40 1975 1177 76

'Brendon Small' 6 44 2116 1262 84
'Brendon Small' 6 45 2161 1271 86
'Brendon Small' 6 44 2091 1222 87
'Brendon Small' 6 48 2236 1377 90
'Brendon Small' 6 47 2198 1288 78
'Brendon Small' 6 46 2190 1284 89
'Jason Penopolis' 7 45 2134 1262 76
'Jason Penopolis' 7 45 2128 1281 80
'Jason Penopolis' 7 46 2190 1305 84
'Jason Penopolis' 7 43 2070 1199 68
'Jason Penopolis' 7 48 2266 1368 85
'Jason Penopolis' 7 47 2216 1340 76
'Jason Penopolis' 7 47 2203 1273 69
'Jason Penopolis' 7 43 2040 1277 86
'Jason Penopolis' 7 48 2248 1329 81
'Melissa Robins' 8 48 2265 1361 67
'Melissa Robins' 8 46 2184 1268 68
'Melissa Robins' 8 53 2441 1380 66
'Melissa Robins' 8 48 2234 1386 65
'Melissa Robins' 8 52 2403 1408 70
'Melissa Robins' 8 53 2438 1380 83
'Melissa Robins' 8 52 2360 1378 74
'Melissa Robins' 8 51 2344 1413 65
'Melissa Robins' 8 51 2351 1400 68
'Paula Small' 9 52 2390 1412 78
'Paula Small' 9 54 2470 1422 62
'Paula Small' 9 49 2280 1382 61
'Paula Small' 9 50 2308 1410 72
'Paula Small' 9 55 2505 1410 80
'Paula Small' 9 52 2409 1382 60
'Paula Small' 9 53 2431 1422 70
'Paula Small' 9 56 2523 1388 79
'Paula Small' 9 50 2315 1404 71
'Coach McGuirk' 10 52 2406 1420 68
'Coach McGuirk' 10 58 2699 1405 65
'Coach McGuirk' 10 57 2571 1400 64
'Coach McGuirk' 10 52 2394 1420 69
'Coach McGuirk' 10 55 2518 1379 70
'Coach McGuirk' 10 52 2379 1393 61
'Coach McGuirk' 10 59 2636 1417 70
'Coach McGuirk' 10 54 2465 1414 59
'Coach McGuirk' 10 54 2479 1383 61
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Check the data frame

551
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Visualizing correlated variables

Correlation does not have independent and dependent variables

Notice in the formulae for the pairs and cor.test functions, that all variables are named to the right of
the ~. One reason for this is that correlation is measured between two variables without assuming
that one variable is the dependent variable and one is the independent variable. By necessity, one
variable must be plotted on the x-axis and one on the y-axis, but their placement is arbitrary.

Multiple correlation
The pairs function can plot multiple numeric or integer variables on a single plot to look for
correlations among the variables.
pairs(data=Data,
~ Grade + Weight + Calories + Sodium + Score)

552
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The corr.test function in the psych package can be used in a similar manner, with the output being a
table of correlation coefficients and a table of p-values. p-values can be adjusted with the adjust=
option. Options for correlation methods are “pearson”, “kendall”, and “spearman”. The function can
produce confidence intervals for the correlation coefficients, but I recommend using the cor.ci function
in the psych package for this task (not shown here).

The corr.test function requires that the data frame contain only numeric or integer variables, so we
will first create a new data frame called Data.num containing only the numeric and integer variables.
Data.num = Data[c("Grade", "Weight", "Calories", "Sodium", "Score")]

library(psych)

corr.test(Data.num,
use = "pairwise",
method = "pearson",
adjust = "none")

Correlation matrix

Grade Weight Calories Sodium Score

Grade 1.00 0.85 0.85 0.79 -0.70
Weight 0.85 1.00 0.99 0.87 -0.48
Calories 0.85 0.99 1.00 0.85 -0.48

553
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Sodium 0.79 0.87 0.85 1.00 -0.45

Score -0.70 -0.48 -0.48 -0.45 1.00

Sample Size
[1] 45

A useful plot of histograms, correlations, and correlation coefficients can be produced with the
chart.Correlation function in the PerformanceAnalytics package.

In the output, the numbers represent the correlation coefficients (r, tau, or rho, depending on whether
Pearson, Kendall, or Spearman correlation is selected, respectively). The stars represent the p-value of
the correlation:
* p < 0.05
** p < 0.01
*** p < 0.001

The function requires that the data frame contain only numeric or integer variables, so we will first
create a new data frame called Data.num containing only the numeric and integer variables.
Data.num = Data[c("Grade", "Weight", "Calories", "Sodium", "Score")]

library(PerformanceAnalytics)

chart.Correlation(Data.num,
method="pearson",
histogram=TRUE,
pch=16)

554
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Pearson correlation
Pearson correlation is a parametric analysis that requires that the relationship between the variables
is linear, and that the data be bivariate normal. Variables should be interval/ratio. The test is
sensitive to outliers.

The correlation coefficient, r, can range from +1 to –1, with +1 being a perfect positive correlation and
–1 being a perfect negative correlation. An r of 0 represents no correlation whatsoever, but a non-
significant result should also be considered a lack of statistical correlation.

Simple plot of the data

Note that the relationship between the y and x variables is not particularly linear.
plot(Sodium ~ Calories,
data=Data,
pch=16,
xlab = "Calories",
ylab = "Sodium")

555
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Pearson correlation
cor.test( ~ Sodium + Calories,
data=Data,
method = "pearson",
conf.level = 0.95)

Pearson's product-moment correlation

t = 10.534, df = 43, p-value = 1.737e-13

alternative hypothesis: true correlation is not equal to 0

95 percent confidence interval:

0.7397691 0.9145785

sample estimates:
cor
0.8489548

Plot residuals
It’s not a bad idea to look at the residuals from Pearson correlation to be sure the data meet the
assumption of bivariate normality. Unfortunately, the cor.test function doesn’t supply residuals. One
solution is to use the lm function, which actually redoes the analysis as a linear regression.
model = lm(Sodium ~ Calories,
data = Data)

x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

556
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Kendall correlation
Kendall correlation is considered a nonparametric analysis. It is a rank-based test that does not
require assumptions about the distribution of the data. Variables can be interval/ratio or ordinal.

The correlation coefficient from the test is tau, which can range from +1 to –1, with +1 being a perfect
positive correlation and –1 being a perfect negative correlation. A tau of 0 represents no correlation
whatsoever, but a non-significant result should also be considered a lack of statistical correlation.

Technically, the cor.test function in R calculates tau-b, which handles ties in ranks well.

The test is relatively robust to outliers in the data. The test is sometimes cited for being reliable when
there are small number of samples or when there are many ties in ranks.

Simple plot of the data

(See Pearson correlation above.)

Kendall correlation
cor.test( ~ Sodium + Calories,
data=Data,
method = "kendall")

Kendall's rank correlation tau

z = 6.2631, p-value = 3.774e-10

alternative hypothesis: true tau is not equal to 0

sample estimates:
tau
0.6490902

Spearman correlation
Spearman correlation is considered a nonparametric analysis. It is a rank-based test that does not
require assumptions about the distribution of the data. Variables can be interval/ratio or ordinal.
557
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The correlation coefficient from the test, rho, can range from +1 to –1, with +1 being a perfect positive
correlation and –1 being a perfect negative correlation. A rho of 0 represents no correlation
whatsoever, but a non-significant result should also be considered a lack of statistical correlation.

Spearman correlation is probably most often used with ordinal data. It tests for a monotonic
relationship between the variables. It is relatively robust to outliers in the data.

Simple plot of the data

(See Pearson correlation above.)

Spearman correlation
cor.test( ~ Sodium + Calories,
data=Data,
method = "spearman")

Spearman's rank correlation rho

S = 2729.7, p-value = 5.443e-12

alternative hypothesis: true rho is not equal to 0

sample estimates:
rho
0.8201766

Linear regression
Linear regression is a very common approach to model the relationship between two interval/ratio
variables. The method assumes that there is a linear relationship between the dependent variable and
the independent variable, and finds a best fit model for this relationship.

Dependent and Independent variables

When plotted, the dependent variable is usually placed on the y-axis, and the independent variable is
usually placed in the x-axis.

Interpretation of coefficients
The outcome of linear regression includes estimating the intercept and the slope of the linear model.
Linear regression can then be used as predictive model, whereby the model can be used to predict a y
value for any given x. In practice, the model shouldn’t be used to predict values beyond the range of
the x values used to develop the model.

Multiple, nominal, and ordinal independent variables

If there are multiple independent variables of interval/ratio type in the model, then linear regression
expands to multiple regression. The polynomial regression example in this chapter is a form of
multiple regression.

If the independent variable were of nominal type, then the linear regression would become a one-way
analysis of variance.

558
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Handling independent variables of ordinal type can be complicated. Often they are treated as either
nominal type or interval/ratio type, although there are drawbacks to each approach.

Assumptions
Linear regression assumes linear relationship between the two variables, normality of the residuals,
independence of the residuals, and homoscedasticity of residuals.

Note on writing r-squared

For bivariate linear regression, the r-squared value often uses a lower case r; however, some authors
prefer to use a capital R. For multiple regression, the R in the R-squared value is usually capitalized.
The name of the statistic may be written out as “r-squared” for convenience, or as r2.

Define model, and produce model coefficients, p-value, and r-squared value
Linear regression can be performed with the lm function, which was the same function we used for
analysis of variance.

The summary function for lm model objects includes estimates for model parameters (intercept and
slope), as well as an r-squared value for the model and p-value for the model. Note that even for
bivariate regression, the output calls the r-squared value “Multiple R-squared”.
model = lm(Sodium ~ Calories,
data = Data)

summary(model)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 519.07547 78.78211 6.589 5.09e-08 ***
Calories 0.35909 0.03409 10.534 1.74e-13 ***

Residual standard error: 38.89 on 43 degrees of freedom

Multiple R-squared: 0.7207, Adjusted R-squared: 0.7142
F-statistic: 111 on 1 and 43 DF, p-value: 1.737e-13

Plot data with best fit line

plot(Sodium ~ Calories,
data=Data,
pch=16,
xlab = "Calories",
ylab = "Sodium")

abline(model,
col = "blue",
lwd = 2)

559
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

There is a clear nonlinearity to the data, suggesting that the simple linear model is not the best fit. This
nonlinearity is apparent in the plot of residuals vs. fitted values as well.

The next chapter will include fitting linear plateau, quadratic plateau, and a curvilinear models with
this data.

The following section will attempt to improve the model fit by adding polynomial terms.

Plots of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

560
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plot(fitted(model),
residuals(model))

Polynomial regression
Polynomial regression adds additional terms to the model, so that the terms include some set of the
linear, quadratic, cubic, and quartic, etc., forms of the independent variable. These terms are
mathematically the independent variable, the square of the independent variable, the cube of the
independent variable, and so on.

Create polynomial terms in the data frame

Because the variable Calories is an integer variable, we will need to convert it to a numeric variable.
Otherwise R will produce errors when we try to square and cube the values of the variable.

Our new variables will be Calories2 for the square of Calories, Calories3 for the cube of Calories, and so
on.
Data$Calories = as.numeric(Data$Calories)

Data$Calories2 = Data$Calories * Data$Calories

Data$Calories3 = Data$Calories * Data$Calories * Data$Calories
Data$Calories4 = Data$Calories * Data$Calories * Data$Calories * Data$Calories

Define models and determine best model

Chances are that we will not need all of the polynomial terms to adequately model our data. One
approach to choosing the best model is to construct several models with increasing numbers of
polynomial terms, and then use a model selection criterion like AIC, AICc, or BIC to choose the best
one. These criteria balance the goodness-of-fit of each model versus its complexity. That is, the goal is
to find a model which adequately explains the data without having too many terms.

We can use the compareLM function to list AIC, AICc, and BIC for a series of models.
model.1 = lm(Sodium ~ Calories,
data = Data)

model.2 = lm(Sodium ~ Calories + Calories2,

561
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

data = Data)

model.3 = lm(Sodium ~ Calories + Calories2 + Calories3,

data = Data)

model.4 = lm(Sodium ~ Calories + Calories2 + Calories3 + Calories4,

data = Data)

library(rcompanion)

compareLM(model.1, model.2, model.3, model.4)

$Models
Formula
1 "Sodium ~ Calories"
2 "Sodium ~ Calories + Calories2"
3 "Sodium ~ Calories + Calories2 + Calories3"
4 "Sodium ~ Calories + Calories2 + Calories3 + Calories4"

$Fit.criteria
Rank Df.res AIC AICc BIC R.squared Adj.R.sq p.value Shapiro.W Shapiro.p
1 2 43 461.1 461.7 466.5 0.7207 0.7142 1.737e-13 0.9696 0.2800
2 3 42 431.4 432.4 438.6 0.8621 0.8556 8.487e-19 0.9791 0.5839
3 4 41 433.2 434.7 442.2 0.8627 0.8526 1.025e-17 0.9749 0.4288
4 5 40 428.5 430.8 439.4 0.8815 0.8696 5.566e-18 0.9625 0.1518

If we use BIC as our model fit criterion, we would choose model.2 as the best model for these data,
since it had the lowest BIC. AIC or AICc could have been used as criteria instead.

BIC tends to penalize models more than the other criteria for having additional parameters, so it will
tend to choose models with fewer terms.

There is not generally accepted advice as to which model fitting criterion to use. My advice might be to
use BIC when a more parsimonious model (one with fewer terms) is a priority, and in other cases use
AICc.

For final model, produce model coefficients, p-value, and R-squared value
summary(model.2)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -3.707e+03 6.463e+02 -5.736 9.53e-07 ***
Calories 4.034e+00 5.604e-01 7.198 7.58e-09 ***
Calories2 -7.946e-04 1.211e-04 -6.563 6.14e-08 ***

Residual standard error: 27.65 on 42 degrees of freedom

Multiple R-squared: 0.8621, Adjusted R-squared: 0.8556
F-statistic: 131.3 on 2 and 42 DF, p-value: < 2.2e-16

562
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Note on retaining lower-order effects

In polynomial regression, we typically keep all lower order effects of any higher order effects we
include. So, in this case, if the final model included Calories2, but the effect of Calories were not
significant, we would still include Calories in the model since we are including Calories2.

Plot data with best fit line

For bivariate data, the function plotPredy will plot the data and the predicted line for the model. It also
works for polynomial functions, if the order option is changed.
library(rcompanion)

plotPredy(data = Data,
y = Sodium,
x = Calories,
x2 = Calories2,
model = model.2,
order = 2,
xlab = "Calories per day",
ylab = "Sodium intake per day")

Plot of best fit line with confidence interval

library(ggplot2)

ggplot(Data,
aes(x = Calories,
y = Sodium)) +
geom_point() +
geom_smooth(method = "lm",
formula = y ~ poly(x, 2, raw=TRUE), ### polynomial of order 2
se = TRUE)

563
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plots of residuals
x = residuals(model.2)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model.2),
residuals(model.2))

564
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

A few of xkcd comics

Correlation
xkcd.com/552/

Extrapolating
xkcd.com/605/

Cat proximity
xkcd.com/231/

Exercises U

1. Considering the data from Brendon, Jason, Melissa, Paula, and McGuirk,

Which two variables are the most strongly correlated?

Which two variables are the least strongly correlated?

Are there any pairs of variables that are uncorrelated? Which?

Name a pair of variables that is positively correlated.

Name a pair of variables that is negatively correlated.

Is Sodium significantly correlated with Calories?

By linear regression, is there a significant linear relationship of Sodium vs. Calories?

Does the quadratic polynomial model fit the Sodium vs. Calories data better than the linear
model?

565
 CORRELATION AND LINEAR REGRESSION SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

How do you know?

2. As part of a professional skills program, a 4-H club tests its members for typing proficiency
(Words.per.minute), Proofreading skill, proficiency with using a Spreadsheet, and acumen in Statistics.
Instructor Grade Words.per.minute Proofreading Spreadsheet Statistics
'Dr. Katz' 6 35 53 75 61
'Dr. Katz' 6 50 77 24 51
'Dr. Katz' 6 55 71 62 55
'Dr. Katz' 6 60 78 27 91
'Dr. Katz' 6 65 84 44 95
'Dr. Katz' 6 60 79 38 50
'Dr. Katz' 6 70 96 12 94
'Dr. Katz' 6 55 61 55 76
'Dr. Katz' 6 45 73 59 75
'Dr. Katz' 6 55 75 55 80
'Dr. Katz' 6 60 85 35 84
'Dr. Katz' 6 45 61 49 80
'Laura' 7 55 59 79 57
'Laura' 7 60 60 60 60
'Laura' 7 75 90 19 64
'Laura' 7 65 87 32 65
'Laura' 7 60 70 33 94
'Laura' 7 70 84 27 54
'Laura' 7 75 87 24 59
'Laura' 7 70 97 38 74
'Laura' 7 65 86 30 52
'Laura' 7 72 91 36 66
'Laura' 7 73 88 20 57
'Laura' 7 65 86 19 71
'Ben Katz' 8 55 84 20 76
'Ben Katz' 8 55 63 44 94
'Ben Katz' 8 70 95 31 88
'Ben Katz' 8 55 63 69 93
'Ben Katz' 8 65 65 47 70
'Ben Katz' 8 60 61 63 92
'Ben Katz' 8 70 80 35 60
'Ben Katz' 8 60 88 38 58
'Ben Katz' 8 60 71 65 99
'Ben Katz' 8 62 78 46 54
'Ben Katz' 8 63 89 17 60
'Ben Katz' 8 65 75 33 77

For each of the following, answer the question, and show the output from the analyses you used to
answer the question.

Which two variables are the most strongly correlated?

Name a pair of variables that are uncorrelated.

566
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Name a pair of variables that is positively correlated.

Name a pair of variables that is negatively correlated.

Consider the correlation between Spreadsheet and Proofreading.

What is the value of the correlation coefficient r for this correlation?

What is the value of tau?

What is the value of rho?

Conduct a linear regression of Proofreading vs. Words.per.minute.

What is the p-value for this model?

What is the r-squared value?

Do the residuals suggest that the linear regression model is an appropriate model?

What can you conclude about the results of the linear regression?

References

“Correlation and Linear Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook
of Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_01.html.

“Multiple Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/e_05.html.

Advanced Parametric Methods

Packages used in this chapter

The packages used in this chapter include:

• robustbase
• psych
• minpack.lm
• car
• rcompanion
• mblm

567
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

The following commands will install these packages if they are not already installed:
if(!require(robustbase)){install.packages("robustbase")}
if(!require(pysch)){install.packages("pysch")}
if(!require(nlstools)){install.packages("nlstools")}
if(!require(minpack.lm)){install.packages("minpack.lm")}
if(!require(car)){install.packages("car")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(mblm)){install.packages("mblm")}

More complex experimental designs

Except for t-tests, the approach of this book for parametric statistics has been to develop linear models
(with the lm function) or mixed effects models (with the nlme or lme4 packages) and then to apply
analysis of variance, model checking, and post-hoc testing.

These relatively simple models can be expanded to include more independent variables of either
continuous or factor type. Likewise, more complex designs can include nested and crossed factors.
Some traditional experimental designs include Latin square, split plot, and incomplete block.

Analysis of co-variance

Analysis of covariance combines continuous independent variables with factor independent variables.

A related design is the paired watershed design which, while it was developed for watershed studies,
can be adapted to various situations in a variety of fields.

“Analysis of Covariance” in Mangiafico, S.S. 2015. An R Companion for the Handbook of

Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_04.html.

Clausen, J.C. and J. Spooner. 1993. Paired Watershed Study Design. 841-F-93-009. United States
Environmental Protection Agency, Office of Water. Washington, DC.
nepis.epa.gov/Exe/ZyPURL.cgi?Dockey=20004PR6.TXT.

Dressing, S.A. and D.W. Meals. 2005. Designing water quality monitoring programs for
watershed projects. U.S. Environmental Protection Agency. Fairfax, VA.
www.epa.gov/sites/production/files/2015-10/documents/technote2_wq_monitoring.pdf.

Nonlinear regression and curvilinear regression

Nonlinear and curvilinear regression are similar to linear regression, except that the relationship
between the dependent variable and the independent variable is curved in some way. Specific
methods include polynomial regression, spline regression, and nonlinear regression.

568
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Polynomial regression was covered briefly in the previous chapter, while some examples of curvilinear
regression are shown below in the “Linear plateau and quadratic plateau models” section in this
chapter.

For further discussion, see:

“Curvilinear Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of

Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_03.html.

Multiple regression

Multiple regression is similar to linear regression, except that the model contains multiple
independent variables.

The polynomial regression example in the previous chapter is one example of multiple regression.

For further discussion, see:

“Multiple Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/e_05.html.

Logistic regression

Logistic regression is similar to linear regression, except that the dependent variable is categorical. In
standard logistic regression, the dependent variable has only two levels, and in multinomial logistic
regression, the dependent variable can have more than two levels.

For examples of logistic regression, see the chapter Models for Nominal Data in this book.

Analysis of count data

When the dependent variable is a counted quantity and not a measured quantity, it is appropriate to
use Poisson regression or related techniques.

For a discussion of when standard parametric approaches may or may not be used with count data,
see the “Count data may not be appropriate for common parametric tests” section in the Introduction
to Parametric Tests chapter.

For examples of appropriate ways to model count data, see the Hermite and Poisson Regression for
Count Data chapter in this book.

Robust techniques

Standard parametric analyses can be sensitive to outliers and other deviations from assumptions of
the analyses. The example below shows that a single point can affect the predicted line from standard
linear regression. It is said that this point has high leverage, as so has high influence in the model.

569
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

One approach in these cases is to use nonparametric techniques.

Another approach is to adjust the model to better fit the data.

A third way is to determine if certain data points are true outliers, and then give them further
examination.

A fourth way is to use a robust technique such as robust regression. The references below discuss
robust regression and robust techniques appropriate for designs lending themselves to an analysis of
variance approach.

“One-way Analysis with Permutation Test” in Mangiafico, S.S. 2015. An R Companion for the
Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_06a.html.

“Two-way Anova with Robust Estimation” in Mangiafico, S.S. 2015. An R Companion for the
Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_08a.html.

Search for Robust regression in the following:

“Correlation and Linear Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook
of Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_01.html.

Example of robust linear regression

Note that this example does not include checking assumptions of the model, but is intended only to
compare the results of ordinary linear regression and robust regression when there is a data point
with high influence.
Input=("
X Y
0 0
1 2
2 5
3 6
4 9
5 10
6 11
7 14
8 16
9 20
10 44
")

Data = read.table(textConnection(Input),header=TRUE)

Linear regression
model = lm(Y ~ X,
data = Data)

summary(model)

570
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -3.1364 3.6614 -0.857 0.413888
X 3.1182 0.6189 5.038 0.000701 ***

Residual standard error: 6.491 on 9 degrees of freedom

Multiple R-squared: 0.7383, Adjusted R-squared: 0.7092
F-statistic: 25.39 on 1 and 9 DF, p-value: 0.0007013

Calculate statistics and plot

Pvalue = pf(summary(model)$fstatistic[1],
summary(model)$fstatistic[2],
summary(model)$fstatistic[3],
lower.tail = FALSE)

R2 = summary(model)$r.squared

t1 = paste0("p-value: ", signif(Pvalue, digits=3))

t2 = paste0("R-squared: ", signif(R2, digits=3))
t3 = paste0("Intercept: ", signif(coef(model)[1], digits=3))
t4 = paste0("Slope: ", signif(coef(model)[2], digits=3))

plot(Y ~ X,
data = Data,
pch = 16)
abline(model,
col="blue",
lwd=2)
text(0, 38, labels = t1, pos=4)
text(0, 33, labels = t2, pos=4)
text(0, 28, labels = t3, pos=4)
text(0, 23, labels = t4, pos=4)

571
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cook’s distance
Cook’s distance is a measure of influence of each data point, specifically determining how much
predicted values would change if the observation were deleted. It is sometimes recommended that a
Cook’s distance of 1 or more merits further examination of the data point. However, there are other
recommendations for critical values for Cook’s distance.

In this case, the high Cook’s distance for observation 11 makes it a candidate for further evaluation.
barplot(cooks.distance(model))

Plot of residuals
This plot suggests that the residuals are not independent of the fitted values. We would probably want
to adjust the model by adding additional terms, using a nonlinear or curvilinear approach, or using a
robust analysis to minimize the influence of certain data points.
plot(fitted(model),
residuals(model))

Robust regression
library(robustbase)

572
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

model.r = lmrob(Y ~ X,
data = Data)

summary(model.r)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.09015 0.32804 0.275 0.79
X 2.04249 0.10970 18.619 1.7e-08 ***

Robust residual standard error: 0.9378

Multiple R-squared: 0.9817, Adjusted R-squared: 0.9796

Calculate statistics and plot

model.null = lmrob(Y ~ 1,
data = Data)

Pvalue = anova(model.r, model.null)[4][2,1]

R2 = summary(model.r)$r.squared

t1 = paste0("p-value: ", signif(Pvalue, digits=3))

t2 = paste0("R-squared: ", signif(R2, digits=3))
t3 = paste0("Intercept: ", signif(coefficients(model.r)[1], digits=3))
t4 = paste0("Slope: ", signif(coefficients(model.r)[2], digits=3))

plot(Y ~ X,
data = Data,
pch = 16)
abline(model.r,
col="blue",
lwd=2)
text(0, 38, labels = t1, pos=4)
text(0, 33, labels = t2, pos=4)
text(0, 28, labels = t3, pos=4)
text(0, 23, labels = t4, pos=4)

573
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Contrasts in Linear Models

Contrasts can be used in linear models to compare groups of treatments to one another. For example,
if you were testing four curricula and wanted to compare the effect of One and Two vs. Three and Four,
you could accomplish this with a single-degree-of freedom contrast.

“Contrasts in Linear Models” in Mangiafico, S.S. 2015. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/h_01.html.

Linear plateau and quadratic plateau models

Linear plateau and quadratic plateau models are segmented models for bivariate data, typically where
the y-value increases with an increase in x to some critical value, beyond which the y-value ceases to
increase. The statistics of interest include the critical value (the x-value above which there is no
further increase in y), and the plateau value (the statistically highest value that y reaches).

For another example of these models, see the “Use the best model” section of the Basic Plots chapter.

Also included in this section is an example of curvilinear regression, with a Mitscherlich–Bray model
fit to the same data.

Examples of linear plateau and quadratic plateau models

Input = ("
Instructor Grade Weight Calories Sodium Score
'Brendon Small' 6 43 2069 1287 77
'Brendon Small' 6 41 1990 1164 76
'Brendon Small' 6 40 1975 1177 76
'Brendon Small' 6 44 2116 1262 84

574
ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Brendon Small' 6 45 2161 1271 86

'Brendon Small' 6 44 2091 1222 87
'Brendon Small' 6 48 2236 1377 90
'Brendon Small' 6 47 2198 1288 78
'Brendon Small' 6 46 2190 1284 89
'Jason Penopolis' 7 45 2134 1262 76
'Jason Penopolis' 7 45 2128 1281 80
'Jason Penopolis' 7 46 2190 1305 84
'Jason Penopolis' 7 43 2070 1199 68
'Jason Penopolis' 7 48 2266 1368 85
'Jason Penopolis' 7 47 2216 1340 76
'Jason Penopolis' 7 47 2203 1273 69
'Jason Penopolis' 7 43 2040 1277 86
'Jason Penopolis' 7 48 2248 1329 81
'Melissa Robins' 8 48 2265 1361 67
'Melissa Robins' 8 46 2184 1268 68
'Melissa Robins' 8 53 2441 1380 66
'Melissa Robins' 8 48 2234 1386 65
'Melissa Robins' 8 52 2403 1408 70
'Melissa Robins' 8 53 2438 1380 83
'Melissa Robins' 8 52 2360 1378 74
'Melissa Robins' 8 51 2344 1413 65
'Melissa Robins' 8 51 2351 1400 68
'Paula Small' 9 52 2390 1412 78
'Paula Small' 9 54 2470 1422 62
'Paula Small' 9 49 2280 1382 61
'Paula Small' 9 50 2308 1410 72
'Paula Small' 9 55 2505 1410 80
'Paula Small' 9 52 2409 1382 60
'Paula Small' 9 53 2431 1422 70
'Paula Small' 9 56 2523 1388 79
'Paula Small' 9 50 2315 1404 71
'Coach McGuirk' 10 52 2406 1420 68
'Coach McGuirk' 10 58 2699 1405 65
'Coach McGuirk' 10 57 2571 1400 64
'Coach McGuirk' 10 52 2394 1420 69
'Coach McGuirk' 10 55 2518 1379 70
'Coach McGuirk' 10 52 2379 1393 61
'Coach McGuirk' 10 59 2636 1417 70
'Coach McGuirk' 10 54 2465 1414 59
'Coach McGuirk' 10 54 2479 1383 61
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

### Check the data frame

library(psych)

575
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Linear plateau model

The nls function in the native stats package can fit nonlinear and curvilinear functions. To accomplish
this, a function—linplat, here—will be defined with the x and y variables (Calories and Sodium) along
with parameters (a, b, and clx). It is the goal of the nls function to find the best fit values for these
parameters for these data.

The code below starts with an attempt to find reasonable starting values for the parameters. If these
starting values don’t yield appropriate results, other starting values can be tried.

The fitted parameters a, b, and clx designate the best fit intercept, slope, and critical x value. The
plateau value is calculated as a + b * clx.
### Find reasonable initial values for parameters

fit.lm = lm(Sodium ~ Calories, data=Data)

a.ini = fit.lm$coefficients[1]
b.ini = fit.lm$coefficients[2]
clx.ini = mean(Data$Calories)

### Define linear plateau function

linplat = function(x, a, b, clx)

{ifelse(x < clx, a + b * x,
a + b * clx)}

### Find best fit parameters

model = nls(Sodium ~ linplat(Calories, a, b, clx),

data = Data,
start = list(a = a.ini,
b = b.ini,
clx = clx.ini),
trace = FALSE,
nls.control(maxiter = 1000))

summary(model)

576
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Parameters:
Estimate Std. Error t value Pr(>|t|)
a -128.85535 116.10819 -1.11 0.273
b 0.65768 0.05343 12.31 1.6e-15 ***
clx 2326.51521 16.82086 138.31 < 2e-16 ***

p-value and pseudo R-squared

### Define null model

nullfunct = function(x, m){m}

m.ini = mean(Data$Sodium)

null = nls(Sodium ~ nullfunct(Calories, m),

data = Data,
start = list(m = m.ini),
trace = FALSE,
nls.control(maxiter = 1000))

### Find p-value and pseudo R-squared

library(rcompanion)

nagelkerke(model,
null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.195417
Cox and Snell (ML) 0.892004
Nagelkerke (Cragg and Uhler) 0.892014

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -50.077 100.15 1.785e-22

Determine confidence intervals for parameter estimates

library(nlstools)

confint2(model,
level = 0.95)

2.5 % 97.5 %
a -363.1711583 105.4604611
b 0.5498523 0.7654999
clx 2292.5693351 2360.4610904

577
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(nlstools)

Boot = nlsBoot(model)

summary(Boot)

------
Bootstrap statistics
Estimate Std. error
a -143.5189740 111.18292997
b 0.6643267 0.05132983
clx 2326.8124177 16.71482729

------
Median of bootstrap estimates and percentile confidence intervals
Median 2.5% 97.5%
a -143.5474796 -376.3224461 73.437548
b 0.6638483 0.5652069 0.771089
clx 2327.1279329 2292.5244830 2363.929178

Plot data with best fit line

library(rcompanion)

plotPredy(data = Data,
x = Calories,
y = Sodium,
model = model,
xlab = "Calories per day",
ylab = "Sodium intake per day")

A plot of daily sodium intake vs. daily caloric intake for students. Best fit
linear plateau model is shown. Critical x value = 2327 calories, and Plateau y =
1401 mg. p < 0.0001. Pseudo R-squared (Nagelkerke) = 0.892.

578
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plots of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model),
residuals(model))

Quadratic plateau model

A quadratic plateau model is similar to a linear plateau model, except that the linear segment is
replaced with a quadratic function.

The fitted parameters a, b, and clx designate the best fit intercept, linear coefficient, and critical x value.
The quadratic coefficient is calculated as –0.5 * b / clx. The plateau value is calculated as a + b * clx –
0.5 * b * clx.

579
ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Find reasonable initial values for parameters

fit.lm = lm(Sodium ~ Calories, data=Data)

a.ini = fit.lm$coefficients[1]
b.ini = fit.lm$coefficients[2]
clx.ini = mean(Data$Calories)

### Define quadratic plateau function

quadplat = function(x, a, b, clx) {

ifelse(x < clx, a + b * x + (-0.5*b/clx) * x * x,
a + b * clx + (-0.5*b/clx) * clx * clx)}

### Find best fit parameters

model = nls(Sodium ~ quadplat(Calories, a, b, clx),

data = Data,
start = list(a = a.ini,
b = b.ini,
clx = clx.ini),
trace = FALSE,
nls.control(maxiter = 1000))

summary(model)

Parameters:
Estimate Std. Error t value Pr(>|t|)
a -4217.7196 935.1139 -4.510 5.13e-05 ***
b 4.4920 0.8251 5.444 2.49e-06 ***
clx 2504.4059 48.9679 51.144 < 2e-16 ***

p-value and pseudo R-squared

### Define null model

nullfunct = function(x, m){m}

m.ini = mean(Data$Sodium)

null = nls(Sodium ~ nullfunct(Calories, m),

data = Data,
start = list(m = m.ini),
trace = FALSE,
nls.control(maxiter = 1000))

### Find p-value and pseudo R-squared

library(rcompanion)

580
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

nagelkerke(model,
null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.175609
Cox and Snell (ML) 0.864674
Nagelkerke (Cragg and Uhler) 0.864683

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -45.001 90.003 2.8583e-20

Determine confidence intervals for parameter estimates

library(nlstools)

confint2(model,
level = 0.95)

2.5 % 97.5 %
a -6104.855900 -2330.583368
b 2.826818 6.157185
clx 2405.584577 2603.227200

library(nlstools)

Boot = nlsBoot(model)

summary(Boot)

------
Bootstrap statistics
Estimate Std. error
a -4294.518746 956.6204011
b 4.560459 0.8448001
clx 2510.154733 53.3134728

------
Median of bootstrap estimates and percentile confidence intervals
Median 2.5% 97.5%
a -4250.030936 -6311.47965 -2546.971334
b 4.515874 3.03532 6.359601
clx 2504.795426 2425.75206 2626.813217

Plot data with best fit line

library(rcompanion)

plotPredy(data = Data,

581
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

x = Calories,
y = Sodium,
model = model,
xlab = "Calories per day",
ylab = "Sodium intake per day")

A plot of daily sodium intake vs. daily caloric intake for students. Best fit
quadratic plateau model is shown. Critical x value = 2504 calories, and Plateau
y = 1403 mg. p < 0.0001. Pseudo R-squared (Nagelkerke) = 0.865.

Plots of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

582
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plot(fitted(model),
residuals(model))

Mitscherlich–Bray model
This model is an example of curvilinear regression, specifically with a general exponential model. The
model here is specified as y = a – be–cx.

In the function below, the constant D is set to the minimum of Calories + 1. This will be subtracted
from the x values because the numerically large values for Calories make the fitting of the exponential
function difficult without doing this subtraction.
### Find reasonable initial values for parameters

a.ini = 1400
b.ini = -120
c.ini = 0.01

### Set a constant to adjust x values

D = 1974

### Define exponential function

expon = function(x, a, b, c) {
a + b * exp(-1 * c * (x - D))}

### Find best fit parameters

model = nls(Sodium ~ expon(Calories, a, b, c),

data = Data,
start = list(a = a.ini,
b = b.ini,
c = c.ini),
trace = FALSE,
nls.control(maxiter = 10000))

583
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

summary(model)

Parameters:
Estimate Std. Error t value Pr(>|t|)
a 1.450e+03 2.309e+01 62.790 < 2e-16 ***
b -2.966e+02 2.142e+01 -13.846 < 2e-16 ***
c 3.807e-03 7.973e-04 4.776 2.2e-05 ***

p-value and pseudo R-squared

### Define null model

nullfunct = function(x, m){m}

m.ini = mean(Data$Sodium)

null = nls(Sodium ~ nullfunct(Calories, m),

data = Data,
start = list(m = m.ini),
trace = FALSE,
nls.control(maxiter = 1000))

### Find p-value and pseudo R-squared

library(rcompanion)

nagelkerke(model,
null)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.162515
Cox and Snell (ML) 0.842908
Nagelkerke (Cragg and Uhler) 0.842918

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -41.646 83.292 8.1931e-19

Determine confidence intervals for parameter estimates

library(nlstools)

confint2(model,
level = 0.95)

2.5 % 97.5 %
a 1.403060e+03 1.496244e+03
b -3.398171e+02 -2.533587e+02
c 2.198512e-03 5.416355e-03

584
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(nlstools)

Boot = nlsBoot(model)

summary(Boot)

------
Bootstrap statistics
Estimate Std. error
a 1.453109e+03 2.522906e+01
b -3.005815e+02 2.244473e+01
c 3.850955e-03 7.832523e-04

------
Median of bootstrap estimates and percentile confidence intervals
Median 2.5% 97.5%
a 1.449234e+03 1.415464e+03 1.513570e+03
b -2.998719e+02 -3.474264e+02 -2.566514e+02
c 3.827519e-03 2.390379e-03 5.410459e-03

Plot data with best fit line

library(rcompanion)

plotPredy(data = Data,
x = Calories,
y = Sodium,
model = model,
xlab = "Calories per day",
ylab = "Sodium intake per day")

A plot of daily sodium intake vs. daily caloric intake for students. Best fit
exponential model is shown. p < 0.0001. Pseudo R-squared (Nagelkerke) = 0.843.

585
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Plots of residuals
x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

plot(fitted(model),
residuals(model))

Better functions to fit nonlinear regression

Finding reasonable starting values for some functions can be difficult, and the nls function may not find
a solution for certain functions if starting values are not close to their best values. One issue is that the
Gauss–Newton method that nls uses will fail in some situations.

586
 ADVANCED PARAMETRIC METHODS SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Typically, I will put my data into a spreadsheet and make a plot of the function I am trying to fit. I can
then change parameter values and watch the curve change until I get the curve reasonably close to the
data.

The package minpack.lm has a function nlsLM, which uses the Levenberg–Marquardt method. It is
more successful at fitting parameters for difficult functions when the initial values for parameters are
poor. An example of using this package is shown in the “Fitting curvilinear models with the
minpack.lm package” below.

(The package nlmrt uses the Marquardt–Nash method, which is also tolerant of difficult functions and
poor starting values for parameters. However, at the time of writing, this package is not available for
newer versions of R.)

Fitting curvilinear models with the minpack.lm package

For a plot of the best fit line and plots of residuals, see the previous section of this chapter.
library(minpack.lm)

### Define exponential function

expon = function(x, a, b, c) {
a + b * exp(-1 * c * (x - D))}

### Find best fit parameters

model = nlsLM(Sodium ~ expon(Calories, a, b, c),

data = Data,
start = list(a = 1400,
b = -1,
c = 0.1))

summary(model)

Parameters:
Estimate Std. Error t value Pr(>|t|)
a 1.450e+03 2.309e+01 62.789 < 2e-16 ***
b -2.966e+02 2.142e+01 -13.846 < 2e-16 ***
c 3.807e-03 7.972e-04 4.776 2.2e-05 ***

Cate–Nelson analysis

Cate–Nelson analysis is a useful approach for some bivariate data that don’t fit neatly into linear
model, linear–plateau model, or curvilinear model. The analysis divides the data into two populations:
those x values that are likely to have high y values, and those x values that are likely to have low y
values.

For more discussion and examples, see:

587
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

“Cate–Nelson Analysis” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological
Statistics, version 1.09. rcompanion.org/rcompanion/h_02.html.

library(rcompanion)

cateNelson(x = Data$Calories,
y = Data$Sodium,
plotit=TRUE,
hollow=TRUE,
xlab="Calories per day",
ylab="Sodium intake, mg per day",
trend="positive",
clx=1,
cly=1,
xthreshold=0.10,
ythreshold=0.15)

Final model:

n CLx SS CLy Q.I Q.II Q.III Q.IV Fisher.p.value

1 45 2209.5 190716.1 1317 0 30 0 15 2.899665e-12

A plot of daily sodium intake vs. daily caloric intake for students, with a Cate–
Nelson analysis applied. Students with a Caloric intake greater than 2210
calories tended to have a sodium intake greater than 1317 mg per day.

Transforming Data
Transforming data is one step in addressing data that do not fit model assumptions, and is also used to
coerce different variables to have similar distributions. Before transforming data, see the “Steps to
handle violations of assumption” section in the Assessing Model Assumptions chapter.
588
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Transforming data

Most parametric tests require that residuals be normally distributed and that the residuals be
homoscedastic.

One approach when residuals fail to meet these conditions is to transform one or more variables to
better follow a normal distribution. Often, just the dependent variable in a model will need to be
transformed. However, in complex models and multiple regression, it is sometimes helpful to
transform both dependent and independent variables that deviate greatly from a normal distribution.

There is nothing illicit in transforming variables, but you must be careful about how the results from
analyses with transformed variables are reported. For example, looking at the turbidity of water
across three locations, you might report, “Locations showed a significant difference in log-transformed
turbidity.” To present means or other summary statistics, you might present the mean of transformed
values, or back transform means to their original units.

Some measurements in nature are naturally normally distributed. Other measurements are naturally
log-normally distributed. These include some natural pollutants in water: There may be many low
values with fewer high values and even fewer very high values.

For right-skewed data—tail is on the right, positive skew—, common transformations include square
root, cube root, and log.

For left-skewed data—tail is on the left, negative skew—, common transformations include square
root (constant – x), cube root (constant – x), and log (constant – x).

Because log (0) is undefined—as is the log of any negative number—, when using a log transformation,
a constant should be added to all values to make them all positive before transformation. It is also
sometimes helpful to add a constant when using other transformations.

Another approach is to use a general power transformation, such as Tukey’s Ladder of Powers or a
Box–Cox transformation. These determine a lambda value, which is used as the power coefficient to
transform values. X.new = X ^ lambda for Tukey, and X.new = (X ^ lambda – 1) / lambda for Box–Cox.

The transformTukey function in the rcompanion package finds the lambda which makes a single vector
of values—that is, one variable—as normally distributed as possible with a simple power
transformation.

The Box–Cox procedure is included in the MASS package with the function boxcox. It uses a log-
likelihood procedure to find the lambda to use to transform the dependent variable for a linear model
(such as an ANOVA or linear regression). It can also be used on a single vector.

Packages used in this chapter

The packages used in this chapter include:

589
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• car
• MASS
• rcompanion

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("car")}
if(!require(MASS)){install.packages("MASS")}
if(!require(rcompanion)){install.packages("rcompanion")}

Example of transforming skewed data

This example uses hypothetical data of river water turbidity. Turbidity is a measure of how cloudy
water is due to suspended material in the water. Water quality parameters such as this are often
naturally log-normally distributed: values are often low, but are occasionally high or very high.

The first plot is a histogram of the Turbidity values, with a normal curve superimposed. Looking at the
gray bars, this data is skewed strongly to the right (positive skew), and looks more or less log-normal.
The gray bars deviate noticeably from the red normal curve.

The second plot is a normal quantile plot (normal Q–Q plot). If the data were normally distributed, the
points would follow the red line fairly closely.

Turbidity = c(1.0, 1.2, 1.1, 1.1, 2.4, 2.2, 2.6, 4.1, 5.0, 10.0, 4.0, 4.1, 4.2, 4.1,
5.1, 4.5, 5.0, 15.2, 10.0, 20.0, 1.1, 1.1, 1.2, 1.6, 2.2, 3.0, 4.0, 10.5)

library(rcompanion)

plotNormalHistogram (Turbidity)

qqnorm(Turbidity,
ylab="Sample Quantiles for Turbidity")

590
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

qqline(Turbidity,
col="red")

Square root transformation

Since the data is right-skewed, we will apply common transformations for right-skewed data: square
root, cube root, and log. The square root transformation improves the distribution of the data
somewhat.
T_sqrt = sqrt(Turbidity)

library(rcompanion)

plotNormalHistogram(T_sqrt)

Cube root transformation

The cube root transformation is stronger than the square root transformation.

591
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

T_cub = sign(Turbidity) * abs(Turbidity)^(1/3) # Avoid complex numbers

# for some cube roots

library(rcompanion)

plotNormalHistogram(T_cub)

Log transformation
The log transformation is a relatively strong transformation. Because certain measurements in nature
are naturally log-normal, it is often a successful transformation for certain data sets. While the
transformed data here does not follow a normal distribution very well, it is probably about as close as
we can get with these particular data.
T_log = log(Turbidity)

library(rcompanion)

plotNormalHistogram(T_log)

592
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Tukey’s Ladder of Powers transformation

The approach of Tukey’s Ladder of Powers uses a power transformation on a data set. For example,
raising data to a 0.5 power is equivalent to applying a square root transformation; raising data to a
0.33 power is equivalent to applying a cube root transformation.

Here, I use the transformTukey function, which performs iterative Shapiro–Wilk tests, and finds the
lambda value that maximizes the W statistic from those tests. In essence, this finds the power
transformation that makes the data fit the normal distribution as closely as possible with this type of
transformation.

Left skewed values should be adjusted with (constant – value), to convert the skew to right skewed,
and perhaps making all values positive. In some cases of right skewed data, it may be beneficial to add
a constant to make all data values positive before transformation. For large values, it may be helpful to
scale values to a more reasonable range.

In this example, the resultant lambda of –0.1 is slightly stronger than a log transformation, since a log
transformation corresponds to a lambda of 0.
### Some options require NCStats

library(rcompanion)

T_tuk =
transformTukey(Turbidity,
plotit=FALSE)

lambda W Shapiro.p.value
397 -0.1 0.935 0.08248

if (lambda > 0){TRANS = x ^ lambda}

if (lambda == 0){TRANS = log(x)}
if (lambda < 0){TRANS = -1 * x ^ lambda}

library(rcompanion)

plotNormalHistogram(T_tuk)

593
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Example of Tukey-transformed data in ANOVA

For an example of how transforming data can improve the distribution of the residuals of a parametric
analysis, we will use the same turbidity values, but assign them to three different locations.

Transforming the turbidity values to be more normally distributed, both improves the distribution of
the residuals of the analysis and makes a more powerful test, lowering the p-value.
Input =("
Location Turbidity
a 1.0
a 1.2
a 1.1
a 1.1
a 2.4
a 2.2
a 2.6
a 4.1
a 5.0
a 10.0
b 4.0
b 4.1
b 4.2
b 4.1
b 5.1
b 4.5
b 5.0
b 15.2
b 10.0
b 20.0
c 1.1
c 1.1
c 1.2
c 1.6
c 2.2
c 3.0
c 4.0
c 10.5

594
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

")

Data = read.table(textConnection(Input),header=TRUE)

Attempt ANOVA on un-transformed data

Here, even though the analysis of variance results in a significant p-value (p = 0.03), the residuals
deviate from the normal distribution enough to make the analysis invalid. The plot of the residuals vs.
the fitted values shows that the residuals are somewhat heteroscedastic, though not terribly so.
boxplot(Turbidity ~ Location,
data = Data,
ylab="Turbidity",
xlab="Location")

model = lm(Turbidity ~ Location,

data=Data)

library(car)

Anova(model, type="II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Location 132.63 2 3.8651 0.03447 *
Residuals 428.95 25

x = (residuals(model))

library(rcompanion)

plotNormalHistogram(x)

qqnorm(residuals(model),
ylab="Sample Quantiles for residuals")

595
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

qqline(residuals(model),
col="red")

plot(fitted(model),
residuals(model))

Transform data
### Some options require NCStats

library(rcompanion)

Data$Turbidity_tuk =
transformTukey(Data$Turbidity,
plotit=FALSE)

596
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

lambda W Shapiro.p.value
397 -0.1 0.935 0.08248

if (lambda > 0){TRANS = x ^ lambda}

if (lambda == 0){TRANS = log(x)}
if (lambda < 0){TRANS = -1 * x ^ lambda}

ANOVA with Tukey-transformed data

After transformation, the residuals from the ANOVA are closer to a normal distribution—although not
perfectly—, making the F-test more appropriate. In addition, the test is more powerful as indicated by
the lower p-value (p = 0.005) than with the untransformed data. The plot of the residuals vs. the fitted
values shows that the residuals are about as heteroscedastic as they were with the untransformed
data.
boxplot(Turbidity_tuk ~ Location,
data = Data,
ylab="Tukey-transformed Turbidity",
xlab="Location")

model = lm(Turbidity_tuk ~ Location,

data=Data)

library(car)

Anova(model, type="II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Location 0.052506 2 6.6018 0.004988 **
Residuals 0.099416 25

597
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

qqnorm(residuals(model),
ylab="Sample Quantiles for residuals")
qqline(residuals(model),
col="red")

plot(fitted(model),
residuals(model))

598
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Box–Cox transformation

The Box–Cox procedure is similar in concept to the Tukey Ladder of Power procedure described
above. However, instead of transforming a single variable, it maximizes a log-likelihood statistic for a
linear model (such as ANOVA or linear regression). It will also work on a single variable using a
formula of x ~ 1.

The Box–Cox procedure is available with the boxcox function in the MASS package. However, a few
steps are needed to extract the lambda value and transform the data set.

This example uses the same turbidity data.

Turbidity = c(1.0, 1.2, 1.1, 1.1, 2.4, 2.2, 2.6, 4.1, 5.0, 10.0, 4.0, 4.1, 4.2, 4.1,
5.1, 4.5, 5.0, 15.2, 10.0, 20.0, 1.1, 1.1, 1.2, 1.6, 2.2, 3.0, 4.0, 10.5)

library(rcompanion)

plotNormalHistogram(Turbidity)

599
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

qqnorm(Turbidity,
ylab="Sample Quantiles for Turbidity")
qqline(Turbidity,
col="red")

Box–Cox transformation for a single variable

library(MASS)

Box = boxcox(Turbidity ~ 1, # Transform Turbidity as a single vector

lambda = seq(-6,6,0.1) # Try values -6 to 6 by 0.1
)

Cox = data.frame(Box$x, Box$y) # Create a data frame with the results

Cox2 = Cox[with(Cox, order(-Cox$Box.y)),] # Order the new data frame by decreasing y

Cox2[1,] # Display the lambda with the greatest

# log likelihood

Box.x Box.y
59 -0.2 -41.35829

lambda = Cox2[1, "Box.x"] # Extract that lambda

T_box = (Turbidity ^ lambda - 1)/lambda # Transform the original data

library(rcompanion)

plotNormalHistogram(T_box)

600
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Example of Box–Cox transformation for ANOVA model

Input =("
Location Turbidity
a 1.0
a 1.2
a 1.1
a 1.1
a 2.4
a 2.2
a 2.6
a 4.1
a 5.0
a 10.0
b 4.0
b 4.1
b 4.2
b 4.1
b 5.1
b 4.5
b 5.0
b 15.2
b 10.0
b 20.0
c 1.1
c 1.1
c 1.2
c 1.6
c 2.2
c 3.0
c 4.0
c 10.5
")

Data = read.table(textConnection(Input),header=TRUE)

601
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Attempt ANOVA on un-transformed data

model = lm(Turbidity ~ Location,
data=Data)

library(car)

Anova(model, type="II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Location 132.63 2 3.8651 0.03447 *
Residuals 428.95 25

x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

qqnorm(residuals(model),
ylab="Sample Quantiles for residuals")
qqline(residuals(model),
col="red")

602
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

plot(fitted(model),
residuals(model))

Transform data
library(MASS)

Box = boxcox(Turbidity ~ Location,

data = Data,
lambda = seq(-6,6,0.1)
)

Cox = data.frame(Box$x, Box$y)

Cox2 = Cox[with(Cox, order(-Cox$Box.y)),]

603
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cox2[1,]

lambda = Cox2[1, "Box.x"]

Data$Turbidity_box = (Data$Turbidity ^ lambda - 1)/lambda

boxplot(Turbidity_box ~ Location,
data = Data,
ylab="Box–Cox-transformed Turbidity",
xlab="Location")

Perform ANOVA and check residuals

model = lm(Turbidity_box ~ Location,
data=Data)

library(car)

Anova(model, type="II")

Anova Table (Type II tests)

Sum Sq Df F value Pr(>F)

Location 0.16657 2 6.6929 0.0047 **
Residuals 0.31110 25

x = residuals(model)

library(rcompanion)

plotNormalHistogram(x)

604
TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

qqnorm(residuals(model),
ylab="Sample Quantiles for residuals")
qqline(residuals(model),
col="red")

plot(fitted(model),
residuals(model))

605
 TRANSFORMING DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Conclusions

Both the Tukey’s Ladder of Powers principle as implemented by the transformTukey function and the
Box–Cox procedure were successful at transforming a single variable to follow a more normal
distribution. They were also both successful at improving the distribution of residuals from a simple
ANOVA.

The Box–Cox procedure has the advantage of dealing with the dependent variable of a linear model,
while the transformTukey function works only for a single variable without considering other
variables. Because of this, the Box–Cox procedure may be advantageous when a relatively simple
model is considered. In cases where there are complex models or multiple regression, it may be
helpful to transform both dependent and independent variables independently.

606
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Analysis of Count Data and Percentage Data

Hermite and Poisson Regression for Count Data

Introduction

Count data
In general, common parametric tests like t-test and anova shouldn’t be used for count data. One
reason is technical in nature: that parametric analyses require continuous data. Count data is by its
nature discrete and is left-censored at zero. (That is, usually counts can’t be less than zero.)

A second reason is more practical in nature. Count data are often highly skewed, and often produce
skewed residuals if a parametric approach is attempted. In this case, the hypothesis tests will not be
accurate.

For further discussion, see the “Count data may not be appropriate for common parametric tests”
section in the Introduction to Parametric Tests chapter.

Regression approaches for count data

The most common regression approach for handling count data is probably Poisson regression.
However, Poisson regression makes assumptions about the distribution of the data that may not be
appropriate in all cases. Hermite regression is a more flexible approach, but at the time of writing
doesn’t have a complete set of support functions in R. Quasi-Poisson regression is also flexible with
data assumptions, but also but at the time of writing doesn’t have a complete set of support functions
in R. Negative binomial regression allows for overdispersion in data; and zero-inflated regression is
useful when there are a high proportion of zero counts in the data.

Cautionary note
Note that model assumptions and pitfalls of these regression techniques are not discussed in depth
here. The reader is urged to understand the assumptions of this kind of modeling before proceeding.

Generalized linear regression

Poisson, Hermite, and related regression approaches are a type of generalized linear model. This
should not be confused with general linear model, which is implemented with the lm function.
Generalized linear models are implemented with the glm function or other functions.

Generalized linear models are used when the dependent variable is count, binary, multinomial, etc.
More information on using the glm function can be found by using help(glm) and help(family). For
examples of logistic regression, see the chapter Models for Nominal Data; the chapter Beta Regression
for Percent and Proportion Data; or Mangiafico (2015) in the “References” section. For a table of
common uses for family and link function in generalized linear models, see the Wikipedia article in the
“References” section for this chapter.

Packages used in this chapter

The packages used in this chapter include:

607
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

• pysch
• hermite
• lattice
• plyr
• boot
• DescTools
• ggplot2
• car
• multcompView
• lsmeans
• MASS
• pscl
• rcompanion
• robust

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(hermite)){install.packages("hermite")}
if(!require(lattice)){install.packages("lattice")}
if(!require(plyr)){install.packages("plyr")}
if(!require(boot)){install.packages("boot")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(ggplot2)){install.packages("ggplot2")}
if(!require(car)){install.packages("car")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(MASS)){install.packages("MASS")}
if(!require(pscl)){install.packages("pscl")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(robust)){install.packages("robust")}

Hermite regression

Hermite regression example

The generalized Hermite distribution is a more general distribution that can handle overdispersion or
multimodality (Moriña and others, 2015). This makes generalized Hermite regression a powerful and
flexible tool for modeling count data. It is implemented with the hermite package.

Fitting models with the hermite package can be somewhat difficult. One issue is that model fitting may
fail without some parameters being specified. Often specifying an appropriate value for the m option
will help.

A further difficulty with this approach is that, at the time writing, the package isn’t supported by the
anova function to compare models, the Anova function to test effects, or other useful functions like
lsmeans for factor effects. The function nagelkerkeHermite can be used to compare models to test
effects.

608
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

In this example, extension researchers have set up garden plots with different suites of plants, with
each suite identified as a level of the variable Garden below. In September, they counted the number of
monarch butterflies in each garden plot.
Input = ("
Garden Monarchs
A 0
A 4
A 2
A 2
A 0
A 6
A 0
A 0
B 5
B 9
B 7
B 5
B 7
B 5
B 9
B 5
C 10
C 14
C 12
C 12
C 10
C 16
C 10
C 10
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Garden = factor(Data$Garden,
levels=unique(Data$Garden))

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

609
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

rm(Input)

Hermite regression
Here, the hermite package is used to conduct hermite regression. The nagelkerkeHermite function is
used to test the effect of Garden by likelihood ratio test, comparing the full model to null model with
the Garden effect excluded. It also produces a pseudo R-squared value for the model by comparison to
a null model.

Here, the m=3 option is specified. Often the default m=NULL can be used. In this case, if the m value is
not specified, the function cannot complete the model fitting, and errors are produced. Using m=2
often works. Here, m=3 was used because it produced a model with a lower AIC than did the m=2
option.
library(hermite)

model = glm.hermite(Monarchs ~ Garden,

data = Data,
link = "log",
m=3)

summary(model)

### Determine p-value and pseudo R-squared for model

null = glm.hermite(Monarchs ~ 1,
data = Data,
link = "log",
m=3)

library(rcompanion)

nagelkerkeHermite(model, null)

$Models

Model: "glm.hermite, Monarchs ~ Garden, Data, log, 3"

Null: "glm.hermite, Monarchs ~ 1, Data, log, 3"

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.249943
Cox and Snell (ML) 0.766548
Nagelkerke (Cragg and Uhler) 0.768829

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -17.457 34.915 2.6204e-08

$AIC
AIC

610
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Model: 112.78
Null: 143.69

### Note that because the null model is the same as the full model
### without the Garden effect, this test can also be considered a
### test of the Garden effect, and the pseudo R-squared akin to
### a partial pseudo R-squared due to the effect of Garden.

Histograms
library(lattice)

histogram(~ Monarchs | Garden,

data=Data,
layout=c(1,3) # columns and rows of individual plots
)

Post-hoc analysis: Medians and confidence intervals

At the time of writing, the lsmeans package does not support post-hoc analysis of regressions produced
with the hermite package.

One imperfect approach for post-hoc analysis would be to examine median counts for treatments and
the confidence intervals of these medians. We can conclude that groups with non-overlapping 95%
confidence intervals for their medians are significantly different.

However, this approach does not represent any information learned from the Hermite regression.

A second issue is that, because the dependent variable is not continuous, the distribution of the
bootstrapped confidence intervals is not likely to be continuous, and so is may not be reliable.

611
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

To get confidence intervals for the medians for each group, we will use the groupwiseMedian function.
Here I used the percentile method for confidence intervals.
library(rcompanion)

Sum = groupwiseMedian(data=Data,
group="Garden",
var="Monarchs",
conf=0.95,
R=5000,
percentile=TRUE,
bca=FALSE,
digits=3)

Sum

Garden n Median Conf.level Percentile.lower Percentile.upper

1 A 8 1 0.95 0 4
2 B 8 6 0.95 5 8
3 C 8 11 0.95 10 14

### In this case, none of the confidence intervals overlap.

Plot of medians and confidence intervals

The data frame Sum created above will be passed to ggplot for plotting. At the end of the code,
annotate is used to add text to the plot to indicate which medians are significantly different from one
another.
library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Garden,
y = Median)) +
geom_errorbar(aes(ymin = Percentile.lower,
ymax = Percentile.upper),
width = 0.05,
size = 1) +
geom_point(shape = 15,
size = 5) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("Median count of monarch butterflies") +

annotate("text",
x = 1:3,
y = c(5, 10, 15),
label = c("Group 3", "Group 2", "Group 1"))

612
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Post-hoc analysis: Pairwise models

Another imperfect approach to post-hoc analysis would be to produce additional hermite regression
models which isolate groups to compare in a pairwise manner.
Data1 = subset(Data,
Garden=="A" | Garden=="B")
Data2 = subset(Data,
Garden=="A" | Garden=="C")
Data3 = subset(Data,
Garden=="B" | Garden=="C")

### Pairwise regressions

library(hermite)

model1 = glm.hermite(Monarchs ~ Garden,

data = Data1,
link = "log",
m=3)

null1 = glm.hermite(Monarchs ~ 1,
data = Data1,
link = "log",
m=3)

library(rcompanion)

nagelkerkeHermite(model1,
null1)

613
HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-1 -6.7312 13.462 0.00024339

library(hermite)

model2 = glm.hermite(Monarchs ~ Garden,

data = Data2,
link = "log",
m=3)

null2 = glm.hermite(Monarchs ~ 1,
data = Data2,
link = "log",
m=3)

library(rcompanion)

nagelkerkeHermite(model2,
null2)

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-1 -13.42 26.84 2.2099e-07

library(hermite)

model3 = glm.hermite(Monarchs ~ Garden,

data = Data3,
link = "log",
m=3)

null3 = glm.hermite(Monarchs ~ 1,
data = Data3,
link = "log",
m=3)

library(rcompanion)

nagelkerkeHermite(model3,
null3)

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-1 -6.0386 12.077 0.00051042

p.value = c(0.00024339, 2.2099e-07, 0.00051042)

p.adj = p.adjust(p.value,

614
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

method = "fdr")

p.adj

[1] 3.65085e-04 6.62970e-07 5.10420e-04

Poisson regression example

Poisson regression makes certain assumptions about the relationship between the mean and the
dispersion of the dependent variable. Because this assumption may not be met for all data sets,
Poisson regression may not be recommended for routine use. Particularly, classic Poisson regression
should be avoided if there is overdispersion in the data or if there are several zero counts in the
dependent variable.

An alternate approach for data with overdispersion is negative binomial regression.

An alternative approach for data with many zeros is zero-inflated Poisson regression.

For further discussion, see the “Count data may not be appropriate for common parametric tests”
section in the Introduction to Parametric Tests chapter.

Note that model assumptions and pitfalls of this approach are not discussed here. The reader is urged
to understand the assumptions of this kind of modeling before proceeding.
model.p = glm(Monarchs ~ Garden,
data=Data,
family="poisson")

library(car)

Anova(model.p,
type="II",
test="LR")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Garden 66.463 2 3.697e-15 ***

library(rcompanion)

nagelkerke(model.p)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.387929
Cox and Snell (ML) 0.937293
Nagelkerke (Cragg and Uhler) 0.938037

$Likelihood.ratio.test

615
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Df.diff LogLik.diff Chisq p.value

-2 -33.231 66.463 3.6967e-15

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model.p,
pairwise ~ Garden,
adjust="tukey") ### Tukey adjustment for comparisons

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey adjustment for multiple comparisons

Garden lsmean SE df asymp.LCL asymp.UCL .group

A 0.5596158 0.2672450 NA -0.07849527 1.197727 a
B 1.8718022 0.1386750 NA 1.54068251 2.202922 b
C 2.4638532 0.1031421 NA 2.21757688 2.710130 c

Results are given on the log (not the response) scale.

Confidence level used: 0.95
Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
Tests are performed on the log scale
significance level used: alpha = 0.05

### Note that estimates are on log scale

Negative binomial regression example

Negative binomial regression is similar in application to Poisson regression, but allows for
overdispersion in the dependent count variable.

This example will use the glm.nb function in the MASS package. The Anova function in the car package
will be used for an analysis of deviance, and the nagelkerke function will be used to determine a p-
value and pseudo R-squared value for the model. Post-hoc analysis can be conducted with the lsmeans
package.

Note that model assumptions and pitfalls of this approach are not discussed here. The reader is urged
to understand the assumptions of this kind of modeling before proceeding.
library(MASS)

model.nb = glm.nb(Monarchs ~ Garden,

data=Data,
control = glm.control(maxit=10000))

616
HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(car)

Anova(model.nb,
type="II",
test="LR")

Analysis of Deviance Table (Type II tests)

LR Chisq Df Pr(>Chisq)
Garden 66.464 2 3.694e-15 ***

Library(rcompanion)

nagelkerke(model.nb)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.255141
Cox and Snell (ML) 0.776007
Nagelkerke (Cragg and Uhler) 0.778217

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-2 -17.954 35.907 1.5952e-08

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model.nb,
pairwise ~ Garden,
adjust="tukey") ### Tukey adjustment

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey adjustment for multiple comparisons

Garden lsmean SE df asymp.LCL asymp.UCL .group

A 0.5596158 0.2672612 NA -0.078534 1.197766 a
B 1.8718022 0.1386750 NA 1.540683 2.202922 b
C 2.4638532 0.1031421 NA 2.217577 2.710130 c

Results are given on the log (not the response) scale.

Confidence level used: 0.95
Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
Tests are performed on the log scale
significance level used: alpha = 0.05

### Note that estimates are on log scale

617
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Zero-inflated regression example

Zero-inflated regression is similar in application to Poisson regression, but allows for an abundance of
zeros in the dependent count variable.

This example will use the zeroinfl function in the pscl package. The Anova function in the car package
will be used for an analysis of deviance, and the nagelkerke function will be used to determine a p-
value and pseudo R-squared value for the model. Post-hoc analysis can be conducted with the lsmeans
package.
library(pscl)

model.zi = zeroinfl(Monarchs ~ Garden,

data = Data,
dist = "poisson")

### dist = "negbin" may be used

summary(model.zi)

Call:
zeroinfl(formula = Monarchs ~ Garden | Garden, data = Data, dist = "poisson")

Count model coefficients (poisson with log link):

Estimate Std. Error z value Pr(>|z|)
(Intercept) 1.2182 0.2847 4.278 1.89e-05 ***
GardenB 0.6536 0.3167 2.064 0.039 *
GardenC 1.2457 0.3029 4.113 3.90e-05 ***

Zero-inflation model coefficients (binomial with logit link):

Estimate Std. Error z value Pr(>|z|)
(Intercept) -7.046e-02 7.363e-01 -0.096 0.924
GardenB -2.057e+01 1.071e+04 -0.002 0.998
GardenC -2.057e+01 1.071e+04 -0.002 0.998

### Note that there are separate coefficients for the

### Poisson part of the analysis and for the zero-inflation part.

library(car)

Anova(model.zi,
type="II",
test="Chisq")

Analysis of Deviance Table (Type II tests)

Df Chisq Pr(>Chisq)
Garden 2 23.914 6.414e-06 ***

618
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(rcompanion)

nagelkerke(model.zi)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.284636
Cox and Snell (ML) 0.797356
Nagelkerke (Cragg and Uhler) 0.800291

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-4 -19.156 38.311 9.6649e-08

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model.zi,
pairwise ~ Garden,
adjust="tukey") ### Tukey adjustment

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters, ### Use lower-case letters for .group
adjust="tukey") ### Tukey adjustment for multiple comparisons

Garden lsmean SE df asymp.LCL asymp.UCL .group

A 1.75 0.7586301 NA -0.06140972 3.561410 a
B 6.50 0.9013877 NA 4.34772229 8.652278 b
C 11.75 1.2119199 NA 8.85625304 14.643747 c

Confidence level used: 0.95

Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

### Note, lsmeans are on the original measurement scale

Robust Poisson regression example

Robust Poisson regression is robust to outliers in the dependent variable.

This example uses the glmRob function in the robust package. The anova function can be used to
conduct an analysis of deviance. The p-value for the model can be found by comparing the model to a
null model. However, at the time of writing, I don’t know of any way to determine AIC or pseudo R-
squared for the model.

At the time of writing, the glmRob function can only use the Poisson and binomial families of models.

619
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

An alternate method is the glmrob function in the robustbase package.

library(robust)

model.rob = glmRob(Monarchs ~ Garden,

data = Data,
family = "poisson")

anova(model.rob, test="Chisq")

Df Deviance Resid. Df Resid. Dev Pr(>Chi)

NULL NA NA 23 430.19850 NA
Garden 2 400.9221 21 29.27641 3.567693e-63

model.rob.null = glmRob(Monarchs ~ 1,
data = Data,
family = "poisson")

anova(model.rob.null, model.rob, test="Chisq")

Terms Resid. Df Resid. Dev Test Df Deviance Pr(>Chi)

1 1 23 95.12606 NA NA NA
2 Garden 21 29.27641 2 65.84965 5.536815e-11

Quasi-Poisson regression

Quasi-Poisson regression is useful since it has a variable dispersion parameter, so that it can model
over-dispersed data. It may be better than negative binomial regression in some circumstances
(Verhoef and Boveng. 2007).

At the time of writing, Quasi-Poisson regression doesn’t have complete set of support functions in R.
Using the quasipoisson family option in the glm function, the results will have the same parameter
coefficients as with the poisson option, but the inference statistics are adjusted in the summary
function. The Anova function in the car package can be used for an analysis of deviance table, and the
lsmeans package can be used for post-hoc comparisons. Since the model doesn’t produce a log-
likelihood value, I don’t know a way to produce a p-value for the mode, for a pseudo R-squared value
for the model.
.
model.qp = glm(Monarchs ~ Garden,
data=Data,
family="quasipoisson")

library(car)

Anova(model.qp,
type="II",
test="LR")

620
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Analysis of Deviance Table (Type II tests)

Response: Monarchs
LR Chisq Df Pr(>Chisq)
Garden 52.286 2 4.429e-12 ***

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model.qp,
pairwise ~ Garden,
adjust="tukey")

leastsquare

cld(leastsquare,
alpha=0.05,
Letters=letters,
adjust="tukey")

A 0.5596158 0.3013057 NA -0.1598233 1.279055 a

B 1.8718022 0.1563493 NA 1.4984809 2.245123 b
C 2.4638532 0.1162877 NA 2.1861887 2.741518 c

Results are given on the log (not the response) scale.

Confidence level used: 0.95
Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

Optional code for chi-square goodness-of-fit test

An alternative approach to handling count data is to sum up the counts for treatments, and use a chi-
square test or related test. Here, a chi-square goodness-of-fit test is used to see if counts differ from
“expected” equal proportions.

Omnibus test
Tabla = xtabs(Monarchs ~ Garden,
data = Data)

Tabla

Garden
A B C
14 52 94

chisq.test(Tabla)

621
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Chi-squared test for given probabilities

X-squared = 60.05, df = 2, p-value = 9.127e-14

Post-hoc chi-square tests

Garden.A = sum(Data$Monarchs[Data$Garden=="A"])
Garden.B = sum(Data$Monarchs[Data$Garden=="B"])
Garden.C = sum(Data$Monarchs[Data$Garden=="C"])

observed = c(Garden.A, Garden.B) # observed frequencies

expected = c(1/2, 1/2) # expected proportions

chisq.test(x = observed,
p = expected)

Chi-squared test for given probabilities

X-squared = 21.879, df = 1, p-value = 2.904e-06

observed = c(Garden.B, Garden.C) # observed frequencies

expected = c(1/2, 1/2) # expected proportions

chisq.test(x = observed,
p = expected)

Chi-squared test for given probabilities

X-squared = 12.082, df = 1, p-value = 0.0005091

observed = c(Garden.A, Garden.C) # observed frequencies

expected = c(1/2, 1/2) # expected proportions

chisq.test(x = observed,
p = expected)

Chi-squared test for given probabilities

X-squared = 59.259, df = 1, p-value = 1.382e-14

Optional analysis: Vuong test to compare Poisson, negative binomial, and zero-
inflated models

The Vuong test, implemented by the pscl package, can test two non-nested models. It works with
negbin, zeroinfl, and some glm model objects which are fitted to the same data.

The null hypothesis is that there is no difference in models. The function produces three tests, a “Raw”
test, an AIC-corrected, and a BIC-corrected, any of which could be used.

622
 HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

It has been suggested that the Vuong test not be used to test for zero-inflation (Wilson, 2015).

Define models
model.p = glm(Monarchs ~ Garden,
data=Data,
family="poisson")

library(MASS)

model.nb = glm.nb(Monarchs ~ Garden,

data=Data,
control = glm.control(maxit=10000))

library(pscl)

model.zi = zeroinfl(Monarchs ~ Garden,

data = Data,
dist = "poisson")

Vuong test
library(pscl)

vuong(model.p,
model.nb,
digits = 4)

Vuong Non-Nested Hypothesis Test-Statistic:

(test-statistic is asymptotically distributed N(0,1) under the
null that the models are indistinguishible)
-------------------------------------------------------------
Vuong z-statistic H_A p-value
Raw 0.03324988 model1 > model2 0.48674
AIC-corrected 0.03324988 model1 > model2 0.48674
BIC-corrected 0.03324988 model1 > model2 0.48674

### Positive Vuong z-statistic suggests that model 1 is superior,

### but, in this case, the difference is not significant,
### and the value of the statistic is probably too tiny to be meaningful.

vuong(model.p,
model.zi,
digits = 4)

Vuong Non-Nested Hypothesis Test-Statistic:

(test-statistic is asymptotically distributed N(0,1) under the
null that the models are indistinguishible)
-------------------------------------------------------------
Vuong z-statistic H_A p-value
Raw -1.4424725 model2 > model1 0.074585
AIC-corrected -0.4335210 model2 > model1 0.332318

623
HERMITE AND POISSON REGRESSION FOR COUNT DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

BIC-corrected 0.1607786 model1 > model2 0.436134

### Negative Vuong z-statistic suggests that model 2 is superior.

### If the Raw statistic is used, p = 0.07 gives some evidence
### that zi model is superior.

vuong(model.nb,
model.zi,
digits = 4)

Vuong Non-Nested Hypothesis Test-Statistic:

(test-statistic is asymptotically distributed N(0,1) under the
null that the models are indistinguishible)
-------------------------------------------------------------
Vuong z-statistic H_A p-value
Raw -1.4424725 model2 > model1 0.074585
AIC-corrected -0.4335210 model2 > model1 0.332318
BIC-corrected 0.1607786 model1 > model2 0.436134

### Negative Vuong z-statistic suggests that model 2 is superior.

### If the Raw statistic is used, p = 0.07 gives some evidence
### that zi model is superior.

References

Moriña, D., M. Higueras, P. Puig, and M. Oliveira. 2015. Generalized Hermite Distribution
Modelling with the R Package hermite. The R Journal 7(2):263–274. journal.r-
project.org/archive/2015-2/morina-higueras-puig-etal.pdf.

help(package="hermite")

library(hermite); ?glm.hermite

library(MASS); ?glm.nb

library(pscl); ?zeroinfl

library(pscl); ?vuong

“Simple Logistic Regression” in Mangiafico, S.S. 2015. An R Companion for the Handbook of
Biological Statistics, version 1.09. rcompanion.org/rcompanion/e_06.html.

"Generalized linear model: Link function". No date. Wikipedia. Retrieved 31 Jan. 2016.
en.wikipedia.org/wiki/Generalized_linear_model#Link_function.

Verhoef, J.M. and P.L. Boveng. 2007. Quasi-Poisson vs. negative binomial regression: How
should we model overdispersed count data? Ecology 88(11) 2766–2772.
http://fisher.utstat.toronto.edu/reid/sta2201s/QUASI-POISSON.pdf.

624
 BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Wilson, P. 2015. The Misuse of the Vuong Test for Non-Nested Models to Test for Zero-Inflation.
Economic Letters 127: 51–53. cybermetrics.wlv.ac.uk/paperdata/misusevuong.pdf

References for count data

Grace-Martin, K. No date. "Regression Models for Count Data". The Analysis Factor.
www.theanalysisfactor.com/regression-models-for-count-data/.

Grace-Martin, K. No date. " Zero-Inflated Poisson Models for Count Outcomes". The Analysis Factor.
www.theanalysisfactor.com/zero-inflated-poisson-models-for-count-outcomes/.

[IDRE] Institute for Digital Research and Education. 2015. “R Data Analysis Examples: Poisson
Regression”. UCLA. www.ats.ucla.edu/stat/r/dae/poissonreg.htm.

[IDRE] Institute for Digital Research and Education. 2015. “R Data Analysis Examples: Negative
Binomial Regression”. UCLA. www.ats.ucla.edu/stat/r/dae/nbreg.htm.

[IDRE] Institute for Digital Research and Education. 2015. “R Data Analysis Examples: Zero-Inflated
Poisson Regression”. UCLA. www.ats.ucla.edu/stat/r/dae/zipoisson.htm.

[IDRE] Institute for Digital Research and Education. 2015. “R Data Analysis Examples: Zero-Truncated
Poisson Regression”. UCLA. www.ats.ucla.edu/stat/r/dae/ztp.htm.

Beta Regression for Percent and Proportion

Data
Introduction

Proportion data
In general, common parametric tests like t-test and anova shouldn’t be used when the dependent
variable is proportion data, since proportion data is by its nature bound at 0 and 1, and is often not
normally distributed or homoscedastic.

Data of proportions, percentages, and rates can be thought of as falling into a few different categories.

Proportion data of discrete counts

Some proportion data is derived from discrete counts of “successes” and “failures”, where the
“successes” are divided by the total counts. The example below with passing and failing counts across
classes is an example of this. Each observation is a percentage from 0 to 100%, or a proportion from 0
to 1. This kind of data can be analyzed with beta regression or can be analyzed with logistic
regression.

625
 BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Proportion data that is inherently proportional

Other proportion data is inherently proportional, in that it’s not possible to count “successes” or
“failures”, but instead is derived, for example, by dividing one continuous variable by a given
denominator value. The sodium intake example below is an example of this. Another case of this kind
of proportion data is when a proportion is assessed by subjective measurement. For example, rating a
diseased lawn subjectively on the area dead, such as “this plot is 10% dead, and this plot is 20% dead”.
Each observation is a percentage from 0 to 100%, or a proportion from 0 to 1. This kind of data can be
analyzed with beta regression.

Rates with different numerators and denominators

Some rates are expressed with numerators and denominators of different measurements or units. For
example, the number of cases of a disease per 100,000 people or the number of televisions per
student’s home. In these cases, the values are not limited to between 0 and 1, and beta regression is
not appropriate.

If the numerator can be considered a count variable, Poisson regression or other methods for count
data are usually suggested. As a complication, often the denominator varies in value. For example, if
the count is disease occurrence in a city and the denominator is the population in a city. (Each city has
a different population.) Or if the numerator is the count of televisions in the home and the
denominator is the number of students in the home. In these cases, Poisson regression or related
methods are often recommended with an offset for the value in the denominator. These examples are
not explored further here, but an example model would be
glm(Count_of_televisions ~ Independent_variable +
offset(log(Number_of_students_in_home)),
family="poisson",
data=Data)

Beta regression
Beta regression can be conducted with the betareg function in the betareg package (Cribari-Neto and
Zeileis, 2010). With this function, the dependent variable varies between 0 and 1, but no observation
can equal exactly zero or exactly one. The model assumes that the data follow a beta distribution.

p-value and pseudo R-squared for the model

The summary function in betareg produces a pseudo R-squared value for the model, and the
recommended test for the p-value for the model is the lrtest function in the lmtest package.

The nagelkerke function in the rcompanion package also works with beta regression objects. The
likelihood ratio test there appears to work fine, but the results for pseudo R-squared may be squirrelly,
and probably should not be relied upon.

Analysis of deviance table

The Anova function in the car package will produce an anova-like table for beta regression objects.
The default “Chisq” type test is probably the best of available options for these models.

626
 BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Final note
Note that model assumptions and pitfalls of this approach are not discussed here. The reader is urged
to understand the assumptions of this kind of modeling before proceeding.

Packages used in this chapter

The packages used in this chapter include:

• pysch
• betareg
• car
• lmtest
• rcompanion
• multcompView
• lsmeans
• ggplot2

The following commands will install these packages if they are not already installed:
if(!require(psych)){install.packages("psych")}
if(!require(betareg)){install.packages("betareg")}
if(!require(car)){install.packages("car")}
if(!require(lmtest)){install.packages("lmtest")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(lsmeans)){install.packages("lsmeans")}
if(!require(ggplot2)){install.packages("ggplot2")}

Beta regression example with discrete counts

The following example uses counts of students passing or failing an exam, and asks if there is a
relationship between the proportion of passing students and Grade. Note that Grade is treated as a
numeric variable in this analysis.
Input = ("
Class Grade Pass Fail
'Bully Hill' 12 14 6
'Keuka Lake' 12 15 5
'Heron Hill' 11 18 2
'Castel Grisch' 11 10 10
'Red Newt' 10 17 3
'Finger Lakes' 10 9 11
'Bellview' 9 12 8
'Auburn Road' 9 8 12
'Balic' 8 10 10
'Cape May' 8 8 12
'Hawk Haven' 7 12 8
'Natali' 7 4 16
")

Data = read.table(textConnection(Input),header=TRUE)

627
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Create the proportion variable

Data$Proportion = Data$Pass / (Data$Pass + Data$Fail)

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Class Grade Pass Fail Proportion

1 Bully Hill 12 14 6 0.7
2 Keuka Lake 12 15 5 0.75
3 Heron Hill 11 18 2 0.9
4 Castel Grisch 11 10 10 0.5
... <NA> ... ... ... ...
9 Balic 8 10 10 0.5
10 Cape May 8 8 12 0.4
11 Hawk Haven 7 12 8 0.6
12 Natali 7 4 16 0.2

Beta regression example

library(betareg)

model.beta = betareg(Proportion ~ Grade,

data = Data)

library(car)

Anova(model.beta,
type="II")

Analysis of Deviance Table (Type II tests)

Response: Proportion
Df Chisq Pr(>Chisq)
Grade 1 7.3314 0.006776 **

628
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(lmtest)

lrtest(model.beta)

Likelihood ratio test

#Df LogLik Df Chisq Pr(>Chisq)

1 3 5.9273
2 2 2.9897 -1 5.8752 0.01536 *

summary(model.beta)

Pseudo R-squared: 0.3785

library(rcompanion)

plotPredy(data = Data,
y = Proportion,
x = Grade,
model = model.beta,
xlab = "Grade",
ylab = "Proportion passed")

plot(fitted(model.beta),
residuals(model.beta))

629
 BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### More diagnostic plots: plot(model.beta)

Alternate logistic regression

Trials = cbind(Data$Pass, Data$Fail) # Successes, Failures

model.log = glm(Trials ~ Grade,

data = Data,
family = binomial(link="logit"))

library(car)

Anova(model.log,
type="II",
test="Wald")

Analysis of Deviance Table (Type II tests)

Df Chisq Pr(>Chisq)
Grade 1 14.305 0.0001554 ***

library(rcompanion)

nagelkerke(model.log)

$Pseudo.R.squared.for.model.vs.null
Pseudo.R.squared
McFadden 0.190924
Cox and Snell (ML) 0.717212
Nagelkerke (Cragg and Uhler) 0.718174

630
 BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

$Likelihood.ratio.test
Df.diff LogLik.diff Chisq p.value
-1 -7.5784 15.157 9.8946e-05

library(rcompanion)

plotPredy(data = Data,
y = Percent,
x = Grade,
model = model.log,
type = "response", # Needed for logistic regression
xlab = "Grade",
ylab = "Proportion passing")

plot(fitted(model.beta),
residuals(model.beta))

### More diagnostic plots: plot(model.beta)

Beta regression example with inherently proportional data

This example revisits the data set from the chapter on two-way analysis of variance. Here, the
dependent variable Proportion is created by dividing daily student sodium intake by the US FDA
“upper safe limit” of 2300 mg. The rest of the analysis is analogous to that of a two-way ANOVA.
Input = ("
Instructor Supplement Sodium
'Brendon Small' A 1200
'Brendon Small' A 1400
'Brendon Small' A 1350
'Brendon Small' A 950
'Brendon Small' A 1400
'Brendon Small' B 1150
'Brendon Small' B 1300
'Brendon Small' B 1325
'Brendon Small' B 1425
'Brendon Small' B 1500
'Brendon Small' C 1250
'Brendon Small' C 1150
'Brendon Small' C 950
'Brendon Small' C 1150
'Brendon Small' C 1600
'Brendon Small' D 1300
'Brendon Small' D 1050
'Brendon Small' D 1300
'Brendon Small' D 1700
'Brendon Small' D 1300
'Coach McGuirk' A 1100
'Coach McGuirk' A 1200
'Coach McGuirk' A 1250
'Coach McGuirk' A 1050

631
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

'Coach McGuirk' A 1200

'Coach McGuirk' B 1250
'Coach McGuirk' B 1350
'Coach McGuirk' B 1350
'Coach McGuirk' B 1325
'Coach McGuirk' B 1525
'Coach McGuirk' C 1225
'Coach McGuirk' C 1125
'Coach McGuirk' C 1000
'Coach McGuirk' C 1125
'Coach McGuirk' C 1400
'Coach McGuirk' D 1200
'Coach McGuirk' D 1150
'Coach McGuirk' D 1400
'Coach McGuirk' D 1500
'Coach McGuirk' D 1200
'Melissa Robins' A 900
'Melissa Robins' A 1100
'Melissa Robins' A 1150
'Melissa Robins' A 950
'Melissa Robins' A 1100
'Melissa Robins' B 1150
'Melissa Robins' B 1250
'Melissa Robins' B 1250
'Melissa Robins' B 1225
'Melissa Robins' B 1325
'Melissa Robins' C 1125
'Melissa Robins' C 1025
'Melissa Robins' C 950
'Melissa Robins' C 925
'Melissa Robins' C 1200
'Melissa Robins' D 1100
'Melissa Robins' D 950
'Melissa Robins' D 1300
'Melissa Robins' D 1400
'Melissa Robins' D 1100
")

Data = read.table(textConnection(Input),header=TRUE)

### Order factors by the order in data frame

### Otherwise, R will alphabetize them

Data$Instructor = factor(Data$Instructor,
levels=unique(Data$Instructor))

Data$Supplement = factor(Data$Supplement,
levels=unique(Data$Supplement))

### Create proportion variable

Data$Proportion = Data$Sodium / 2300

632
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

### Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

Instructor Supplement Sodium Proportion

1 Brendon Small A 1200 0.52
2 Brendon Small A 1400 0.61
3 Brendon Small A 1350 0.59
4 Brendon Small A 950 0.41
... <NA> <NA> ... ...
57 Melissa Robins D 950 0.41
58 Melissa Robins D 1300 0.57
59 Melissa Robins D 1400 0.61
60 Melissa Robins D 1100 0.48

Beta regression
library(betareg)

model = betareg(Proportion ~ Instructor + Supplement + Instructor:Supplement,

data = Data)

library(car)

Anova(model,
type="II")

Analysis of Deviance Table (Type II tests)

Df Chisq Pr(>Chisq)
Instructor 2 14.9873 0.0005566 ***
Supplement 3 15.3766 0.0015216 **
Instructor:Supplement 6 1.1985 0.9769603

library(lmtest)

lrtest(model)

Likelihood ratio test

633
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

#Df LogLik Df Chisq Pr(>Chisq)

1 13 82.956
2 2 70.260 -11 25.392 0.007984 **

summary(model)

Pseudo R-squared: 0.3431

plot(fitted(model),
residuals(model))

### More diagnostic plots: plot(model)

library(multcompView)

library(lsmeans)

leastsquare = lsmeans(model,
pairwise ~ Instructor,
adjust = "tukey")

leastsquare

Sum = cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lowercase letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Sum

634
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Instructor lsmean SE df asymp.LCL asymp.UCL .group

Melissa Robins 0.4886613 0.01361908 NA 0.4561425 0.5211801 a
Coach McGuirk 0.5417084 0.01357579 NA 0.5092930 0.5741238 b
Brendon Small 0.5606374 0.01354438 NA 0.5282970 0.5929779 b

Results are averaged over the levels of: Supplement

Confidence level used: 0.95
Conf-level adjustment: sidak method for 3 estimates
P value adjustment: tukey method for comparing a family of 3 estimates
significance level used: alpha = 0.05

### Order data frame for plotting

Sum = Sum[order(factor(Sum$Instructor,
levels=c("Brendon Small",
"Coach McGuirk",
"Melissa Robins"))),]
Sum

Instructor lsmean SE df asymp.LCL asymp.UCL .group

Brendon Small 0.5606374 0.01354438 NA 0.5282970 0.5929779 b
Coach McGuirk 0.5417084 0.01357579 NA 0.5092930 0.5741238 b
Melissa Robins 0.4886613 0.01361908 NA 0.4561425 0.5211801 a

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Instructor,
y = lsmean)) +
geom_errorbar(aes(ymin = asymp.LCL,
ymax = asymp.UCL),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("LS mean proportion of sodium intake") +

annotate("text",
x = 1:3,
y = Sum$asymp.UCL + 0.008,
label = gsub(" ", "", Sum$.group))

635
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

leastsquare = lsmeans(model,
pairwise ~ Supplement,
adjust = "tukey")

leastsquare

Sum = cld(leastsquare,
alpha = 0.05,
Letters = letters, ### Use lowercase letters for .group
adjust = "tukey") ### Tukey-adjusted comparisons

Sum

### Order data frame for plotting

Sum = Sum[order(factor(Sum$Supplement,
levels=c("A", "B", "C", "D"))),]

Sum

636
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

library(ggplot2)

ggplot(Sum, ### The data frame to use.

aes(x = Supplement,
y = lsmean)) +
geom_errorbar(aes(ymin = asymp.LCL,
ymax = asymp.UCL),
width = 0.05,
size = 0.5) +
geom_point(shape = 15,
size = 4) +
theme_bw() +
theme(axis.title = element_text(face = "bold")) +
ylab("LS mean proportion of sodium intake") +

annotate("text",
x = 1:4,
y = Sum$asymp.UCL + 0.008,
label = gsub(" ", "", Sum$.group))

References

637
BETA REGRESSION FOR PERCENT AND PROPORTION DATA SUMMARY AND ANALYSIS OF EXTENSION EDUCATION PROGRAM
EVALUATION IN R

Cribari-Neto, F. and Zeileis, A. 2010. Beta Regression in R. Journal of Statistical Software 34(2).
www.jstatsoft.org/article/view/v034i02/v34i02.pdf.

638