Utah State University

DigitalCommons@USU

All Graduate Theses and Dissertations Graduate Studies

12-2019

Mental Health Awareness and Advocacy: Assessment Tool

Development and an Evaluation of a College-Based Curriculum
Ty B. Aller
Utah State University

Follow this and additional works at: https://digitalcommons.usu.edu/etd

Part of the Family, Life Course, and Society Commons

Recommended Citation
Aller, Ty B., "Mental Health Awareness and Advocacy: Assessment Tool Development and an Evaluation of
a College-Based Curriculum" (2019). All Graduate Theses and Dissertations. 7701.
https://digitalcommons.usu.edu/etd/7701

This Dissertation is brought to you for free and open

access by the Graduate Studies at
DigitalCommons@USU. It has been accepted for
inclusion in All Graduate Theses and Dissertations by an
authorized administrator of DigitalCommons@USU. For
more information, please contact
[email protected].
 MENTAL HEALTH AWARENESS AND ADVOCACY: ASSESSMENT TOOL

DEVELOPMENT AND AN EVALUATION OF A

COLLEGE-BASED CURRICULUM

by

Ty B. Aller

A dissertation submitted in partial fulfillment

of the requirements for the degree

of

DOCTOR OF PHILOSOPHY

In

Family and Human Development

Approved:

______________________ ____________________
Elizabeth Fauth, Ph.D. Scot Allgood, Ph.D.
Co-Major Professor Co-Major Professor

______________________ ____________________
Ryan Seedall, Ph.D. Josh Novak, Ph.D.
Committee Member Committee Member

______________________ ____________________
Gretchen Peacock, Ph.D. Richard S. Inouye, Ph.D.
Committee Member Vice Provost for Graduate Studies

UTAH STATE UNIVERSITY

Logan, Utah

2019
 ii

Copyright © Ty B. Aller 2019

All Rights Reserved

 iii

ABSTRACT

Mental Health Awareness and Advocacy: Assessment Tool Development and an

Evaluation of a College-Based Curriculum

by

Ty B. Aller, Doctor of Philosophy

Utah State University, 2019

Major Professors: Dr. Elizabeth Fauth & Dr. Scot Allgood

Department: Human Development and Family Science

This multi-paper dissertation consists of two studies related to mental health

literacy on a college campus. The purpose of study one was to create and evaluate the

Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT), which uses a

process-based approach to evaluate mental health literacy programs in a college-sample.

A sample of 296 college attending participants recruited from Amazon’s Mechanical

Turk was used to assess the psychometric properties of the MHAA-AT. Psychometric

properties of the MHAA-AT were examined through item response theory (declarative

knowledge items, only), exploratory factor analyses, and bivariate correlations. Results

indicate that the MHAA-AT is a sound measurement device and demonstrates

appropriate item, person, and trait characteristics on declarative knowledge items and

single factor structures on self-efficacy and behavior items. The results of study one also

demonstrates moderate to high reliability (internal consistency) and high levels of

construct validity. The MHAA-AT needs to be tested in other samples, but initial results
 iv

suggest that it is a quality assessment tool and appropriate for evaluating mental health

literacy programs in college samples. The purpose of study two was to create and

evaluate the effectiveness of the Mental Health Awareness and Advocacy (MHAA)

curriculum using a quasi-experimental design with a sample of 160 college students. The

MHAA curriculum is unique in that it is process-based and can be offered in multiple

course formats (both face-to-face and online) as part of a degree seeking academic

program. Results of study two suggest that the MHAA curriculum is associated with

improved outcomes in knowledge and self-efficacy related to mental health literacy.

Specially, students in the MHAA course had improved knowledge and self-efficacy as

compared to a control group taking lifespan development. Improvements occurred for

both face-to-face and online formats. Future research is needed to better determine the

use of the MHAA-AT in assessing behavioral change in participants and the influence of

the MHAA curriculum on students’ specific behaviors related to mental health literacy.

In sum, the two studies of this dissertation provide a unique, process-based approach to

delivering and assessing mental health literacy programs on a college campus.

(159 pages)
 v

PUBLIC ABSTRACT

Mental Health Awareness and Advocacy: Assessment Tool Development and an

Evaluation of a College-Based Curriculum

Ty B. Aller, MMFT LMFT

Students’ mental health issues are a common concern on college campuses and

are often addressed via prevention programming called mental health literacy. This

dissertation consists of two studies regarding mental health literacy programming for

college students at a western university in the United States. In study one, the Mental

Health Awareness and Advocacy Assessment Tool (MHAA-AT) was created and

evaluated for its utility in assessing college students’ mental health literacy. This

assessment tool is unique in that it is built upon a process-based approach to mental

health literacy. The assessment tool demonstrated adequate psychometric properties and

it was deemed an appropriate tool to assess college students’ mental health literacy,

specifically their declarative knowledge, self-efficacy, and behaviors. In study two the

Mental Health Awareness and Advocacy (MHAA) curriculum was created and evaluated

in a college student population. The MHAA curriculum is unique in that is taught in-

person or online in a degree seeking program at a college or university. Results from

study two suggest that the MHAA curriculum was effective in increasing college

students’ mental health literacy scores, specifically their declarative knowledge and self-

efficacy. The benefit of this two-study dissertation is that it provides a unique way to

deliver and evaluate effective mental health literacy prevention programming on a larger

scale via a degree-seeking program to college students.

 vi

DEDICATION

To anyone that might be able to help relieve the suffering of another through a bit of

knowledge and the comfort of a helping hand.

 vii

ACKNOWLEDGMENTS

I would like to first thank my chairpersons for their willingness to have me as a

student. I send deep appreciation to Dr. Scot Allgood who helped guide me to this field as

an undergraduate, providing the opportunities for this dissertation. I am especially

grateful for Dr. Elizabeth Fauth who supported me through some of the most challenging

times of my academic and personal life. She continues to be a phenomenal mentor and

friend and I feel honored to have been one of her students. I am also appreciative for Drs.

Ryan Seedall, Josh Novak, and Gretchen Peacock who each contributed to my project

and personal growth in unique and rewarding ways. I also send appreciation to Drs.

Sheila Anderson and Lori Roggman for taking me on as an undergraduate student and

helping mentor me. Without their persistent and encouraging support, I do not think I

would have been able to make it this far in my schooling.

I send very special and heartfelt thanks to my family for supporting me

throughout my education. To my mother, Mikall, and my aunts, McKenzie and Adrienne,

you provided me with insight and support that I found invaluable. There were times

where I lost sight of the reasons for why I was on this pursuit and each of you helped

anchor me back to my values. I also want to thank each of my friends, especially Logan,

Shane, Dylan, Derek, Drake, Kent, Teresa, and Nicky. Each of you helped enrich my

experience while at USU and made the difficulty of graduate school a bit more enjoyable.

Your support, encouragement, patience, laughter, and understanding helped me persist.

Thank you.

Ty B. Aller
 viii

CONTENTS

Page

ABSTRACT................................................................................................................... iii

PUBLIC ABSTRACT ................................................................................................... v

DEDICATION ............................................................................................................... vi

ACKNOWLEDGMENTS ............................................................................................. vii

LIST OF TABLES ......................................................................................................... x

LIST OF FIGURES ....................................................................................................... xii

CHAPTER

1. GENERAL INTRODUCTION .......................................................................... 1

Higher Education’s Approach to Student’s Mental Health Issues..................... 3

Theoretical Underpinnings of the Current Approach and Measuring
Outcomes ..................................................................................................... 6
Description of Present Studies ........................................................................... 8
References .......................................................................................................... 9

2. STUDY 1: MEASURING MENTAL HEALTH LITERACY: CREATION

AND VALIDATION OF THE MENTAL HEALTH AWARENESS AND
ADVOCACY ASSESSMENT TOOL (MHAA-AT) IN A COLLEGE
SAMPLE ............................................................................................................ 13

Introduction ........................................................................................................ 13
Method ............................................................................................................... 23
Results ................................................................................................................ 33
Discussion .......................................................................................................... 42
Implications for Future Research ....................................................................... 46
Implications for Interventionists ........................................................................ 47
Limitations ......................................................................................................... 48
Conclusion ......................................................................................................... 49
References .......................................................................................................... 49
 ix

Page

3. STUDY 2: MENTAL HEALTH AWARENESS AND ADVOCACY

(MHAA): AN EVALUATION OF A COLLEGE-BASED MENTAL
HEALTH LITERACY CURRICULUM ........................................................... 54

Introduction ........................................................................................................ 54
Method ............................................................................................................... 72
Results ................................................................................................................ 85
Discussion .......................................................................................................... 91
Limitations ......................................................................................................... 96
Conclusion ......................................................................................................... 97
References .......................................................................................................... 98

4. GENERAL DISCUSSION ................................................................................ 104

A Process-Based Approach to Assessing Mental Health Literacy .................... 105

Can Mental Health Literacy be Offered in Course Format as Part of a
Degree-Seeking Program? ........................................................................... 108
Does the Curriculum Improve Mental Health Literacy? ................................... 109
Does Modality Influence Mental Health Literacy Outcomes? .......................... 111
Future Direction ................................................................................................. 112
Conclusion ......................................................................................................... 113
References .......................................................................................................... 113

APPENDIX: MENTAL HEALTH AWARENESS AND ADVOCACY

ASSESSMENT TOOL ............................................................................ 115

CURRICULUM VITAE ................................................................................................ 134

 x

LIST OF TABLES

Table Page

2.1 Key Sample Characteristics ............................................................................... 24

2.2 Three IRT Analyses: Item Fit Characteristics (MSQ) for MHAA-AT.............. 34

2.3 Three IRT Analyses: Cronbach’s Alpha of the MHAA-AT Declarative

Knowledge Items ............................................................................................... 34

2.4 Three IRT Analyses: Item Difficulty Estimates (Eta) and Conditional
Probabilities for MHAA-AT .............................................................................. 35

2.5 Three IRT Analyses: Raw to Scaled Scores Conversions for MHAA-AT ........ 35

2.6 Self-Efficacy Items of the MHAA-AT: Communalities and Factor

Loadings for Principal Axis Factoring............................................................... 39

2.7 Behavior Items of the MHAA-AT: Communalities and Factor Loadings for
Principal Axis Factoring .................................................................................... 40

2.8 Mean, Standard Deviations, Possible Range and Raw Percent Correct of
Key Outcome Variables at Pretest ..................................................................... 43

2.9 Correlations Among MHAA-AT Microprocess Items and Key Measures ....... 44

3.1 Key Sample Characteristics ............................................................................... 75

3.2 Correlations Between Key Outcome Variables at Pretest ................................. 76

3.3 Correlations Between Key Outcome Variables at Posttest ................................ 77

3.4 Means and Standard Deviations with Available Data between Conditions
on Outcome Measures........................................................................................ 86

3.5 Time X Condition Results of a Two-Way Repeated Measures ANOVA

Analysis on Key Outcome Variables ................................................................. 87

3.6 Means and Standard Deviations with Available Data between Conditions
on Mental Health Outcomes .............................................................................. 87
 xi

Table Page

3.7 Time X Modality Results of a Two-Way Repeated Measures ANOVA

Analysis on Key Outcome Variables ................................................................. 89

3.8 Means and Standard Deviations with Available Data Between

Conditions on Mental Health Outcomes ............................................................ 91

3.9 Time X Condition and Time X Modality Results of a Two-Way Repeated

Measures ANOVA Analysis on Mental Health ................................................. 91
 xii

LIST OF FIGURES

Figure Page

2.1 Process-based model of mental health awareness and advocacy ..................... 20

2.2 Person-item maps for three separate IRT analyses: MHAA-AT ..................... 36

3.1 Process-based model of mental health awareness and advocacy curriculum .. 69

3.2 Participant flow diagram .................................................................................. 73

3.3 Mean plots of significant interactions for Time X Condition .......................... 88

3.4 Mean plots of significant interactions for Time X Modality............................ 90

4.1 Process-based model of mental health awareness and advocacy ..................... 106
 CHAPTER 1

GENERAL INTRODUCTION

College students’ mental health issues are a common concern on college

campuses in the United States (Auerbach et al., 2018; Center for Collegiate Mental

Health, 2017; Eisenberg, Hunt, & Speer, 2013; Lipson, Lattie, & Eisenberg, 2018).

Mental health issues commonly refer to mental illnesses (e.g., major depressive disorder,

generalized anxiety disorder, bipolar disorder, schizophrenia) that cause clinically

significant distress in an individual’s life. Studies use multiple terms to describe

diagnoses of mental illness including serious mental illness, mental disorders, mental

conditions, and mental health issues. Often these are used interchangeably, although

diagnoses should be used only in cases where a trained mental health professional has

ensured diagnostic criteria have been met (American Psychiatric Association, 2013). For

the purpose of this dissertation, I use the common term, mental health issues. This term is

more relatable to community populations and directs participants and readers away from

developing an identity that is assumed by clinical training.

Epidemiological studies of college students suggest that the college student

population experiences depression and anxiety symptoms at similar rates as those

reported by the Diagnostic and Statistical Manual of Mental Disorders in the general

population (DSM-5; i.e. 15.6% of undergraduates and 13.0% of graduate students have

depression and/or anxiety, and the general population experiences anxiety and depression

at 18.1% and 6.7%, respectively; American Psychiatric Association, 2013; Eisenberg,

Gollust, Golberstein, & Hefner, 2007). Other studies support that college students
 2

experience mental health issues at a higher rate than the prevalence statistics provided by

the DSM-5 (60% of all students surveyed; Zivin, Eisenberg, Gollust, & Golberstein,

2009).

Mental health issues are often associated with other negative outcomes. For

instance, students experiencing depression are more likely to have lower GPAs in their

first two years of school and this negative effect is stronger in students that also have a

comorbid anxiety disorder (Eisenberg, Golberstein, & Hunt, 2009). These same

symptoms are associated with lower levels of campus involvement, retention, and

graduation rates (Eisenberg et al., 2009; Salzer, 2012). Not surprisingly, depression and

anxiety are the most common mental health issues of students on college campuses and

are often precursors to students’ suicide ideation (Center for Collegiate Mental Health,

2018; Kisch, Leino, & Silverman, 2005; Wilcox et al., 2010). The American College

Health Association (2015) reported that 9.6% of college students (N = 19,861) across the

United States have considered suicide in the past 12 months. Additional studies on

college students report that 2% of all students have experienced suicide ideation in the

past four weeks (Eisenberg et al., 2007), and that 37% of undergraduates (N = 15,000)

and 30% of graduate students (N = 11,441) have indicated that they “wish this all would

just end” in the past 12 months (Drum, Brownson, Burton, Denmark, & Smith, 2009, p.

216). This relatively high rate of suicidal ideation poses unique concerns for college

campuses across the United States (Kitzrow, 2009).

Mental health issues in the college context often persist due to the unique

stressors that college engenders, including pressures related to academic performance and
 3

post-graduation plans (Beiter et al., 2015). Many students facing mental health issues on

college campuses are actively seeking therapy services from either campus-based

services and/or community-based services, including online resources (Eisenberg, Hunt,

& Speer, 2011; Kern, Hong, Song, Lipson, & Eisenberg; 2018;). Literature suggests that

college-based therapy services are seeing dramatic increases in usage (Beiter et al., 2015;

Castillo & Schwartz, 2013; Center for Collegiate Health, 2018). There are, however, still

concerns of unmet needs. Partially illustrating this point, a study of undergraduate and

graduate students reported that 43.2% of students had never received information from

their school about anxiety or depression despite 53.2% reporting that they are interested

in receiving this information (n = 19,861; ACHA, 2015). Collectively these findings

suggest that mental health issues are prevalent on college campuses and are associated

with both suicidal ideation and school-related outcomes.

Higher Education’s Approach to Student’s Mental Health Issues

Traditionally, college campuses emphasize individual treatments such as therapy

to approach college students’ mental health issues. While direct therapy interventions are

empirically supported as being effective, these resources are often overburdened due to

the high volume of student needs, specifically students that are in crisis (Center for

Collegiate Health, 2018; Kitzrow, 2009). Direct therapy is often one-to-one, thus the

ability to reach a majority of the student body is limited. To address this concern, many

universities now employ community wide interventions in line with the World Health

Organizations on prevention strategies to try and prevent mental health issues from
 4

reaching a crisis point (World Health Organization, 2004). These interventions are often

offered as educational seminars. The seminars target students’ mental health issues by

helping educate students to identify at-risk students and then encourage students to help

prevent mental health issues through referrals to treatment. In the following sections, the

three most common health education approaches used on college campuses are briefly

summarized.

Programs for Identifying Mental Health Issues

Programs targeting the identification of mental health issues, typically referred to

as mental health literacy programs, are commonly defined as programs that address

knowledge and beliefs about mental disorders which aid in their recognition,

management, or prevention (Jorm, 2000; Jorm et al., 1997). In a review of common

mental health literacy programs, many of the programs use a face-to-face or online forum

to educate participants about the signs and symptoms of mental health issues including

depression, anxiety, and suicide risk (Francis, Pirkis, Dunt, Blood, & Davis, 2002).

Limited studies have been conducted in college samples, but general improvements in a

secondary education setting include reduced stigma of mental health issues, increased

empathy towards those struggling, and a better understanding of how to access resources

(Wei, Hayden, Kutcher, Zygmunt, & McGrath, 2013). While these programs are effective

in improving knowledge about these problems, many of the current programs do not

address a students’ ability or confidence in responding to mental health issues.

Additionally, many studies conducted on programs targeting the identification of mental

health issues are specific to Australian samples (Kitchener & Jorm, 2006) or secondary
 5

education samples (Wei et al., 2013) and are only implicitly guided by theory.

Programs for Locating Evidence-Based Resources

Literature on help-seeking behavior, including locating evidence-based resources,

consistently reports a number of reasons that students do not seek mental health services.

These include stigma surrounding mental health, students not thinking they need mental

health services, thinking their problems are not severe enough, or lacking understanding

of how to access resources (Hunt & Eisenberg, 2010). Interventionists have recognized

these barriers to services and have sought approaches that help address them. For

instance, several programs emphasizing locating evidence-based resources use people

with mental health issues to facilitate interventions. This approach helps elucidate the

deficits those with mental health issues might experience by increasing empathy and

understanding of the severity of these problems by using first-hand accounts (Campbell,

2005). These programs are associated with increased empathy and understanding of

mental health issues (Rones & Hoagwood, 2000), but making generalizability claims to

the college population is limited. Additionally, much of the evidence doesn’t illustrate

whether a student’s ability to locate high-quality resources to treat mental health issues is

increased.

Programs for Responding to Mental Health Issues

Arguably the most common form of helping students learn to respond to mental

health issues are found in varying forms of gatekeeper trainings (Lipson, Speer,

Brunwasser, Hahn, & Eisenberg, 2014). Gatekeeper trainings are typically characterized
 6

by components of psychoeducation and skill development (Lipson et al., 2014). For

instance, one common gatekeeper training used on college campuses is called Question,

Persuade, Refer (QPR; Quinnett, 1995, 2007). The QPR training helps students learn

questions to identify and clarify suicide risk level in fellow students. The training then

helps students learn to persuade an at-risk student to seek professional help. Lastly, the

training helps students to identify resources they can then refer at-risk students to

immediately. While QPR is a common gatekeeper training used on college campuses,

there are more intensive trainings that are used (e.g., Mental Health First Aid). These

gatekeeper trainings present varying benefits to students including providing valuable

information about identifying mental health issues, specifically depression, anxiety,

bipolar disorder, schizophrenia, and suicidal thoughts and actions. They also provide a

skillset that can be used to help deescalate distressed students (Lipson et al., 2014). While

these programs help increase students’ knowledge, self-efficacy, and responsive

behaviors (as per self-report), there is little evidence showing a direct impact on use of

services or a decrease in suicide rates on campuses. In sum, there are not, to my

knowledge, college-based curriculums that address each of these empirically supported

areas and are explicitly theory driven.

Theoretical Underpinnings of the Current Approach and

Measuring Outcomes

A college-based curriculum that addresses identifying mental health issues,

locating evidence-based resources, and responding to mental health issues would better
 7

account for the theoretical propositions of the health belief model (Becker, 1974). This

model proposes that the perceived susceptibility, severity, benefits and barriers help

predict a person’s likelihood of trying to prevent, screen, or control an illness (Becker,

1974). Additionally, the model explains that an individual’s self-efficacy (Bandura, 1997)

influences his or her likelihood of responding to a health issue. The concept of self-

efficacy, directly explained in social cognitive theory (Bandura, 1997), provides an

explanation of learning via a developmental process. While a curriculum that

incorporates identifying mental health issues, locating evidence-based resources, and

responding to mental health issues would better address the health belief model and

social cognitive theory, there is not currently an established, process-based measure that

can effectively evaluate this type of program.

Currently, identifying mental health issues is most commonly evaluated using

vignettes depicting an individual with a mental health issue and asking respondents to

determine if the individual has any significant problem (Jorm, 2012). Another form of

evaluating identifying mental health issues and locating evidence-based resources comes

by using measures that asses a student’s declarative and perceived knowledge (Wyman et

al., 2008). In evaluating students’ ability to respond to mental health issues, one of the

most commonly used methods is to assess a student’s self-efficacy in identifying and

appropriately responding to a mental health issues (Lipson et al., 2014). While each of

these methods posit unique strengths, they do not evaluate each of the factors the health

belief model proposes as important for determining whether individuals will take action

and respond to a health issue. A measure based in both theory (Bandura, 1997; Becker,
 8

1974;) and empirical literature, would consider students’ declarative knowledge, self-

efficacy, and direct behavioral responses in identifying mental health issues, locating

evidence-based resources, and responding to mental health issues concurrently.

Description of Present Studies

The use of programs targeting students’ ability to identify mental health issues,

locate evidence-based resources, and respond to mental health issues have shown

promising results in addressing the negative effects of college students’ mental health

issues (see Lipson et al., 2014 for a detailed review). There have not, however, been

evaluation studies of college-based curriculums that incorporate identifying mental health

issues, locating evidence-based resources, and responding to mental health issues

simultaneously. Accordingly, I propose a two-part study that will first test the validity

and reliability of the Mental Health Awareness and Advocacy Assessment Tool (MHAA-

AT), created and presented here for the first time. Second, I will evaluate a college-based

curriculum titled, “Mental Health Awareness and Advocacy,” and the effectiveness of

this curriculum in improving college students’ declarative knowledge, self-efficacy, and

behaviors related to identifying mental health issues, locating evidence-based resources,

and responding to mental health issues using the MHAA-AT. To accomplish these goals,

the following research questions will be addressed:

Study One

1. Using Item-Response Theory, what are the item and trait level characteristics

of the declarative knowledge items in the MHAA-AT?

 9

2. Using exploratory factor analysis, what is the factor structure of the self-

efficacy and behavior items of the MHAA-AT?

3. Does the newly created MHAA-AT demonstrate strong psychometric

properties (e.g., construct validity, internal consistency)?

Study Two

1. Do students that participate in the college-based, Mental Health Awareness

and Advocacy curriculum improve their scores on the MHAA-AT in comparison to the

control group when accounting for students’ key demographic factors?

a. Analytic comparisons will include:

i. All treatments (in-person, online curriculum) versus control group.

ii. In-person curriculum versus online curriculum.

iii. In-person treatment curriculum versus in-person control group.

iv. Online treatment curriculum versus online control group

References

American College Health Association. (2015). American College Health Association-

National College Health Assessment II: Reference Group Executive Summary
Spring 2015. Hanover, MD: American College Health Association.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental

disorders (5th ed.). Washington, DC: Author.

Auerbach, R. P., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C., Cuijpers, P., … WHO
WMH-ICS Collaborators. (2018). WHO world mental health surveys
international college student project: Prevalence and distribution of mental
disorders. Journal of Abnormal Psychology. Advance online publication. doi:
10.1037/abn0000362
 10

Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H.

Freeman.

Becker, M. H. (1974). The Health Belief Model and personal health behavior. Health
Education Monographs, 2, 324-473.

Beiter, R., Nash, R., McCrady, M., Rhoades, D., Linscomb, M., Clarahan, M., &
Sammut, S. (2015). The prevalence and correlates of depression, anxiety, and
stress in a sample of college students. Journal of Affective Disorders, 173, 90-96.

Campbell, J. (2005). The historical and philosophical development of peer-run support

programs. In B. Schell, P. Corrigan, & R.O. Ralph (Eds.), On our own, together:
Peer programs for people with mental illness (pp. 17-66). Nashville, TN:
Vanderbilt University Press.

Castillo, L. G. & Schwartz, S. J. (2013). Introduction to the special issue on college

student mental health. Journal of Clinical Psychology, 69(4), 291-297.

Center for Collegiate Mental Health. (2018). 2017 Annual report (Publication No. STA
18-166). University Park, PA; Author.

Drum, D. J., Brownson, C., Burton Denmark, A., & Smith, S. E. (2009). New data on the
nature of suicidal crises in college students: Shifting the paradigm. Professional
Psychology: Research and Practice, 40(3), 213-222. doi: 10.1037/a0014465

Eisenberg, D., Golberstein, E., & Hunt, J. B. (2009). Mental health and academic success
in college. The BE Journal of Economic Analysis & Policy, 9(1), 1935-1682. doi:
10.2202/1935-1682.2191.

Eisenberg, D., Gollust, S. E., Golberstein, E., & Hefner, J. L. (2007). Prevalence and
correlates of depression, anxiety, and suicidality among university
students. American Journal of Orthopsychiatry, 77(4), 534-542.

Eisenberg, D., Hunt, J., & Speer, N. (2011). Mental health in American colleges and
universities: Variation across student subgroups and across campuses. The
Journal of Nervous and Mental Disease, 201(1), 60-67.

Eisenberg, D., Hunt, J., Speer, N., & Zivin, K. (2011). Mental health service utilization
among college students in the United States. The Journal of Nervous and Mental
Disease, 199(5), 301-308.

Francis, C., Pirkis, J., Dunt, D., Blood, R. W., & Davis, C. (2002). Improving mental
health literacy: A review of the literature. Retrieved from
http://www.health.gov.au/internet/main/publishing.nsf/
content/6A5554955150A9B9CA2571FF0005184D/$File/literacy.pdf
 11

Hunt, J., & Eisenberg, D. (2010). Mental health problems and help-seeking behavior
among college students. Journal of Adolescent Health, 46(1), 3-10.

Jorm, A. F. (2000). Mental health literacy: Public knowledge and beliefs about mental
disorders. The British Journal of Psychiatry, 177(5), 396-401.

Jorm, A. F. (2012). Mental health literacy: Empowering the community to take action for
better mental health. American Psychologist, 67(3), 231.

Jorm, A. F., Korten, A. E., Jacomb, P. A., Christensen, H., Rodgers, B., & Pollitt, P.
(1997). “Mental health literacy”: A survey of the public's ability to recognise
mental disorders and their beliefs about the effectiveness of treatment. Medical
Journal of Australia, 166(4), 182-186.

Kern, A., Hong, V., Song, J., Lipson, S.K., & Eisenberg, D. (2018). Mental health apps in
a college setting: openness, usage, and attitudes. mHealth, 4, 20-32. doi:
10.21037/mhealth.2018.06.01

Kisch, J., Leino, E. V., & Silverman, M. M. (2005). Aspects of suicidal behavior,
depression, and treatment in college students: Results from the Spring 2000
National College Health Assessment Survey. Suicide and Life-Threatening
Behavior, 35(1), 3-13.

Kitchener, B. A., & Jorm, A. F. (2006). Mental health first aid training: review of
evaluation studies. Australian and New Zealand Journal of Psychiatry, 40(1), 6-8.

Kitzrow, M. A. (2009). The mental health needs of today's college students: Challenges
and recommendations. NASPA Journal, 46(4), 646-660.

Lipson, S. K. (2014). A comprehensive review of mental health gatekeeper-trainings for

adolescents and young adults. International Journal of Adolescent Medicine and
Health, 26(3), 309-320.

Lipson, S. K., Lattie, E. G., & Esenberg, D. (2018). Increased rates of mental health
service utilization by U.S. college students: 10-year population-level trends
(2007-2017). Psychiatric Services, 70(1), 60-63. doi: 10.1176/appi.ps.201800332

Lipson, S. K., Speer, N., Brunwasser, S., Hahn, E., & Eisenberg, D. (2014). Gatekeeper
training and access to mental health care at universities and colleges. Journal of
Adolescent Health, 55(5), 612-619.

Quinnett, P. (1995). QPR: Ask a question, save a life. The QPR Institute, Spokane,
Washington. Retrieved from: www.qprinstitute.com.

Quinnett, P. (2007). QPR gatekeeper training for suicide prevention: The model,
rationale, and theory. Retrieved from https://qprinstitute.com/research-theory
 12

Rones, M., & Hoagwood, K. (2000). School-based mental health services: A research
review. Clinical Child and Family Psychology Review, 3(4), 223-241.

Salzer, M. S. (2012). A comparative study of campus experiences of college students

with mental illnesses versus a general college sample. Journal of American
College Health, 60(1), 1-7. doi: 10.1080/07448481.2011.552537

Wei, Y., Hayden, J. A., Kutcher, S., Zygmunt, A., & McGrath, P. (2013). The
effectiveness of school mental health literacy programs to address knowledge,
attitudes and help seeking among youth. Early Intervention in Psychiatry, 7(2),
109-121.

Wilcox, H. C., Arria, A. M., Caldeira, K. M., Vincent, K. B., Pinchevsky, G. M., &
O'Grady, K. E. (2010). Prevalence and predictors of persistent suicide ideation,
plans, and attempts during college. Journal of Affective Disorders, 127(1-3), 287-
294.

World Health Organization. (2004). Prevention of mental disorders: Effective

interventions and policy options: Summary report. Geneva, Switzerland: World
Health Organization.

Wyman, P. A., Brown, C. H., Inman, J., Cross, W., Schmeelk-Cone, K., Guo, J., & Pena,
J. B. (2008). Randomized trial of a gatekeeper program for suicide prevention: 1-
year impact on secondary school staff. Journal of Consulting and Clinical
Psychology, 76(1), 104.

Zivin, K., Eisenberg, D., Gollust, S. E., & Golberstein, E. (2009). Persistence of mental
health problems and needs in a college student population. Journal of Affective
Disorders, 117(3), 180-185.
 13

CHAPTER 2

STUDY 1: MEASURING MENTAL HEALTH LITERACY: CREATION AND

VALIDATION OF THE MENTAL HEALTH AWARENESS AND ADVOCACY

ASSESSMENT TOOL (MHAA-AT) IN A COLLEGE SAMPLE 1

Introduction

Mental health issues (e.g., major depressive disorder, generalized anxiety

disorder, bipolar disorder) affect nearly one in every five adults in the United States in

any given year (National Institute of Mental Health, 2013). The onset of these problems

is often in late adolescence or early adulthood (18-25 years old), which also corresponds

to “the college years”, for many individuals. According to the American Psychological

Association (APA, 2013), the incidence of mental health conditions in this developmental

phase is likely multifactorial. Individuals are still experiencing more rapid rates of change

in post-pubertal biological processes (e.g., neural development, hormonal changes), while

simultaneously managing psychosocial factors (e.g., identity development, changing

friendships), independently managing health behaviors (resulting in potentially poorer

sleep, food choices, etc.), and managing contextual factors (e.g., moving away from

home and parents, increased financial stress). Because of the prevalence of these issues in

college aged populations, students’ mental health issues have become a common and

concerning problem across campuses in the United States (Auerbach et al., 2018; Center

for Collegiate Mental Health, 2018; Eisenberg, Hunt, & Speer, 2013; Kadison &

1
Contributing authors: Elizabeth Fauth, Joshua Novak, and Sarah Schwartz.
 14

DiGeronimo, 2004).

Epidemiological studies examining college students’ mental health issues report

that the estimated prevalence of undergraduate students experiencing depression or

anxiety is 15.6% and 13% for graduate students, with 2% of all students reporting

suicidal ideation in the past four weeks (Eisenberg, Gollust, Golberstein, & Hefner,

2007b). These mental health issues are often associated with lower grade point averages

and reduced likelihood of graduating (Eisenberg, Golberstein, & Hunt, 2009). Because of

these negative effects, colleges have a vested interest in providing cost-efficient,

community level prevention services that target students’ mental health needs (Kitzrow,

2009). This study briefly reviews the literature supporting the effectiveness of community

and education-based programs using the mental health literacy approach in mental health

issues prevention, as well as traditional measurement techniques used in related program

evaluation. We then highlight the rationale for expanding existing measurement to be

more processed-based. Lastly, we present a new, practical, and psychometrically strong

measure that simultaneously assesses the key components of a participants’ declarative

knowledge, self-efficacy, and behaviors in mental health literacy.

Mental Health Literacy

Mental health literacy is a concept that is defined by Jorm and colleagues (1997),

as knowledge and beliefs about mental disorders that aid in their recognition,

management or prevention. In a seminal study seeking to understand mental health

literacy in Australia, approximately 39% of participants could identify depression while

only 27% of participants could identify schizophrenia (Jorm et al., 1997). Likewise, a
 15

more recent study reported that less than 50% of participants could identify depression in

Japan and Sweden (Dahlberg et al., 2008; Jorm et al., 2005). Responding to the low

proportion of the population’s ability to identify mental health issues, prevention

scientists developed programs with a positive influence on participants’ mental health

literacy (e.g., ability to identify and respond to mental health issues by referring

individuals to appropriate resources; Dahlberg et al., 2008; Jorm, 2012; Jorm et al., 1997,

2005).

Programs designed to improve mental health literacy often address the following

topics: (a) the ability to recognize specific disorders or different types of psychological

distress; (b) knowledge and beliefs about risk factors and causes; (c) knowledge and

beliefs about self-help interventions; (d) knowledge and beliefs about professional help

available; (e) attitudes which facilitate recognition and appropriate help-seeking; and (f)

knowledge of how to seek mental health information (Jorm et al., 1997). Community-

based prevention programs using the concept of mental health literacy as their foundation

have demonstrated consistent support in the research literature at increasing each of the

aforementioned areas (see Jorm, 2012 for a full review). These programs are empirically

supported across varying populations, including Australian financial counselors (Bond,

Jorm, Miller, Rodda, Reavley, Kelly & Kitchener, 2016), Australian high school students

(Jorm, Kitchener, Sawyer, Scales, & Cvetkovski, 2010), a population-based Australian

sample (Jorm, et al., 2005) and a population-based Swedish sample (Dahlberg et al.,

2008).

Measuring Outcomes in Mental Health Literacy

 16

Existing approaches assessing declarative knowledge. Declarative knowledge

of mental health issues refers to general facts needed to effectively identify and more

comprehensively understand mental health issues. Declarative knowledge of mental

health issues is often assessed using vignettes or Likert scales (Jorm et al., 1997, 2005;

Jung, von Sternberg, & Davis, 2016; Reavley, Morgan, & Jorm, 2014). Typically,

vignettes are written by clinicians and describe specific symptomology based on

diagnostic criteria from the most recent version of the Diagnostics and Statistical Manual

of Mental Disorders (DSM; American Psychological Association, 2013). After reading

the vignette, the respondent is asked to identify what is happening for the individual, and

responses are evaluated for correct answers (i.e., accurately identifying the issue

described in the vignette). While these vignettes are effective at fully describing the

symptoms of a clinical problem, and map onto a person’s knowledge of the issues, they

are tedious to evaluate on a large scale (O’Connor & Casey, 2015).

To facilitate assessment of declarative knowledge with studies using larger

sample sizes, studies often use items with Likert-scale responses, for example, “Relative

to the average person, how knowledgeable are you about mental illnesses (such as

depression and anxiety disorders) and their treatments?” (responses range from 1 [Not at

all], to 5 [extremely]; Lipson, Speer, Brunwasser, Hahn, & Eisenberg, 2014). While the

Likert scale approach is more efficient at assessing large samples of participants, the

items included often do not fully depict the construct of ‘knowledge’, and are more akin

to the construct of ‘metacognition’ (e.g., what do you think you know about the

construct). A more effective approach to assessing participants’ declarative knowledge

 17

may include using multiple-choice questions that have item content that would require

participants to recognize mental health symptoms, similarities and differences among

mental health issues, resources to treat these disorders, and skills related to responding to

these issues. Currently, there are several studies incorporating this approach, but these

measurements only commonly report the internal consistency of items and do not report

other important psychometric properties (Quinnett, 2007; Wyman et al., 2008;). There is,

however, one measure titled the Mental Health Literacy Scale that reports strong

psychometric properties (i.e., validity and reliability estimates) and maps onto the

concept of mental health literacy seamlessly (O’Connor & Casey, 2015). Our measure

builds upon the success of this assessment by incorporating these types of items into a

processed-based measure with additional domains.

Existing approaches assessing self-efficacy. Self-efficacy is defined as the belief

that an individual can successfully complete a behavior that is requisite to produce a

desired outcome (Bandura, 1982). Measures assessing participants’ self-efficacy are

arguably the most common measurement strategy implemented in studies evaluating

mental health literacy and are most often completed by using traditional Likert scales.

(Mitchell, Kader, Darrow, Haggerty, & Keating, 2013; Tompkins, Witt, & Abraibesh,

2010; Wyman et al., 2008). For instance, one study assessed participants’ self-efficacy in

their knowledge of mental health literacy by asking participants to respond to a 5-point

Likert scale question, “I have a good idea of how to recognize that a student is in

emotional or mental distress” and “I know what mental health and counseling resources

are available for students” (strongly agree to strongly disagree; Lipson et al., 2014). Self-
 18

efficacy is used to evaluate responding to mental health issues by asking questions

including, “I am aware of warning signs for suicide” and “I can recognize students

contemplating suicide by the way they behave” (Wyman et al., 2008). In the research

literature, these scales have demonstrated high internal consistency and are predictive of

other health behaviors (see Sheeran et al., 2016 for a meta-analytic review). In other

measures of mental health literacy, factor analyses have supported multiple factor models

(single and multiple factor iterations) that include knowledge, beliefs, and resource

oriented mental health literacy questions independently and combined (Jung, von

Sternberg, & Davis, 2016).

Existing approaches assessing behavioral outcomes. Behavioral outcomes

included in past studies typically assess participants’ self-reported response of either their

own mental health issue or an issue for someone they know well in a retrospective

account (Mitchell et al., 2013; Lipson et al., 2014; Wyman et al., 2008). There are two

common approaches to measuring behavioral outcomes in this domain: (1) the likelihood

of responding to mental health issues and (2) responding or providing referrals to

someone that is experiencing a mental health issue via a retrospective self-report. One

study measured likelihood of responding to a mental health issues on a three-point Likert

scale (not very likely, somewhat likely, or highly likely), based on the Question Persuade

Refer (QPR) Institute’s survey (Mitchell et al., 2013; Quinnett, 2007). Researchers asked

participants to rate themselves on the likelihood of engaging in certain suicide prevention

behaviors including: telling a suicidal person where to get help, calling a crisis line to get

help for a suicidal person, and going with a suicidal person to get help. In another study,
 19

participants were asked to indicate how many times they had referred an individual

experiencing suicidal thoughts to professional resources (Wyman et al., 2008). These

measurement strategies assess if participants are responding to mental health issues via

their self-report of their own behavior retrospectively, however the diversity of content

they assess are limited to one or two issues (e.g., suicidality, seeking professional help),

and typically do not assess mastery of identifying a mental health issue or locating

evidenced-based resources. An assessment tool that emphasizes the process-based

approach to becoming literate in mental health can address these holes in current

evaluation approaches.

Mental Health Awareness and Advocacy

Assessment Tool: A Process-Based Approach

Mental health literacy is a well-articulated descriptive approach outlining varying

elements of the mental health field that need to be addressed in community-based

prevention programs. While there are varying useful, psychometrically-sound

measurement approaches to examine mental health literacy (e.g., Mental Health Literacy

Scale; O’Connor & Casey, 2015), we believe current measurement approaches can be

strengthened by using a process-oriented approach (defined below). The process-based

measure developed and examined in this study is titled the Mental Health Awareness and

Advocacy assessment tool (MHAA-AT; additional details on measurement development

are described in the methods section).

The MHAA-AT is made up of three progressive domains that emphasize the

process of mental health literacy: (1) the ability to identify signs and symptoms of mental
 20

health issues (Identifying Domain); (2) the ability to identify and access evidence-based

mental health resources (Locating Domain); and (3) the ability to effectively and

appropriately respond to mental health issues (Responding Domain; see Figure 2.1). The

MHAA-AT then examines the overall process of mental health literacy by breaking these

three domains into three micro-processes: acquiring knowledge (knowledge), building

self-efficacy (self-efficacy), and applying skills (behaviors).

Identifying Domain

a) Declarative Knowledge
b) Self-Efficacy
c) Behaviors

Locating Domain Responding Domain

a) Declarative Knowledge a) Declarative Knowledge

b) Self-Efficacy b) Self-Efficacy
c) Behaviors c) Behaviors

Note. The circles represent the macro-processes. Micro-processes are listed within each macro-process.
Declarative knowledge refers to the micro-process of acquiring knowledge; Self-efficacy refers to building
self-efficacy, and behaviors refers to applying skills.

Figure 2.1. Process-based model of mental health awareness and advocacy.

The emphasis of the process-based approach in the MHAA-AT is the integration

of micro-level processes (acquiring knowledge, building self-efficacy, and applying skills

to respond) into each of the macro-level processes outlined in mental health literacy. The

following example illustrates the micro-level processes within the macro-level processes:

A student in a mental health class learns about the signs and symptoms of depression and

is able to correctly state or recall the facts they learned about identifying depressive

symptomatology (Identifying domain: acquiring knowledge). The student may then feel
 21

more confident in his or her ability to identify depression (Identifying domain: building

self-efficacy) and can ask pertinent questions to others, or to identify, directly, the key

symptoms of depression in those around them (Identifying domain: applying skills). This

student may want to learn about empirically-based resources for a person identified as

needing help (Locating domain). In a similar process as explained above, the student

progresses through knowing what resources are available (Locating: acquiring

knowledge), feeling more confident in knowing that the resources are trustworthy and

appropriate for the clinical issue (Locating: building self-efficacy), and getting contact

information about a specific supportive service for the person in need (Locating: applying

skills). Lastly, the student might respond to the person experiencing a mental health issue

(Responding domain). The student learns about appropriate responsive behaviors

(Responding: acquiring knowledge), he or she gains confidence in his or her ability to

respond effectively (Responding: building self-efficacy) and does something specific to

respond to the person in need, such as making a referral to a resource (Responding:

applying skills). Although we provide these steps in a linear fashion, that is just for

descriptive purposes. In reality, the student might have performed steps concurrently or in

a different order.

In sum, the MHAA-AT assesses the macro-level processes identified in mental

health literacy (Identifying, Locating, and Responding), and assesses the more micro-

level processes within each domain, related to student’s learning and understanding

(acquiring knowledge), mastery and confidence in using the appropriate skills and

resources (building self-efficacy), and acting on this confidence appropriately

 22

(behaviors). By including items for each micro-process within each macro-process, the

MHAA-AT encompasses a more systematic and integrated assessment of the

participants’ mental health literacy.

The Present Study

The primary purpose of this study was to create a new, process-oriented, practical,

and psychometrically strong assessment tool that assesses students’ declarative

knowledge, self-efficacy, and behavioral outcomes related to mental health literacy called

the Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT). To develop

such an assessment tool, we divided mental health literacy into three progressive

processes: (1) the ability to identify signs and symptoms of mental health issues

(Identifying domain); (2) the ability to identify and access evidence-based mental health

resources (Locating domain); and (3) the ability to effectively and appropriately respond

to mental health issues (Responding domain). Then using the guidance of theory,

research literature, past measures used to evaluate mental health literacy, and content

experts in the field of mental health, we developed and tested the Mental Health

Awareness and Advocacy-Assessment Tool (MHAA-AT) in a college population to

address the following research questions:

RQ1: What are the item and respondent characteristics of the declarative

knowledge items of the MHAA-AT?

RQ2: What is the underlying factor structure of the self-efficacy and behavior

items of the MHAA-AT?

RQ3: Does the MHAA-AT demonstrate strong reliability and validity?

 23

Method

Participants

We wanted a sample from a wide range of colleges outside of our own institution

and geographic/cultural region, thus we recruited via Amazon's Mechanical Turk, and

only accepted those participants that indicated that they self-identified as a college

student (MTurk; Buhrmester, Kwang, & Gosling, 2011). Based on recommendations

found in the research literature on factor analysis, a minimum of three participants per

item were collected (Costello & Osborne, 2005). Participants were included in the study

if they were over the age of 17 and under the age of 26, as the ages of 18-25 are

commonly reflect the “traditional” college student. Individuals included in the study

indicated they were proficient in the English language.

The final measurement sample included 296 college students. Participants

included 296 college-attending 18- to 25-year-old students (M = 22.67, SD =1.79; see

Table 2.1 for key sample characteristics). Of the 296 participants, the sample averaged in

the mild depression range on the PHQ-9 (M = 7.82, S.D. = 6.8) and averaged in the mild

anxiety range on the GAD-7 (M = 6.62, S.D. = 5.85). About one-third ( n= 109, 36.8%)

of the participants reported they had been diagnosed with a mental health issue, 168

(56.8%) reported they were emotionally close with someone that had experienced a

mental health issue, 63 (21.3%) reported they had experienced suicidal thoughts in the

past six months, 105 (35.5%) reported they had known someone that had experienced

suicidal thoughts in the past 6 months, and 56 (18.9%) reported they had received therapy
 24

in the past six months.

Table 2.1

Key Sample Characteristics

Variable N % of sample
Year in School 296
Freshman 25 8.40
Sophomore 56 18.90
Junior 82 27.70
Senior 108 36.50
Graduate Professional 25 8.40

Gender Identity 296

Female 156 52.70
Transgender Female 1 0.30
Male 132 44.60
Transgender Male 2 0.70
Gender-Questioning 2 0.70
Two-Spirit 2 0.70
Other 1 0.30

Race/Ethnicity 294
Black or African/American 35 11.80
White/European American 185 62.50
American Indian 7 2.40
Asian 33 11.10
Hispanic or Latino 26 8.80
Bi-Racial 8 2.70

Mother’s Level of Education 296

Some High School 15 5.10
High School Graduate 60 20.30
Some College 63 21.30
Associate Degree 25 8.40
Bachelor’s Degree 86 29.10
Master’s Degree 33 11.10
Doctorate Level Degree 10 3.40

Father’s Level of Education 280

Some High School 16 5.40
High School Graduate 76 25.70
Some College 47 15.90
Associate Degree 22 7.40
Bachelor’s Degree 73 24.70
Master’s Degree 37 12.50
Doctorate-level Degree 9 3.00

Financial Stress Growing Up 295

Not at all concerned 78 26.40
Somewhat concerned 156 52.70
 25

Very Concerned 61 20.60

Procedure

Participants were routed to a survey on Qualtrics.com after selecting the mental

health awareness and advocacy assessment tool study on the MTurk system. The survey

contained a general overview of the study (i.e., letter of information) and the measure,

itself. After reading the letter of information, participants who chose to continue

completed a demographics questionnaire (i.e., age, gender identity, ethnicity, income,

education, etc.). Participants failing to meet the age requirement (18-25 years old) and

educational requirement (attending college) were excluded from further participation

based on Institutional Review Board approved inclusion criteria. Participants qualifying

for the study received $1 for participating in the study, which is in line with MTurk

time/payment standards. Previous research has suggested that while MTurk can provide

quick data in a cost-efficient manner, this data can be of lower quality at times

(Buhrmester et al., 2011). In response, quality insurance safeguards were embedded in

the current study, and included several Instructional Manipulation Checks (IMCs). The

first safeguard was accomplished by using “captcha” or “reverse-turing test” questions,

including questions that have verifiable answers, (“What is 2 +2?”; Mason & Suri, 2012).

Therefore, we embedded several quality-control items in the questionnaire to confirm that

participants attended to the survey (e.g., “Select ‘disagree’ as the answer to this question).

Additionally, a “captcha” phrase to reduce the possibility of completion by bots was

included. Lastly, we blocked repeated Internet Protocol Addresses and MTurk worker

identification numbers to prevent duplicate responses.

 26

Item development. We used a three-step process to create items included in the

measure: (1) initial item development and editing; (2) item review by a panel of content

experts; (3) a review by a bachelor-level student panel to increase plain language usage.

First, we conducted a thorough literature review to examine studies evaluating programs

covering the concept of mental health literacy. We drafted items within the declarative

knowledge, self-efficacy, and behavioral outcomes section (see below) based on the

guidance of previous measures in the research literature (Lipson et al., 2014; Quinett,

2007; Wyman et al., 2008). We used these items as a benchmark to guide content

development but did not use the items verbatim. Next, an extensive review of factors that

hinder or facilitate help-seeking behaviors in college populations (e.g., Eisenberg et al.,

2007a), correlates of mental health issues in college populations (e.g., Eisenberg et al.,

2007b), and information regarding effective responses to mental health issues (e.g.,

Quinett, 1995, 2007) was completed to generate additional items. Behavioral outcomes

included in the measure (e.g., important to key stakeholders such as counseling centers,

administrators, student affairs officers) were generated based on the process-based model

of mental health awareness and advocacy described previously (e.g., knowing about

depression, makes you more confident to talk to someone about depression, which leads

to the student helping the person with depression to seek help).

The first and second author reviewed and revised the initial items to identify any

potential syntax errors, content holes, and other logistical problems. Next, a panel of five

content experts working in the mental health field (e.g., clinical faculty, researchers,

teachers) reviewed items for face validity with particular attention to identifying content
 27

holes within the three domains. Three iterations of this process were completed, followed

by presenting items to an informal focus group of four individuals with a bachelor’s level

education. These individuals were asked to review the plain language approach to

questions and to identify any confusion in items. The informal focus group then provided

feedback they deemed appropriate related to the accessibility of the language used in the

measure. In all, this process created 66 items that were included in the evaluation of the

MHAA-AT.

Measurement

Declarative knowledge. There were 30 items related to knowledge with ten items

assessing each of the three content areas (i.e., Identifying mental health issues, Locating

empirically based resources, and Responding to mental health issues). Knowledge items

were selected for inclusion if the panel agreed the items had unambiguous “right” and

“wrong” answers, based on consistent findings or evidence, and included topics that

should be addressed in education on that specific domain. All items in the knowledge

domain were assessed using a five-answer multiple-choice test. Sample multiple choice

items assessing knowledge included: “Individuals are more likely to experience

symptoms of depression when they are between the ages of: a) 6-17 years old, b) 18-29

years old, c) 30-41 years old, d) 41-52 years old, e) I don’t know the answer” and “Which

of the following has been identified by research as an effective treatment for severe major

depressive disorder?: a) Talk Therapy, b) Journaling, c) Herbal Supplements, d)

Exercise, e) I don’t know the answer”. Items were coded as a one if they are correct and a

0 if they were incorrect. The items were then scored zero to ten with the raw score then
 28

being converted using a logarithmic function based on the non-linearity of item difficulty.

Self-efficacy. There were 20 self-efficacy items included that assessed each of the

three content areas. All self-efficacy items were assessed using a 6-point Likert scale (0 =

Not at all confident; 5 = Completely confident). Sample items assessing self-efficacy

included: “I can identify each of the diagnostic criteria for major depressive disorder.;”

“In my experience, having conversations about mental health issues could help to

decrease stigma attached to mental health.;” and “I can talk to someone about accessing

mental health resources for depression or anxiety issues in a kind and empathetic

manner.” The 20 self-efficacy items were then averaged to give each participant an

average that ranged from 0 to 7 for the self-efficacy domain.

Behavioral outcomes. There were 15 items included in the behavioral outcomes

section, all using a frequency count (N/A; No one I know has mental health issues, 0

times, 1 time, 2 times, 3 times, 4-5 times, 6+ times). Sample items assessing behavioral

outcomes included: “How often in the past three months have you recognized that

someone’s mental state (e.g., sadness, nervousness, uneasiness) could be indicative of a

diagnosable mental health issue?”, How often in the past three months have you engaged

someone in a conversation about the importance of professionally treating mental health

issues?” and “How often in the past three months have you asked someone who showed

signs/symptoms of a mental health issue if they are doing ‘okay’ or if they needed help?”

The 15 behavior items were then averaged to give each participant an average that ranged

from 0 to 7 for the self-efficacy domain.

 29

Validity Procedures

To test construct and discriminate validity, each domain of the measure was

correlated with scores from measures of similar constructs used in the research literature.

These measures include the knowledge subscale from the Question, Persuade, Refer

(QPR) institute (Quinett, 2009), a self-efficacy subscale used to assess gatekeeping

training (Wyman et al., 2008) and general measures assessing mental health of an

individual (Löwe, Unützer, Callahan, Perkins, & Kroenke, 2004; Spitzer, Kroenke,

Williams, & Lowe, 2006).

QPR knowledge scale. The QPR knowledge scale (Quinnett, 2007) is a 14-

question measure used to assess knowledge related to suicide prevention. This quiz-like

(e.g., true or false, multiple choice, multiple answer, etc.) measure is used to assess the

knowledge gained by participating in QPR training (Quinett, 2007; Wyman et al., 2008).

No psychometric properties are reported on this measure, but in the paper outlining the

theoretical underpinnings of QPR training, the items are stated to support key knowledge

required to be effective at responding as a gatekeeper (Quinett, 2007). Two items that

required selecting multiple responses were excluded due to errors in data collection.

Wyman and colleagues (2008) self-efficacy subscale. The self-efficacy subscale

was developed by Wyman and colleagues (2008) to evaluate the effectiveness of QPR

training in the residential housing center at varying colleges. This seven-item measure

uses a 7-point Likert scale containing confidence statements to evaluate perceived self-

efficacy of gatekeeping behaviors with higher scores suggesting more confidence.

Sample items include: “If a student experiencing thoughts of suicide does not
 30

acknowledge the situation, there is very little that I can do to help”; “If a student

contemplating suicide refuses to seek help, it should not be forced upon him/her.”

Cronbach’s alpha of the seven items was reported as .796 (Wyman et al., 2008) and .779

in the current sample.

Patient Health Questionnaire 9. The Patient Health Questionnaire 9 (PHQ-9;

Löwe et al., 2004) is a nine-item Likert questionnaire assessing depressive symptoms.

The measure asks participants to respond on a 4-point Likert scale (‘Not at all’ = 0, to

‘Nearly every day’ = 3) to being bothered by a variety of symptoms in the past two

weeks. Higher sum scores on the measure indicate higher levels of depression. Symptoms

included mirror diagnostic criteria for major depressive disorder, such as the following:

“Little interest or pleasure in doing things”; Feeling bad about yourself — or that you

are a failure or have let yourself or your family down.” Cronbach’s alpha of the scale

was reported to be .89 and test-retest reliability was reported at 0.84 (Kroenke, Spitzer, &

Williams, 2001). The Cronbach’s alpha in the current sample was .925. The measure also

has strong evidence for construct validity and criterion validity (Kroenke et al., 2001).

Generalized Anxiety Disorder 7. The Generalized Anxiety Scale 7 (GAD-7;

Spitzer et al., 2006) is a seven-item Likert questionnaire that assesses generalized

anxiety. The measure asks participants to respond on a 4-point Likert scale (‘Not at all’ =

0 to ‘Nearly every day’ = 3) to being bothered by a variety of symptoms in the past two

weeks. Higher sum scores on the measure indicate higher levels of anxiety. Symptoms

included in the measure mirror diagnostic criteria for generalized anxiety disorder and

include the following: “Feeling nervous, anxious, or on edge”; “Worrying too much
 31

about different things.” Cronbach’s alpha is reported at .92 (Spitzer et al., 2006) and was

.933 in the current sample. The scale is also reported as having good procedural validity

and diagnostic criterion validity (Spitzer et al., 2006).

Analytic Approach

Item Response Theory (IRT; Bond & Fox, 2015) was used to assess the

psychometric properties of the knowledge items from each of the three domains at the

item level and to provide scale scores for respondents. In addition, exploratory principal

components axis factor analysis was used to examine the underlying factor structure of

the self-efficacy and behavior items. Lastly, bivariate correlations were used to examine

reliability and construct validity of the MHAA-AT.

Item Response Theory

IRT evaluates and scores response data by simultaneously modelling item and

respondent characteristics, and has measurement advantages over classical test theory

(Ostini & Nering, 2005). The mathematical foundation of IRT models the probability of a

correct response to each item given the respondent's trait level (e.g. amount of declarative

knowledge in a specific domain) using logistic regression. It simultaneously and

interpedently estimates each respondents’ trait level and each items difficulty level on the

same latent dimension (Ostini & Nering, 2005).

A one-parameter (Rasch-type) dichotomous IRT model was fit to each set of 10

declarative knowledge items from each domain (i.e., Identifying, Locating, Responding)

data using the ltm package version 1.1-1(Rizopoulos, 2006) in the R software version
 32

3.5.2 (R Core Team, 2018). The relative appropriateness of 1-parameter model in each of

the domains was evaluated by examining item fit statistics, item parameter estimates

standard errors, and person item maps. Respondent knowledge scores were then

estimated for each subset of items separately. Descriptive characteristics for the three

knowledge score distributions were calculated. Lastly, analyses were conducted to

provide validity information on the declarative knowledge items within each domain.

Exploratory Factor Analysis

In order to determine the underlying factor structure of the self-efficacy and

behavior items of the MHAA-AT, a principal axis factor analysis was performed.

Principal axis factor analysis was selected because of the non-normal distribution of data,

smaller sample size, the need to account for shared variance, and to avoid any inflation of

estimates of variance accounted for (Costello & Osbourne, 2005). An oblique rotation

method was selected as suggested by Costello and Osborne (2005) due to being the more

accurate and possibly more reproducible solution than orthogonal rotation for social

science data. A scree plot test (Catell, 1966) identified breaking points of factors. Factors

with eigenvalues of one or higher were retained. Lastly, appropriateness of factor analysis

in regard to sample size was tested using SPSS Version 25.

Bivariate Correlations

To determine the convergent validity of the MHAA-AT, bivariate correlations

between the MHAA-AT and similar measures used to assess mental health awareness and

advocacy was completed (Rodgers & Nicewander, 1988).

 33

Results

Research Question #1

Research question #1 asked: What are the item and respondent characteristics of

the declarative knowledge items of the MHAA-AT?

Reliability and dimensionality analyses. For the purpose of data analyses,

responses to the declarative knowledge items were coded in a binary fashion (correct or

incorrect) with “I don’t know” responses recoded as incorrect. Due to the process-based

nature of the MHAA-AT, the 30 declarative items were broken into the three domains

(i.e., Identifying, Locating, and Responding domains) prior to analysis. Although

exploratory factor analysis (EFA) and scree plot interpretation suggest there are multiple

underlying factors in each domain, EFA is not an appropriate analytic strategy for binary

data because of the lack of continuous spread of data (Van der Eijk & Rose, 2015),

accordingly IRT was used to assess the unidimensionality and reliability of the

declarative knowledge items.

In the IRT framework, a one-parameter Rasch Model was applied to the data.

Mean square fit statistics (mean squared error, MSW infit and outfit; see Table 2.2)

suggested adequate unidimensionality of each of the domains (Bond & Fox, 2001).

Reliability statistics of each subdomain indicate fair internal consistency (see Table 2.3

for Cronbach’s alpha of each domain). IRT simultaneously estimated both item

difficulties (beta) and person-specific knowledge levels (theta) by maximum likelihood

(see Table 2.4). Figure 2.2 contains the Person-item maps which present the overall

spread of difficulty on items. Last, Table 2.5 contains the raw to scaled-score conversions
 34

Table 2.2

Three IRT Analyses: Item Fit Characteristics (MSQ) for MHAA-AT

Domain
───────────────────────────────────────
Identifying Locating Responding
─────────── ─────────── ───────────
Item Outfit Infit Outfit Infit Outfit Infit
1 0.90 0.86 1.22 a
1.10 0.98 0.97
2 0.92 0.92 0.87 0.89 1.18 1.00
3 1.41 a 1.22 a 0.79 0.87 0.85 0.89
4 0.94 0.97 0.94 0.97 0.99 0.90
5 1.07 1.09 1.05 1.00 1.04 0.98
6 1.14 1.16 1.91 a
1.22 a
1.40 a
0.99
7 0.73 0.77 0.84 0.87 0.96 0.99
8 0.83 0.85 0.88 0.91 0.95 0.94
9 0.70 0.80 1.07 1.00 0.73 0.83
10 1.45 a
0.86 0.74 0.83 0.88 0.83
a
Denotes MSQ-values outside the range of +/- 1.2 which may indicate
inappropriate fit for the selected item in the selected domain (Bond & Fox, 2001).

Table 2.3

Three IRT Analyses: Cronbach’s Alpha of the MHAA-AT

Declarative Knowledge Items
Domain
────────────────────────────
Excluding Item Identifying Locating Responding
All items included .62 .68 .60
1 .58 .68 .56
2 .58 .64 .60
3 .64 .64 .54
4 .59 .65 .59
5 .62 .68 .58
6 .63 .71 .61
7 .55 .63 .58
8 .57 .64 .59
9 .55 .65 .53
10 .63 .62 .54
 35

Table 2.4

Three IRT Analyses: Item Difficulty Estimates (Eta) and Conditional Probabilities for
MHAA-AT

Domain
─────────────────────────────────────────────────────
Identifying Locating Responding
──────────────── ──────────────── ────────────────
Item Eta Probability Eta Probability Eta Probability
1 1.25 .22 0.61 .34 -0.06 .52
2 1.01 .26 0.37 .40 1.43 .20
3 -0.55 .64 0.46 .38 -0.66 .66
4 -.30 .58 -0.04 .51 2.01 .12
5 1.35 .20 1.34 .19 0.59 .36
6 1.29 .21 1.63 .15 2.43 .08
7 .98 .27 -0.61 .66 0.76 .32
8 1.11 .24 0.21 .44 1.71 .16
9 -0.53 .63 -0.91 .73 -1.15 .76
10 4.10 .01 -0.45 .62 -0.76 .68
Note. Estimates are on the logit scale. Items that require more knowledge in order to answer correctly have
higher values and items that discriminate at a lower level of knowledge will have smaller values. The
probability is the chance of correctly responding to each item, conditional on having a knowledge level of
0.

Table 2.5

Three IRT Analyses: Raw to Scaled Scores Conversions for MHAA-AT

Domain
─────────────────────────────────────────────────────
Identifying Locating Responding
Raw ──────────────── ──────────────── ────────────────
score Est SE Est SE Est SE
1 -3.67 - -3.39 - -3.69 -
2 -2.70 1.10 -2.45 1.08 -2.70 1.11
3 -1.79 .85 -1.58 .83 -1.77 .87
4 -1.15 .76 -.98 .73 -1.10 .78
5 -.60 72 -.48 .69 -.53 .74
6 -.09 71 -.01 .68 .01 .73
7 .42 .73 .46 .70 .54 .74
8 .99 .79 .97 .74 1.11 .77
9 1.73 .94 1.58 .83 1.77 .86
10 - 1.58 .83 2.69 1.10
3.28 - - - - -
Note. The Est. denotes the estimated score for each sub-domain given a particular raw score. For example,
a raw score of 6 on the identifying domain equates to a converted score of .42.
 (A) Identifying Domain (B) Locating Domain (C) Responding Domain

Figure 2.2. Person-item maps for three separate IRT analyses: MHAA-AT.

36
 37

for each domain. In the following sections, each of the aforementioned domain specific

statistics are more thoroughly described.

Identifying domain. According to fit indices, the identifying domain is

sufficiently unidimensional (MSQ’s < 1.5; see Table 2.2) with the exception of item 1

and item 10. Due to the nature of these items (e.g., symptoms of depression and age of

onset of anxiety disorders) having face validity with the identifying domain, the authors

opted to keep these items. Internal consistency of the scale (Cronbach’s alpha = .62) was

adequate and was not highly influenced by the dropping of any particular item. The

person item map for the identifying domain [see Panel (A) of Figure 2.2] depicts the

spread of the data across the latent dimension of ‘identifying mental health issues.’ As is

seen in this figure, the questions tend to fall within +/- 1 on the logit scale suggesting

there is need for easier and more difficult questions to increase the variability of difficulty

of the items on the subscale. Last, due to the relative non-linear shape of the slope of

difficulty of items it is suggest that raw scores be converted to weighted scores in

interpretation (see Table 2.5).

Locating domain. The Locating domain fit indices suggest the domain is

sufficiently unidimensional (see Table 2.2). Items 1 and item 6 are slightly outside of the

range of acceptable MSQ, but were kept due to the MSQ guidelines proposed by Bond

and Fox (2001) being highly influenced by sample size and our sample size being

moderate. Internal consistency of the locating domain (Cronbach’s alpha = .68) was

moderate and was not highly influenced by the dropping of any particular item. The

person item map of the Locating domain [see Panel (B) of Figure 2.2] suggests more
 38

spread in difficulty of items compared to the Identifying domain, but there is still need for

more questions that cover the poles of difficulty. The Locating domain also depicted a

non-linear shape of the slope on difficulty of items suggesting that raw scores should be

converted to weighted scores in interpretation (see Table 2.5).

Responding domain. The fit indices of the Responding domain are also within

normal ranges and suggest the items as being unidimensional (see Table 2.2). Internal

consistency of the locating domain (Cronbach’s alpha = .60) was adequate. The internal

consistency ranges do drop below ranges of acceptability suggesting that more work is

needed on the scale to identify areas of “lumpiness” within the single factor. The person

item map [see Panel (C) of Figure 2.2] of the Responding domain shows the most spread

in difficulty of questions comparatively to the Identifying and Locating domains.

Increasing variability in difficulty of questions could strengthen the measure but are not

necessarily required to improve the utility of this domain. Lastly, the Responding domain

would best benefit from converting raw scores to weighted scores for interpretation (see

Table 2.5).

Research Question #2

Research question #2 asked: What is the underlying factor structure of the

MHAA-AT?

Because of the intent of creating a process-based assessment tool, the self-efficacy

items and behavior items were independently analyzed using principal axis factor

analysis. The Kaiser-Meyere-Olkin (KMO) measure of sampling adequacy (values closer

to 1.0 indicate appropriateness for factor analysis) and the Bartlett’s test of Sphericity (p
 39

values less than .05 indicate appropriateness for factor analysis; Cerny & Kaiser, 1977)

was used to determine if the underlying assumptions of principal axis factor analysis

were met. The internal structures of the self-efficacy and behavior items are explained in

the following sections and in Tables 2.6 and 2.7.

Self-efficacy items. The self-efficacy items had a KMO = .95 and Bartlett’s X2 =

3849.33, df = 190, p < .001 suggesting that the data was suitable for factor analysis. The

anti-image correlation matrices were all greater than .5, supporting the inclusion of each

Table 2.6

Self-Efficacy Items of the MHAA-AT: Communalities and Factor Loadings for Principal
Axis Factoring

Items Initial communalities Extraction communalities Final loadings

1 .709 .721 .708
2 .694 .701 .714
3 .684 .686 .791
4 .633 .618 .748
5 .686 .687 .810
6 .685 .679 .790
7 .685 .679 .769
8 .661 .655 .601
9 .438 .436 .578
10 .452 .501 .733
11 .577 .536 .792
12 .666 .624 .630
13 .489 .394 .520
14 .355 .318 .642
15 .626 .750 .670
16 .620 .643 .742
17 .556 .622 .687
18 .622 .615 .739
19 .526 .534 .691
20 .561 .562 .712
Eigen Value 10.49
% of Variance 50.58%
 40

Table 2.7

Behavior Items of the MHAA-AT: Communalities and Factor Loadings for Principal Axis
Factoring

Items Initial Communalities Extraction Communalities Final Loadings

1 .778 .769 .773
2 .838 .849 .819
3 .686 .653 .764
4 .735 .723 .784
5 .790 .767 .843
6 .708 .665 .819
7 .677 .586 .762
8 .760 .769 .766
9 .704 .697 .724
10 .582 .569 .672
11 .709 .629 .719
12 .796 .696 .840
13 .660 .582 .746
14 .615 .464 .608
15 .611 .526 .554
Eigen Value 8.86
% of Variance 56.96%

item in the factor analysis (Field, 2005). Initial outcomes from the self-efficacy items

without a fixed number of factors to extract, extracted 3 factors with eigenvalues higher

than 1. A scree plot test (Cattell, 1966) showed the breaking point after three factors. To

add clarity in a single factor structure, multiple manual factor extractions from 1 to 3

were performed. Based on recommendations from Costello and Osborne (2005; item

loadings above .30, no or few cross loadings, and no factors with fewer than three items,

p. 3), clarity of a single-factor remained clear. The one factor structure of the self-

efficacy items explained 50.58% of the variance in the MHAA-AT self-efficacy items

(see Table 2.6 for initial and extraction communalities and final loadings).
 41

Behavior items. The behavior items had a KMO = .92 and Bartlett’s X2 =

3840.04, df = 105, p < .001 suggesting that the data was suitable for factor analysis. The

diagonals of the anti-image correlation matrices for the behavior items were greater than

.5, supporting that the inclusion of each item in the factor analysis (Field, 2005). Initial

outcomes from the self-efficacy items without a fixed number of factors to extract,

extracted 2 factors with eigenvalues higher than 1. A scree plot test (Cattell, 1966)

showed the breaking point after two factors. To add clarity in a single factor structure,

multiple manual factor extractions from 1 to 2 were performed. Based on

recommendations from Costello and Osborne (2005) described above, the items from the

single factor remained clear. The one factor structure of the behavior items explained

56.96% of the variance in the MHAA-AT behavior items (see Table 2.7 for initial and

extraction communalities and final loadings).

Research Question #3

Research question #3 asked, “Does the MHAA-AT demonstrate strong reliability

and validity statistics”?

Reliability statistics for the MHAA-AT was assessed in multiple ways. First, the

internal consistency of the declarative knowledge items was assessed by breaking the

thirty items into each of the three domains (see IRT section). The Identifying domain,

Locating domain, and Responding domain each demonstrated adequate internal

consistency (Cronbach’s alpha = .62, .68, and .60 respectively; see Table 2.3). The

underlying factor-structure of the self-efficacy and behavior questions of the MHAA-AT

suggested that the items should not be separated into the three distinct domains and
 42

should instead be interpreted as one factor (i.e., self-efficacy items and behavior items).

The internal consistency of the self-efficacy and behavior items was good (Self-efficacy

items Cronbach’s alpha = .95; Behavior items Cronbach’s alpha = .95).

Construct validity of the MHAA-AT was assessed by completing bivariate

correlations (Carmbines & Zeller, 1979) between the micro-processes (declarative

knowledge, self-efficacy, and behavior items) of the MHAA-AT and psychometrically

sound measures commonly used to evaluate mental health awareness and advocacy (see

Table 2.8 for scoring). The declarative knowledge items were significantly correlated

with the QPR Knowledge subscale (r = .44, p < .01) and the Wyman and colleagues

(2008) self-efficacy subscale (r = .13, p < .05). Additionally, the MHAA-AT self-efficacy

subscale was positively correlated with the Wyman and colleagues (2008) self-efficacy

subscale (r = .51, p < .01). Lastly, the MHAA-AT subscales were also correlated with

one another (declarative knowledge positively correlated with self-efficacy; self-efficacy

positively correlated with behaviors), PHQ-9, and GAD-7 scores (see Table 2.9).

Discussion

Following preliminary development and appropriate analyses, we determined the

MHAA-AT is a reliable and valid assessment tool for assessing college students’

declarative knowledge, self-efficacy, and behaviors in identifying mental health issues,

locating evidence-based resources, and responding to mental health issues. IRT analyses

provide sufficient evidence that the declarative knowledge items within each of the three

domains is sufficiently univariate. Accordingly, the MHAA-AT declarative knowledge

 43

Table 2.8

Mean, Standard Deviations, Possible Range and Raw Percent Correct of Key Outcome
Variables at Pretest

Heading M SD Possible range Raw % corrected

MHAA-AT: Declarative Knowledge
Identifying domain 3.44 2.09 0-10 34.44
Locating domain 4.52 2.40 0-10 45.22
Responding domain 3.95 2.05 0-10 39.52

MHAA-AT
Self-efficacy 4.20 .66 1-6 NA
Behaviors .86 .87 0-5 NA

QPR knowledge 8.64 2.12 0-12 72.00

Self-efficacy (Wyman) 4.21 .66 1-7 NA
PHQ-9 7.83 6.80 0-27 NA
GAD-7 6.62 5.85 0-21 NA

items should be scored and interpreted using the number of correct responses on each

domain and then converted using the theta score adjustments found in Table 2.5.

Principal axis factor analyses demonstrated that a one factor model is appropriate for

interpretation of the self-efficacy (one factor accounted for 50.58% of the variance) and

behavior items (one factor accounted for 56.96% of the variance) of the MHAA-AT.

Higher scores on self-efficacy and behavior items indicate higher self-efficacy in each

domain and higher level of behaviors deemed appropriate for effective demonstration of

mental health literacy.

IRT analyses of Knowledge items indicated that the item difficulty appropriately

covers the range of knowledge exhibited by the sampled population, but with room for

general improvement. For instance, in the Identifying domain, item difficulty scores

range from -2 to 3 on the logit scale (see Figure 2.2), indicating that we may need to
Table 2.9

Correlations Among MHAA-AT Microprocess Items and Key Measures

Declarative OPR Self-efficacy

Measure knowledge Self-efficacy Behavior knowledge (Wyman et al.) PHQ9 GAD-7
MHAA-AT:
Declarative knowledge 1
Self-efficacy .31** 1
Behavior .10 .43** 1

QPR knowledge .44** -.01 -.01 1

Self-efficacy (Wyman) .13* .51** .26** -.02 1
PHQ-9 .02 .26** .49** .41 .13* 1
GAD-7 .06 .27** .46** .03 .09 .82** 1
Note. The acquiring declarative knowledge, building self-efficacy, and applying skills (behaviors) items are microprocess subscales from the
MHAA-AT domains of identifying, locating, and responding. The QPR knowledge scale is used with permission from the QPR Institute.
Self-Efficacy is a subscale from Wyman et al., 2008 on gatekeeping behaviors used with permission from authors. PHQ-9 assesses
depressive symptoms. GAD-7 assesses anxiety symptoms.
* p < .05 (2-tailed).
** p< .01 (2-tailed).

44
 45

consider developing questions that are less difficult (closer to -3), of average (between -1

and 0) and of moderate difficulty (between 1 and 3). The Locating domain, while more

spread across the logit scale on item difficulty, could benefit from questions that are

toward the two poles of difficulty (closer to -3 and 3 on the logit scale). The Responding

domain has the most spread in item difficulty, but might still benefit from questions that

are deemed toward the two poles of difficulty. That being said, the MHAA-AT is a

reliable measure of declarative knowledge for a college population. Internal consistency

coefficients ranged from acceptable to good. These findings are notable given the

inherent challenges to analyzing binary response choice measures.

The principal axis factor analysis supported the self-efficacy items and behavior

items as fitting a one factor model. Each item was retained with an appropriate factor

loading and demonstrated high internal consistency (Self-efficacy, Cronbach’s alpha =

.95; Behaviors, Cronbach’s alpha = .95). This finding was slightly surprising due to the

proposal of the three domains being three separate micro-processes within mental health

awareness and advocacy (see Figure 2.1). That being said, the overall macro-process

(e.g., knowledge leading to self-efficacy and self-efficacy leading to behaviors) proposed

via theory was initially supported by this study.

The MHAA-AT also demonstrated strong convergent validity (see Table 2.8). As

would be expected, the MHAA-AT declarative knowledge items were significantly

correlated (r = .44, p < .01) with the QPR knowledge items, a measure commonly used in

the literature base (Lipson et al., 2014; Mitchell et al., 2013; Reis & Cornell, 2008). The

MHAA-AT self-efficacy items were also significantly correlated with the Wyman and
 46

colleagues (2008) measure of self-efficacy in gatekeeping knowledge and behaviors (r =

.51, p < .01). The MHAA-AT self-efficacy items were also significantly correlated with

measurements of mental health issues (PHQ-9, r = .26, p < .01; GAD 7, r = .27, p < .01),

but in a direction that would not be expected (Bandura, 2005). These findings could be

due to personal exposure to mental health symptoms, treatments, and responses based on

personal experiences positively influencing more participant confidence in the material

assessed on the MHAA-AT.

Of particular interest was the MHAA-AT statistics that partially support the

assessment tool being process-based. Specifically, the MHAA-AT declarative knowledge

items were significantly correlated with the MHAA-AT self-efficacy items, but not the

behavior. This provides partial support for the process-based model in that as

participants’ knowledge increased so did their self-efficacy, but as Bandura (2005)

suggests, knowledge does not equate to action. Participants’ self-efficacy was

significantly correlated with their behavior. In short, the data seem to suggest that as

declarative knowledge increases, as does self-efficacy, but knowledge isn’t directly

linked to self-reported behaviors.

Implications for Future Research

While this study was the first attempt to use the MHAA-AT to assess college

students’ declarative knowledge, self-efficacy, and behaviors in identifying mental health

issues, locating empirically-based resources, and responding to mental health issues, it

effectively assessed desired outcomes in a process-oriented manner. This complements

 47

the work of O’Conner and Casey (2015) by providing an assessment device that is more

oriented to developmental theory and adequately measures mental health literacy.

Additional research on the MHAA-AT is needed to address the ability to demonstrate

strong psychometric properties in other populations (e.g., community members, teachers,

K-12 students, etc.) as mental health literacy programs have and are still being

implemented in varying contexts (Jorm, 2012). Further, future research using larger

sample sizes may add further clarity to the items in the measure that are most strongly

predictive of key behavioral outcomes important for interventionists.

Future research efforts should be directed toward replicating results found in this

study in similarly large and diverse samples that also use multiple data-points to help

identify stability of measured constructs (e.g., test-retest reliability). Lastly, future

research is needed to examine the ability of the MHAA-AT to identify participants’

growth over time to determine if it is an appropriate assessment tool for the evaluation of

interventions.

Implications for Interventionists

Of particular interest in this study is the focus of the MHAA-AT to help identify

the process by which participants are learning and applying the information. For instance,

if a student scores lower in particular areas of declarative knowledge (e.g., identifying

mental health issues) they were less likely to be confident in the same area and ergo less

likely to identify mental health issues in a variety of contexts. This is especially important

for interventionists wishing to tailor their interventions to most directly influence a

 48

specific type of outcome. Despite this being a first study addressing the psychometric

properties of the MHAA-AT, we believe that the results suggest the tool is ready for use

in larger, intervention-based research projects on college campuses to test its ability to

track change in participants. The assessment tool could also lend itself to informing

interventionists decisions on the most appropriate intervention to use.

Limitations

One limitation of the current study is the use of MTurk for data collection. While

this data collection approach is more commonly used in the social sciences, there are

intrinsic limitations, including participant inattention, associated with survey methods.

We attempted to address these limitations through the use of attention questions (see

methods section), but these threats cannot be fully accounted for on online data collection

methods. Additionally, participants were compensated via Amazon’s Mechanical Turk

which could have influenced their responses on the survey and the participants self-

selected into the study. Due to these issues, the sample is not fully representative of an

average population on a college campus in the United States. Secondly, the sample here

was higher than average in anxiety and depression (see Results). There is not clear

evidence in the extant literature describing how this might influence specific domains of

the measure, but some theory suggests that higher levels of depression can negatively

influence knowledge, self-efficacy, and behaviors (Bandura, 1989). Results of the study

also suggest that item difficulty needs additional work due to the spread of responses.

Future iterations could include additional questions that help address this limitation.
 49

Conclusion

The results of this study suggest the MHAA-AT has strong psychometric

properties in three domains of macro-processes, Identifying Locating, Responding, each

assessed via items of three micro-processes: acquiring knowledge; building self-efficacy,

and applying skills (behaviors). MHAA-AT was tested on a diverse college sample and is

appropriate for persons wishing to use a process-focused and theory driven approach for

assessing mental health advocacy and awareness. Additional research is needed to

determine if the MHAA-AT can be used in community populations and in intervention

studies to track change of participants.

References

Auerbach, R. P., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C., Cuijpers, P., … WHO
WMH-ICS Collaborators (2018, September 13). WHO World Mental Health
Surveys International College Student Project: Prevalence and Distribution of
Mental Disorders. Journal of Abnormal Psychology. Advance online publication.
http://dx.doi.org/10.1037/abn0000362

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental

disorder (5th ed.). Washington, DC: Author.

Bandura, A. (1982). Self-efficacy mechanism in human agency. American

Psychologist, 37(2), 122-147.

Bandura, A. (1989). Regulation of cognitive processes through perceived self-

efficacy. Developmental Psychology, 25(5), 365-380.

Bandura, A. (2005). The primacy of self‐regulation in health promotion. Applied

Psychology, 54(2), 245-254.

Bond, T., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in
the human sciences (3rd ed.). New York, NY: Routledge.
 50

Bond, K. S., Jorm, A. F., Miller, H. E., Rodda, S. N., Reavley, N. J., Kelly, C. M., &
Kitchener, B. A. (2016). How a concerned family member, friend or member of
the public can help someone with gambling problems: A Delphi consensus
study. BMC Psychology, 4(1), 6-18. doi: 10.1186/s40359-016-0110-y

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new
source of inexpensive, yet high-quality, data? Perspectives on Psychological
Science, 6(1), 3-5.

Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral
Research, 1(2), 245-276.

Center for Collegiate Mental Health. (2018, January). 2017 Annual report (Publication
No. STA 18-166). University Park, PA: Author.

Cerny, B. A., & Kaiser, H. F. (1977). A study of a measure of sampling adequacy for
factor-analytic correlation matrices. Multivariate Behavioral Research, 12(1), 43-
47.

Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis:
Four recommendations for getting the most from your analysis. Practical
Assessment, Research and Evaluation, 10(7), 1-9.

Dahlberg, K. M., Waern, M., & Runeson, B. (2008). Mental health literacy and attitudes
in a Swedish community sample-Investigating the role of personal experience of
mental health care. BMC Public Health, 8(1), 8-18. doi: 10.1186/1471-2458-8-8

Eisenberg, D., Golberstein, E., & Gollust, S. E. (2007a). Help-seeking and access to
mental health care in a university student population. Medical Care, 45(7), 594-
601.

Eisenberg, D., Golberstein, E., & Hunt, J. B. (2009). Mental health and academic success
in college. The BE Journal of Economic Analysis & Policy, 9(1). Retrieved from
doi:10.2202/1935-1682.2191

Eisenberg, D., Gollust, S. E., Golberstein, E., & Hefner, J. L. (2007b). Prevalence and
correlates of depression, anxiety, and suicidality among university
students. American Journal of Orthopsychiatry, 77(4), 534-542.

Eisenberg, D., Hunt, J., & Speer, N. (2013). Mental health in American colleges and
universities: Variation across student subgroups and across campuses. The
Journal of Nervous and Mental Disease, 201(1), 60-67.

Field, A. P. (2005). Discovering statistics using SPSS. London, UK: Sage.

Jorm, A. F. (2012). Mental health literacy: Empowering the community to take action for
 51

better mental health. American Psychologist, 67(3), 231-243. doi: 10.1037/

a0025957

Jorm, A. F., Kitchener, B. A., Sawyer, M. G., Scales, H., & Cvetkovski, S. (2010).
Mental health first aid training for high school teachers: A cluster randomized
trial. BMC Psychiatry, 10(1), 51-63. doi: 10.1186/1471-244X-10-51

Jorm, A. F., Korten, A. E., Jacomb, P. A., Christensen, H., Rodgers, B., & Pollitt, P.
(1997). “Mental health literacy”: A survey of the public's ability to recognize
mental disorders and their beliefs about the effectiveness of treatment. Medical
Journal of Australia, 166(4), 182-186.

Jorm, A. F., Nakane, Y., Christensen, H., Yoshioka, K., Griffiths, K. M., & Wata, Y.
(2005). Public beliefs about treatment and outcome of mental disorders: a
comparison of Australia and Japan. BMC Medicine, 3(1), 12-26. doi:
10.1186/1741-7015-3-12.

Jung, H., von Sternberg, K., & Davis, K. (2016). Expanding a measure of mental health
literacy: Development and validation of a multicomponent mental health literacy
measure. Psychiatry Research, 243, 278-286.

Kadison, R., & DiGeronimo, T. F. (2004). College of the overwhelmed: The campus
mental health crisis and what to do about it. Growth: The journal of the
Association for Christians in Student Development, 77-84.

Kitzrow, M. A. (2009). The mental health needs of today's college students: Challenges
and recommendations. NASPA Journal, 46(4), 646-660.

Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ‐9: Validity of a brief
depression severity measure. Journal of General Internal Medicine, 16(9), 606-
613.

Lipson, S. K., Speer, N., Brunwasser, S., Hahn, E., & Eisenberg, D. (2014). Gatekeeper
training and access to mental health care at universities and colleges. Journal of
Adolescent Health, 55(5), 612-619.

Löwe, B., Unützer, J., Callahan, C. M., Perkins, A. J., & Kroenke, K. (2004). Monitoring
depression treatment outcomes with the Patient Health questionnaire-9. Medical
Care, 42(12), 1194-1201.

Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical
Turk. Behavior Research Methods, 44(1), 1-23.

Mitchell, S. L., Kader, M., Darrow, S. A., Haggerty, M. Z., & Keating, N. L. (2013).
Evaluating question, persuade, refer (QPR) suicide prevention training in a
college setting. Journal of College Student Psychotherapy, 27(2), 138-148.
 52

National Institute of Mental Health (2013). Mental Health Information: Any mental
illness statistics. Retrieved from:
https://www.nimh.nih.gov/health/statistics/index.shtml.

O’Connor, M., & Casey, L. (2015). The Mental Health Literacy Scale (MHLS): A new
scale-based measure of mental health literacy. Psychiatry Research, 229(1-2),
511-516.

Ostini, R., & Nering, M. L. (2005). Polytomous item response theory models. Newbury
Park, CA: Sage.

Quinnett, P. (1995). QPR: Ask a Question, Save a Life. The QPR Institute, Spokane,
Washington. Retrieved from www.qprinstitute.com

Quinnett, P. (2007). QPR gatekeeper training for suicide prevention: The model,
rationale, and theory. Retrieved from https://qprinstitute.com/research-theory

Reavley, N. J., Morgan, A. J., & Jorm, A. F. (2014). Development of scales to assess
mental health literacy relating to recognition of and interventions for depression,
anxiety disorders and schizophrenia/psychosis. Australian & New Zealand
Journal of Psychiatry, 48(1), 61-69.

Reis, C., & Cornell, D. (2008). An evaluation of suicide gatekeeper training for school
counselors and teachers. Professional School Counseling, 11(6), 386-394. doi:
10.1177/2156759X0801100605

Rizopoulos, D. (2006). ltm: An R package for latent variable modeling and item response
theory analyses. Journal of Statistical Software, 17(5), 1-25.

Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen ways to look at the correlation
coefficient. The American Statistician, 42(1), 59-66.

R Core Team (2018). R: A language and environment for statistical computing. Vienna,
Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-
project.org/

Sheeran, P., Maki, A., Montanaro, E., Avishai-Yitshak, A., Bryan, A., Klein, W. M., &
Rothman, A. J. (2016). The impact of changing attitudes, norms, and self-efficacy
on health-related intentions and behavior: a meta-analysis. Health Psychology,
35(11), 1178-1218.

Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief measure for
assessing generalized anxiety disorder: the GAD-7. Archives of Internal
Medicine, 166(10), 1092-1097.

Substance Abuse and Mental Health Services Administration. (2018). Key substance use
 53

and mental health indicators in the United States: Results from the 2017 National
Survey on Drug Use and Health (HHS Publication No. SMA 18-5068, NSDUH
Series H-53). Rockville, MD: Center for Behavioral Health Statistics and Quality,
Substance Abuse and Mental Health Services Administration. Retrieved from
https://www.samhsa.gov/data/

Tompkins, T. L., Witt, J., & Abraibesh, N. (2010). Does a gatekeeper suicide prevention
program work in a school setting? Evaluating training outcome and moderators of
effectiveness. Suicide and Life-threatening Behavior, 40(5), 506-515.

Van der Eijk, C., & Rose, J. (2015). Risky business: factor analysis of survey data-
assessing the probability of incorrect dimensionalisation. PloS one, 10(3),
e0118900. doi: 10.1371/journal.pone.0118900

Wyman, P. A., Brown, C. H., Inman, J., Cross, W., Schmeelk-Cone, K., Guo, J., & Pena,
J. B. (2008). Randomized trial of a gatekeeper program for suicide prevention: 1-
year impact on secondary school staff. Journal of Consulting and Clinical
Psychology, 76(1), 104-115. doi: 10.1037/0022-006X.76.1.104

Xiao, H., Carney, D. M., Youn, S. J., Janis, R. A., Castonguay, L. G., Hayes, J. A., &
Locke, B. D. (2017). Are we in crisis? National mental health and treatment
trends in college counseling centers. Psychological Services, 14(4), 407-415.
 54

CHAPTER 3

STUDY 2: MENTAL HEALTH AWARENESS AND ADVOCACY (MHAA): AN

EVALUATION OF A COLLEGE-BASED MENTAL HEALTH LITERACY

CURRICULUM 2

Introduction

Mental health issues (e.g., depression, anxiety, bipolar, schizophrenia, etc.) are a

common concern on college campuses currently affecting approximately one in seven

students with depression and anxiety diagnoses being the most common (ACHA, 2015;

Center for Collegiate Mental Health, 2018). These issues negatively influence students’

educational experience, often leading to decreased GPA and graduation rates, and

sometimes eliciting suicide ideation (Center for Collegiate Mental Health, 2018;

Eisenberg, Hunt, & Speer, 2013). College-based mental health services effectively treat

most mental health issues, but the sheer number of students now seeking services often

surpasses the capacity of these resources (Auerbach et al., 2018; Center for Collegiate

Health, 2017; Kitzrow, 2009). Many universities now draw upon health education

programs that provide mental health education to larger quantities of the student body to

try and prevent mental health issues from developing or worsening (Zalsman et al.,

2016).

Mental health literacy (Jorm et al., 1997) is a common mental health education

approach used internationally to prevent the development and worsening of mental health

2
Contributing authors are: Elizabeth Fauth and Ryan Sedall.
 55

issues. In other words, while some mental health interventions, like therapy, target

decreasing individuals’ psychological distress via one-on-one therapy or group formats,

mental health literacy targets decreasing mental health issues through earlier detection

and prevention of the problem worsening via education. Mental health literacy covers six

key content areas: (1) the ability to recognize specific disorders or psychological distress;

(2) knowledge and beliefs about risk factors and causes of mental health issues; (3)

knowledge and beliefs about self-help interventions; (4) knowledge and beliefs about

professional help available; (5) attitudes which facilitate recognition and appropriate

help-seeking; and (6) knowledge of how to seek mental health information (Jorm et al.,

1997). For the current study, these six content areas are grouped into three main

processes: (a) identifying mental health issues; (b) locating evidenced-based resources;

and (c) responding to mental health issues.

Mental health literacy programs have demonstrated positive increases in

participants’ knowledge and self-efficacy related to identifying and responding to mental

health issues in a variety of populations (Hanisch et al., 2016; Mehta et al., 2015). There

is not, to our knowledge, a college-based curriculum that is formatted as a course-for-

credit, and empirically evaluated as being effective in improving mental health literacy

and related outcomes. Having mental health literacy curriculum included, for credit, as

part of a social science degree requirement or general education elective may help

motivate more students to take the course, due to it fulfilling part of their degree

requirements, and thereby offer another effective way to disseminate a prevention

program. This format may also allow more depth and more content covered than what is
 56

possible in a workshop format. The current study briefly reviews the literature explaining

the three main processes of mental health literacy (i.e., identifying mental health issues,

locating empirically based resources, and responding to mental health issues) and the

effectiveness of these approaches. We then outline the theoretical approach used to create

and evaluate a novel Mental Health Awareness and Advocacy curriculum, appropriate to

offer as a credit earning course in a college setting.

Identifying Mental Health Issues

Community studies have examined individuals’ ability to identify mental health

issues in Australia, the United Kingdom, Canada, Japan, Sweden, and the United States

(Dahlberg, Waern, & Runeson, 2008; Jorm et al., 1997; Nakane, et al., 2005). In a

prominent study on identifying mental health issues using an Australian sample,

approximately 39% of participants could identify depression while only 27% of

participants could identify schizophrenia (Jorm et al., 1997). This lack of recognition

seems to mirror other populations with a more recent study showing that less than 50% of

participants could identify depression in Japan and Sweden (Dahlberg et al., 2008; Jorm,

et al., 2005). In a United States sample, 58% of participants could identify a child with

depression (Pescosolido et al., 2008). Adolescent participants in similar studies

examining mental health literacy were more likely to label mental health issues as a

common life stressor or simply being sad (Burns & Rapee, 2006). While it is encouraging

that participants can identify that there is a problem, when these mental health issues are

not identified as a serious, diagnosable condition people are less likely to receive

professional help (Goldney, Fisher, & Wilson, 2001). Adding to this, mental health
 57

literacy of college-based populations has been found to mirror that of larger populations

(Furnham, Cook, Martin, & Batey, 2011). Because of the relatively low level of mental

health literacy in varying populations, mental health literacy programs aim to increase an

individual’s ability to recognize a diagnosable mental health issue, specifically the most

common issues of depression and anxiety, to help increase the rate by which individuals

seek help to prevent problems from developing or worsening (Jorm, 2012).

In a review of programs promoting identification of mental health issues, four

program types were identified as being effective (with three being pertinent to the skill

set of identifying mental health issues; Kelly, Jorm, & Wright, 2007). These four types of

programs include: (1) whole of community campaigns; (2) community campaigns that

are targeted toward a youth audience; (3) school-based interventions that help teachers,

staff, and students improve identification skills, help-seeking behaviors, or resilience; and

(4) programs training to better intervene in a mental health crisis (Kelly et al., 2007).

Whole of community campaigns do not seek to target a specific demographic of

participants and instead try to focus on improving the entire community’s ability to

identify mental health issues (Dumesnil & Verger, 2009; Francis, Pirkis, Dunt, Blood, &

Davis, 2002). Specific strategies implemented in whole of community campaigns

typically target mass media campaigns due to their cost effectiveness and their ability to

scale the program (Francis et al., 2002). More targeted approaches tend to focus on

specific age groups (e.g., adolescents; Battaglia, Coverdale, & Bushong, 1990; Pinto-g52

Foltz, Logsdon, & Myers, 2011). These programs seek to inform educators and equip

them with a specific skillset to increase identification of mental health issues, or the
 58

programs inform students to aid in prevention of mental health issues. These programs

are typically delivered in course formats that vary from a one-day seminar to a series of

activities over a week duration.

In systematic and narrative reviews of these various approaches to educational

programs, results indicate that identification of mental health issues can be improved

(Francis et al., 2002; Jorm, 2012). For instance, a study evaluating the beyondblue

curriculum in Australia found that participants engaging in the curriculum reported a

greater understanding of depression, effective treatments for depression, and more

openness toward talking about depression (Jorm, Christensen, & Griffiths, 2006). This

curriculum used varying approaches from whole of community orientations (e.g., public

service announcements, newspaper articles, internet articles, etc.) but also recruited high

profile speakers to talk about depression in varying settings (Hickie, 2004). More current

studies have evaluated a curriculum titled In Our Own Voice that uses the experiences of

high school students to educate fellow students about depression and other mental health

issues (Pinto-Foltz et al., 2011). Results from this study indicated that students improved

their identification of mental health issues at four and six-week follow ups (Pinto-Foltz et

al., 2011). While these programs are often effective, considering age and education level

of participants being evaluated (Reavley, McCann, & Jorm, 2012) is crucial for designing

a highly effective program. For instance, in a study of an Australian college students, \

age and educational status was positively correlated with correct identification of mental

health issues (Reavley et al., 2012). Once a mental health issue is identified, it is

important for individuals to be able to effectively locate empirically-based resources to

 59

refer those experiencing mental health issues to for effective treatment.

Locating Empirically Based Resources

In young people, several factors facilitate or hinder help-seeking behaviors to

address mental health issues. These factors include, but are not limited to, mental health

stigma, perceived severity of the problems, understanding of how to receive professional

help, and the perceived effectiveness of treatments (Gullliver, Griffiths, & Christensen,

2010). In college-aged populations, similar results have been found regarding barriers

prohibiting help seeking behaviors (Czyz, Horwitz, Eisenberg, Kramer, & King, 2013).

College students also experience self-stigma, lower perceived benefits of treatment, and

self-disclosure of their mental health issues as potential barriers to help seeking.

Additionally, these students often do not think their problem is serious enough for

professional treatment (Czyz et al., 2013; Nam et al., 2013). Lastly, according to one

meta-analysis, college students still perceive seeking professional help for mental health

issues very negatively, decreasing the likelihood that they seek out services (Mackenzie,

Erickson, Deane, & Wright, 2014).

Programs addressing locating empirically supported resources are often

implemented in whole of community campaigns and programs targeting specific

demographic groups (Francis et al., 2002). These programs raise awareness of specific

mental health issues, the effects they have on the public, and how to access professional

help. At times, programs addressing locating evidence-based resources use the

experiences of individuals that have experienced a mental health issue. By doing this,

these programs communicate to others what their experience was like and then try to
 60

motivate participants to shape their possibly negative beliefs (Pickett-Schenk, Cook, &

Laris, 2000). Other programs are more targeted and use community members to facilitate

group communication in a psychoeducation format (Pickett-Schenk et al., 2000). These

programs seem to address the goal of educating individuals about mental health issues

and effective treatment options. More specifically, they help address the negative stigma

of participants and encourage use of high-quality resources to treat mental health issues.

The programs addressing locating evidence-based resources consistently emphasize the

need to increase awareness and empathy surrounding mental health issues and the use of

effective treatments. They do not, however, consistently educate individuals about the

complexities of the healthcare system in relation to mental health issues and how to

effectively access help (Francis et al., 2002). There is considerable need to help students

on college campuses identify specific resources outside of the college community that

effectively treat these issues. This becomes increasingly important when considering the

ever-changing insurance market in the United States (Eisenberg, Golberstein, & Gollust,

2007).

According to reviews on programs addressing locating empirically based

resources, the most common methods used to increase access to high-quality resources

are mass media campaigns (Francis et al., 2002). One mass media campaign implemented

in Australia called the Community Awareness Program sought to reduce stigma and raise

awareness of mental health issues (Evans Research, 1999). This program used media

activities, television commercials, and informational brochures. The review of this

program focused primarily on the informational brochures and results of the study
 61

indicated that these brochures were highly useful for community members as ranked by

general practitioner doctors. Additionally, the study found that many community

members (76% of those surveyed) had seen the brochures and engaged with them in

some way (Evans Research, 1999). The results of this study did not, however, indicate

whether the brochures helped improve the ability to locate evidence-based resource and

then successfully access them.

Whole of community approaches have also been evaluated in the form of media

campaigns in the educational setting (Wolff, Pathare, Craig, & Leff, 1996a, 1996b,

1996c). In one educational campaign in the United Kingdom, three unique elements were

used to influence participants’ concept of advocacy: a social component, a dyadic

component, and a mixed component that included a formal reception and informal

discussion meetings. Findings from this intervention reported that 91% of participants (N

= 215) sought more information about mental health issues after completing the

educational course, but only one third of the participants accepted additional information

related to mental health issues from the course instructors when offered (Wolff et al.,

1996c). Additionally, participants in the study reported an increase in behavioral

intentions (e.g., talking about mental health issues) after completing the educational

program. This program suggests that talking about mental health issues in a dyadic

component that is complemented by social connection increases participants’ willingness

to talk and advocate for more resources related to mental health issues.

Educational programs have also been evaluated in the community college setting

in Chicago (Holmes, Corrigan, Williams, Canar, & Kubiak, 1999). The course, titled,
 62

Severe Mental Illness and Psychiatric Rehabilitation, addressed schizophrenia rather than

depression or anxiety. Students participating in the course completed a series of tasks

including lectures about causes, treatments, and rehabilitation of individuals with

schizophrenia (Holmes et al., 1999). Results of the study indicated that students that

participated in the intervention improved their benevolence and social restrictiveness

attitudes, but the study did not assess specific behavioral outcomes. The results reported

in this study were also influenced by participants’ prior knowledge and exposure to

mental health issues. Other studies evaluating school-based approaches have also

suggested their relative effectiveness (Battaglia et al., 1990; Pinto-Foltz et al., 2011). In

an evaluation study of a program used in a United States high school, results of one

program reported that students were more likely to state they would seek treatment for

mental health issues after receiving a talk by trained psychiatrists (Battaglia et al., 1990).

Because having experience with mental health issues seems to positively

influence program results, the National Alliance on Mental Illness (NAMI) has

implemented support programs run by community members who, themselves, have

experienced past mental health issues (NAMI, 2017). The Journey for Hope program

originally implemented in 1993 (Pickett-Schenk et al., 2000) and now implemented in

updated programs with differing names (NAMI, 2017) draws upon the experience of

those that have experienced mental health issues. Through psychoeducation on healthy

caregiving behaviors for those with mental health issues, combined with group

participation, the Journey for Hope program evaluations report positive results. For

instance, of the 424 program participants evaluated, a large majority indicated that the
 63

program had helped increase their knowledge of causes and treatment of mental illness

(86%), their knowledge of the mental health care system (86%) and their overall morale

(79%; Pickett-Schenk et al., 2000). This program concludes that design features such as

drawing upon experiences of those with mental health issues and fostering support

between group members are important to include in future interventions.

Responding to Mental Health Issues

Several studies indicate that college students often do not respond to mental

health issues because they do not possess the knowledge of how to effectively help their

peers (Eisenberg, Hunt, & Speer, 2012). Additionally, students often do not recognize

that a mental health issue is serious enough for professional attention, prohibiting their

response (Hunt & Eisenberg, 2010). This lack of education could largely be due to

schools not providing the appropriate resources. In a national survey of over 19,000

college students, approximately 46% of students stated they have never received

information about mental health issues from their school, but 52% of these same students

indicated they would want information related to mental health issues from their school

(ACHA, 2015). To address this discrepancy, schools across the United States and other

countries have begun to establish gatekeeper trainings more systematically.

The most common educational approach to increasing students’ ability to respond

to mental health issues are called Gatekeeper trainings. The most common Gatekeeper

trainings identified in the literature and used on college campuses are the Question,

Persuade, Refer (QPR) gatekeeper training (Quinnett, 2007), Mental Health First Aid

(MHFA; Kitchener & Jorm, 2002), and more professional, therapy-based programs
 64

(Conley, Durlak, & Kirsch, 2015). These programs share similarities in that they typically

target specific demographics rather than focusing on whole of community strategies.

Both QPR and MHFA programs provide a component of education about mental health

issues, skills to effectively assess individuals’ need for more treatment, and how to

effectively refer people to help. These programs are more suited to educating large

populations, in part because the instructor does not need clinical training (Quinnett, 2007;

Kitchener & Jorm, 2002). The more therapy-based courses typically implement cognitive

behavioral therapy (CBT) techniques to help improve students’ skills in handling their

own mental health (Conley et al., 2015), and while these programs are also effective, they

are not as adaptable to educational course formats, due to scaling concerns (e.g., having

therapists to run courses, funding to provide specialized training for each instructor, etc.),

Web-based prevention and intervention programs are being widely used on

college campuses, especially when trying to reach more rural students (Davies, Morriss,

& Glazebrook, 2014; Kern, Hong, Song, Lipson, & Eisenberg, 2018; Kauer, Mangan, &

Sanci, 2014; Lancaster et al., 2014). These programs often implement similar strategies

as QPR and MHFA in that they educate students about mental health issues, how to ask

assessment-based questions, and how to refer others to evidence-based resources

(Lancaster et al., 2014). Many of these programs are demonstrating promising effects in

increasing students’ ability to respond to mental health issues (Davies et al., 2014; Kauer

et al., 2014), however online programs face challenges in retention. For instance, a meta-

analysis of interventions (online and in-person) on college campuses found that some

online programs are ineffective, and interventions that are effective typically have
 65

supervisory oversight of skill development (Conley et al., 2015). Oversight of skill

development can also be considerably harder to deliver in an online format and could

potentially decrease the implementation and effectiveness of online programs. Because of

this, it is important to better understand if skill development can be facilitated via online

course formats that are often limited to less immediate feedback on specific skills.

Question, Persuade, Refer (QPR) gatekeeper training is based upon the idea that

there are important gatekeepers, or people that come into regular contact with at-risk

individuals, that can help prevent mental health issues from worsening (Quinnett, 2007).

QPR teaches participants to ask appropriate questions regarding suicidality, persuade an

individual that is currently suicidal to get help, and learn of appropriate referral sources

for an individual with these programs. In teaching these three skills, QPR attempts to

complete four goals to help decrease suicides: 1) early recognition of suicide warning

signs; 2) directly asking people if they are suicidal which may immediately decrease

anxiety and enhance protective factors for an individual with a mental health issue; 3)

increase early referrals to professional resources and 4) receive early professional

assessment and referrals to therapy (Quinnett, 2007). By using this program, both

secondary education participants and college participants have seen an increase in their

knowledge, skills, and behavioral outcomes (e.g., referring a suicidal individual to a

professional, having conversations about suicide risk, etc.) related to gatekeeping

behaviors.

The Saving and Empowering Young Lives in Europe project, a project designed to

help evaluate the effectiveness of school-based suicide prevention programs, evaluated

 66

the effectiveness of QPR in secondary education populations (Wasserman et al., 2015).

The study implemented a large, multi-site study that included 2,209 participants that

showed no significant effects for decreasing actual suicide attempts in comparison to the

control group (Wasserman et al., 2015). There are, however, studies that report QPR

helps increase the knowledge and self-efficacy of secondary education staff participating

in the program in relation to their ability to respond to someone experiencing suicidality

(Tompkins, Witt, & Abraibesh, 2010; Wyman et al., 2008). In a study of secondary

education staff, QPR training increased self-reported knowledge, appraisals of efficacy,

and service access (Wyman et al., 2008). These results indicate that it might be harder to

evaluate a direct effect between prevention programs and decreasing actual suicides and

that adults make more effective gatekeepers than secondary students themselves. These

programs may also increase important prevention behaviors like education and

communication, but not directly decrease suicide attempts immediately.

QPR programs implemented in the college use a 90-minute lecture related to

warning signs of suicide and other mental health issues and how to access appropriate

resources (Mitchell et al., 2013). An evaluative study of college based QPR using a

pretest/posttest quasi-experimental design with a six-month follow-up indicated that

students participating in QPR significantly improved their knowledge of suicide

prevention and skills related to responding to mental health issues. These skills revolved

around identifying warning signs, how to ask about suicide, knowing how to get help,

and having a knowledge of local resources (Mitchell et al., 2013). These promising

results indicate that college students can improve important outcomes related to
 67

responding to mental health issues.

MHFA has also shown promising results in a variety of settings at improving

similar outcomes. MHFA helps participants increase understanding of mental health

issues and how to appropriately respond to these issues using resources found in their

community (Kitchener & Jorm, 2006). The program provides training in four, three-hour

sessions (twelve total hours) by a trained instructor (1-week of training prior to teaching

the course). The MHFA program focuses on five goals: (1) assess risk of suicide or harm;

(2) listen nonjudgmentally; (3) give reassurance and information, (4) encourage the

person to get appropriate professional help; and (5) encourage self-help strategies

(Kitchener & Jorm, 2006). As MHFA was first implemented as a whole-of-community

program, there have been numerous studies evaluating the effectiveness of the program in

community samples (Kitchener & Jorm, 2006). In a meta-analytic review including

fifteen studies, results indicated that MHFA increases participants’ knowledge regarding

mental health, decreases their negative attitudes, and increases supportive behaviors (e.g.,

self-report of referrals, self-report of likelihood of referring an individual) toward

individuals with mental health problems (Hadlaczky, Hokby, Mkrtchian, Carli, &

Wasserman, 2014).

There have also been various studies of MHFA in college populations supporting

the effectiveness of this program. For example, MHFA has been used to train residence

hall leaders at varying universities (Lipson, Speer, Brunwasser, Hahn, & Eisenberg,

2014). In a study of 32 colleges and universities, the MHFA training was implemented by

instructing residence hall advisers how to identify and respond to mental health issues to
 68

help decrease the negative effects of mental health issues in the college population

(Lipson et al., 2014). More specifically, the study sought to examine service utilization,

knowledge and attitudes about services, self-efficacy, intervention behaviors, and mental

health symptoms. Results from the study indicated that the intervention increased

residence hall advisors’ self-perceived knowledge and self-perceived ability to identify

students in distress (Lipson et al., 2014). There were not, however, any observed effects

in utilization of mental health care in the student communities where the training took

place (Lipson et al., 2014).

Mental Health Awareness and Advocacy

Curriculum

As is evidenced by the above literature, programs seeking to improve participants’

ability to identify mental health issues, locate evidence-based resources to treat these

issues, and to respond effectively to mental health issues are effective in a variety of

settings, including higher education (Tompkins et al., 2010; Wyman et al., 2008). More

specifically, these programs have been effective at improving students’ declarative

knowledge, self-efficacy, and perceived ability to respond appropriately to mental health

issues, primarily suicidality. Given these strengths, a college-based curriculum seeking to

improve students’ ability to respond to mental health issues should implement strategies

that have already been supported as being effective in a process-based manner.

The MHAA curriculum is made up of three progressive domains that emphasize

the process of mental health literarcy: (1) the ability to identify signs and symptoms of

mental health issues (Identifying domain); (2) the ability to identify and access evidence-
 69

based mental health resources (Locating domain); and (3) the ability to effectively and

appropriately respond to mental health issues (Responding domain; see Figure 3.1). The

curriculum emphasizes the overall process of mental health literacy by breaking these

three domains into three micro-processes: acquiring knowledge (knowledge), building

self-efficacy (self-efficacy), and applying skills (behaviors). What is unique to the

MHAA curriculum is this process-based approach, its format (course-for-credit design)

and the use of two theoretical models: 1) the health belief model (Becker, 1974) and 2)

social cognitive theory (Bandura, 2005) to guide the creation of the curriculum and to

evaluate its effectiveness.

Identifying Domain

a) Declarative Knowledge
b) Self-Efficacy
c) Behaviors

Locating Domain Responding Domain

a) Declarative Knowledge a) Declarative Knowledge

b) Self-Efficacy b) Self-Efficacy
c) Behaviors c) Behaviors

Note. The circles represent the macroprocesses. Microprocesses are listed within each macroprocess:
Declarative knowledge refers to the microprocess of acquiring knowledge; Self-efficacy refers to building
self-efficacy, and behaviors refers to applying skills.

Figure 3.1. Process-based model of mental health awareness and advocacy curriculum.

The health belief model. The health belief model (Becker, 1974) seeks to explain
 70

factors that influence an individual’s likelihood of preventing, screening, or controlling

an illness. Using the health belief model, examining how students respond to mental

health issues can be better understood by the following factors: perceived susceptibility,

severity, benefits, barriers, and cues to action (Champion & Skinner, 2008). Perceived

susceptibility is defined as an individual’s belief that there is a possibility of contracting

an illness. Perceived severity describes an individual’s concern over the seriousness of

consequences, both physically and socially, if they contract the illness. Perceived barriers

explain the possible negative effects of acting to prevent or respond to the illness. Lastly,

cues to action, a concept not empirically studied, was originally proposed as an external

event (e.g., media campaign, class, meeting) that would facilitate action.

Social-cognitive theory. In later iterations of the health belief model

(Rosenstock, Strecher, & Becker, 1988), the concept of self-efficacy (Bandura, 2005)

from social cognitive theory was introduced as an important construct to better explain an

individual’s likelihood of responding to a health issue. Social-cognitive theory posits that

responding to health issues can be better understood by considering environmental

factors, individual factors, and individual behavior (triadic reciprocal determinism

(Bandura, 1978). Self-efficacy, an individual factor defined by Bandura (1997), explains

the individual’s belief that they can successfully complete a behavior that is requisite to

produce a desired outcome. This construct that has been extensively researched and

supported as being an important factor in predicting behavior (Bandura, 1982). More

specifically, social cognitive theory argues that it is important to understand students’

intentionality, forethought, self-reactiveness, and self-reflectiveness while also

 71

considering their abilities in responding, past successes and cognitive reinforcements. By

gauging each of these individual factors, a curriculum can better meet the needs of

students in college on an individual level. This is especially important when considering

that the effectiveness of the health belief model is largely dependent on responding and

influencing the perceptions of an individual. By using the health belief model as an

overarching framework complemented by social cognitive theory, the MHAA curriculum

better fits the needs of students and help facilitate responses to mental health issues.

The Present Study

The primary purpose of this study was to examine the effectiveness of the Mental

Health Awareness and Advocacy curriculum in improving students’ microprocesses of

acquiring knowledge, building self-efficacy, and applying skills/behaviors in broader

macroprocess domains of identifying mental health issues, locating evidence-based

resources, and responding to mental health issues. The study addresses the following

research questions:

RQ 1: Do students that participate in the MHAA curriculum improve on specified

outcomes in comparison to the control group when accounting for students’ key

demographic factors?

RQ 2: Do treatment effects of the MHAA curriculum vary by type of course

delivery (face-to-face vs. online) when accounting for students’ key demographic

factors?

RQ 3: Do students that participate in MHAA improve in self-reported mental

health assessments (i.e., depressive and anxiety symptoms) in comparison to the control
 72

group when accounting for students’ key demographic factors?

Method

Sample

Participants were recruited from the spring 2019 undergraduate student

population at a western college, excluding those aged 17 years or younger. Participants

for the treatment group were recruited via an existing course titled: Human Development

and Family Studies (HDFS)/Psychology (PSY) 3700: Mental Health Awareness and

Advocacy. Recruitment for control group participants came from an existing course

titled: HDFS 1500: Development across the Lifespan and followed typical course

enrollment procedures of the university. In week one of both courses, students were

notified via electronic message and in-class announcement (for face-to-face classes) that

a research opportunity was available and optional and part of a dissertation research

study. Course instructors were not present at the time students were invited to participate

in the research; all invitations were conducted by an independent research assistant.

Students had the opportunity to opt into or out of the research study by indicating their

intention to participate on the informed consent.

A total of 275 participants completed the pretest survey and 270 participants

completed the posttest survey. Of these participants, only 162 completed both pretest and

posttest surveys. Two participants only completed demographic questions and didn’t

complete outcome measures and thus were excluded from the study; this resulted in a

total study sample of 160 participants (see Figure 3.2 for participant flow diagram). There
 73

Figure 3.2. Participant flow diagram.

 74

were no significant differences in completion rates between conditions. Participants were

included in the study if they were over the age of 17, enrolled in one of the treatment or

control classes included, and had pretest and posttest scores on key outcome measures.

Ages of participants ranged from 18 to 60 (M = 23.87, S.D. =7.74).

Full demographic characteristics of the sample are provided in Table 3.1;

bivariate correlations between key outcome variables at pretest and posttest are provided

in Tables 3.2 and 3.3; and sample size, means, and standard deviations of each key

variable are provided in Table 3.4 (shown later in this chapter). Because all demographic

variables were categorical, chi-square tests were conducted to identify any pre-existing

group differences for the treatment and control groups. Results indicate that the

treatment group was significantly more likely to be at a higher year in school (Χ2(4) =

47.95, p < .001). Additional descriptive variables were included to determine prior

exposure to mental health issues. Of the 160 participants, 151 (94.4%) had never

participated in QPR training, 120 (75%) had never been diagnosed with a mental health

issue, 117 (78%) described themselves as being emotionally close with someone with a

mental health issue, 114 (88.1%) said they had never experienced suicidal thoughts, 88

(55%) explained they knew someone that had experienced suicidal thoughts, 88 (51.9%)

explained they had experienced a mental health issue, 148 (92.5%) explained they knew

someone with a mental health issue, and 138 (86.3%) explained they had never received

therapy. Based on independent samples t tests, none of these items differed statistically

between treatment and control groups.

 75

Table 3.1

Key Sample Characteristics

Control Treatment
Variable Sample N Lifespan course MHAA course
Year in School* 161
Freshman 41 8
Sophomore 25 13
Junior 20 25
Senior 2 27

Gender Identity (see note) 162

Female 81 64
Male 8 9

Race/Ethnicity 162
White/European American 83 70
American Indian 1 0
Asian 0 1
Hispanic or Latino 3 1
Bi-Racial 2 1

Mother’s Level of Education 160

Some High School 2 3
High School Graduate 20 10
Some College 19 25
Associate Degree 7 6
Bachelor’s Degree 30 23
Master’s Degree 9 5
Doctorate Level Degree 1 0

Father’s Level of Education 157

Some High School 1 2
High School Graduate 14 9
Some College 11 12
Associate Degree 6 4
Bachelor’s Degree 31 24
Master’s Degree 19 15
Doctorate Level Degree 5 4

Financial Stress Growing Up 162

Not at all concerned 34 32
Somewhat concerned 40 31
Very Concerned 15 10
Note. Gender was assessed in a nonbinary format but responses were all either male or female.

*Indicates that treatment and control groups differed in a chi-square test at a level of p < .05.
Table 3.2

Correlations Between Key Outcome Variables at Pretest

Declarative OPR Self-efficacy

Measure knowledge Self-efficacy Behavior knowledge (Wyman et al.) PHQ9 GAD-7
MHAA-AT:
Declarative knowledge 1
Self-efficacy .58** 1
Behavior
.16* .36** 1
QPR knowledge -.06 -.06 -.02 1
Self-efficacy (Wyman) .32** .59** .28** -.02 1
PHQ-9 .08 .12 .22** -.10 .06 1
GAD-7 .09 .10 .19** .05 -.05 .76** 1
Note. The acquiring declarative knowledge, building self-efficacy, and applying skills (behaviors) items are microprocess subscales from the
MHAA-AT domains of identifying, locating, and responding. The QPR knowledge scale is used with permission from the QPR Institute.
Self-Efficacy is a subscale from Wyman et al., 2008 on gatekeeping behaviors used with permission from authors. PHQ-9 assesses
depressive symptoms. GAD-7 assesses anxiety symptoms.
* p < .05 (2-tailed).
** p< .01 (2-tailed).

76
Table 3.3

Correlations Between Key Outcome Variables at Posttest

Declarative OPR Self-efficacy

Measure knowledge Self-efficacy Behavior knowledge (Wyman et al.) PHQ9 GAD-7
MHAA-AT:
Declarative knowledge 1
Self-efficacy .60** 1
Behavior
.05 .08 1
QPR knowledge .12 .07 -.11 1
Self-efficacy (Wyman) .47* .61** .17** -.02 1
PHQ-9 -.07 .01 .18** -.15* -.02 1
GAD-7 -.11 -.05 .22** -.18* -.09 .80** 1
Note. The acquiring declarative knowledge, building self-efficacy, and applying skills (behaviors) items are microprocess subscales from the
MHAA-AT domains of identifying, locating, and responding. The QPR knowledge scale is used with permission from the QPR Institute.
Self-Efficacy is a subscale from Wyman et al., 2008 on gatekeeping behaviors used with permission from authors. PHQ-9 assesses
depressive symptoms. GAD-7 assesses anxiety symptoms.
* p < .05 (2-tailed).
** p< .01 (2-tailed).

77
 78

Procedure

Students that opted into participation in the study completed the Mental Health

Awareness and Advocacy Assessment Tool (MHAA-AT; see appendix one for survey).

This survey consisted of assessments designed to evaluate students’ declarative

knowledge, self-efficacy, and behavioral outcomes in identifying mental health issues,

locating evidence-based resources, and responding to mental health issues. The pretest

survey also included key demographic variables, as well as variables related to their

exposure to mental health issues (the latter for descriptive purposes).

After completing the informed consent and pretest survey, students completed the

assigned requirements of the 16-week curriculum for their respective course. Upon

completion of the course, students were asked to complete the posttest MHAA-AT and

other key outcome measurements. The pretest and posttest surveys took approximately

thirty minutes to complete and were delivered via the Qualtrics system using an

anonymous link posted to the course management (Canvas) home page. Students

received extra credit (1% of total grade) for completing both the pretest and posttest

assessments. At the conclusion of both surveys, students were provided with mental

health resources including: The National Suicide Prevention Lifeline, Crisis Text Line,

and area specific mental health resources via PsychologyToday.com. The curricula for

the treatment group and control group are explained in the following sections.

Treatment group. The treatment group completed a 16-week in-person or online,

undergraduate course in the spring of 2019 taught by the same instructor (the first

author). The undergraduate course used the Mental Health Awareness and Advocacy
 79

curriculum that addresses three goals: (1) increase undergraduate students’ declarative

knowledge; (2) increase self-efficacy; and (3) increase frequency of appropriate

behaviors in identifying mental health issues, locating evidence-based resources, and

responding to mental health issues. The course uses the following syllabus description,

This course is designed to provide introductory knowledge of mental health

issues, their effects on systems (e.g. family, educational, judicial), and specific
advocacy efforts to more effectively support individuals with mental health needs.
You will learn about the sociocultural history of mental health as well as current
epidemiology and impacts of these issues. This course will increase critical
thinking skills through analysis of current research and help you develop skills
that will prepare you to be effective advocates and responders to mental health
issues.

The curriculum contained three sections to help accomplish the identified goals: Section

One - Identifying mental health issues; Section two - Locating evidence-based resources;

and Section three - Responding to mental health issues. Each section consists of five

lectures, two quizzes, one assignment, and one exam (with the third section exam being a

comprehensive exam). Each section was five weeks of the total course time with one

week being held for final examinations.

Identifying mental health issues. The identifying mental health issues section

consisted of five different sub-topics: (1) building social support; (2) theory related to

mental health issues; (3) mood disorders across the lifespan; (4) anxiety disorders across

the lifespan; (5) bipolar and psychotic disorders across the lifespan. During each of the

sub-topics, students were asked basic mastery questions during lectures and provided

immediate feedback (in-person course) or via delayed response in an online lecture.

Locating empirical resources. The locating empirical resources section consisted

of five different subtopics: (1) advocacy theory and epistemology; (2) empirically
 80

supported community programs for mental health advocacy; (3) empirically based

treatment and self-help options; (4) identifying and accessing quality mental health

resources; (5) advocating for mental health issues in your community and state. During

each of the subtopics, students were asked basic mastery questions during lectures and

provided feedback in the same manner described above.

Responding to mental health crises. The responding to mental health crises

module consisted of five different sub-topics: (1) epidemiology of suicide; (2) identifying

at risk individuals; (3) persuading at risk individuals to seek help; (4) referring

individuals to quality mental health resources; and (5) review of each individual section.

During each of the sub-topics, students were asked basic mastery questions during

lectures and provided feedback on their skill development.

Pedagogical approach. The course was taught using the following methods: (1)

course readings, (2) multi-media engagement, (3) in-class and/or online discussions, (4)

supervised feedback on each assignment. The primary teaching goal was twofold: First,

exposure to the content material was accomplished through course readings and multi-

media engagement (e.g., videos, news articles, social media). Second, students were

encouraged to have open conversations about this material to help deepen their

understanding of the content. Upon communicating their ideas and understanding of the

content, detailed feedback was provided to students during class discussions and

independently on individual assignments to help address strengths and deficits of each

students’ individual skillset related to mental health literacy.

Control group. The control group completed either a 16-week in-person or

 81

online, undergraduate Human Development and Family Studies (HDFS) course in spring

2019 titled Development across the Lifespan taught by two separate instructors. This

course was selected as a control group to help reduce the potential confounds of self-

selection (e.g., psychology majors) and prior exposure to courses related to mental health

(e.g., higher division courses in HDFS and Psychology courses often specialize in mental

health issues) that is more likely in an upper division course. The already established

Development across the Lifespan course is required for all HDFS majors at a western

college and meets general education requirements of most degrees widening the possible

type of student enrolled in the course. The course syllabus description states the

following,

This course will introduce students to the concepts and science of human
development and the changes in development that occur across the life span from
conception through death. We will focus on the physical, cognitive, and
socioemotional changes that occur as individuals grow and develop. In addition,
this class will introduce students to the major theoretical perspectives associated
with human development, incorporate topics into “real world” examples, and
present a contextual perspective of human development.

The Development across the Lifespan course shares none of the same goals as the

Mental Health Awareness and Advocacy course but provides approximately the

same level of academic rigor expected of a college course that meets major degree

requirements.

Measurement

MHAA-AT. The mental health Mental Health Awareness and Advocacy

Assessment Tool (MHAA-AT) was used to evaluate students’ growth related to mental

health literacy. The measure evaluates students’ microprocess skills of acquiring

 82

declarative knowledge, building self-efficacy, and applying skills/behaviors within three

macro-process domains: (1) identifying mental health issues, (2) locating empirical

resources, and (3) responding to mental health issues. The MHAA-AT demonstrates high

content validity in the declarative knowledge items (see Table 3.2 for bivariate

correlations between key outcome variables at pretest and Table 3.3 for correlations

between key outcome variables at posttest). The MHAA-AT declarative knowledge

questions demonstrated moderate internal consistency with Cronbach’s alphas of .48, .70,

.55 for the Identifying, Locating, and Responding domains respectively. While the

internal consistency figures were only moderate, this could be attributed to the

dichotomous responses to the questions (see study one). The self-efficacy and behavior

items had strong internal consistency with Cronbach’s alphas of .97 and .90 respectively.

Last, each of the declarative knowledge subscales ranged from 0-10 on score, and a sum

score was used. For the self-efficacy and behavior items, a mean conversion of the scale

was used in interpretation.

QPR knowledge scale. The QPR knowledge scale (Quinnett, 1997, 2005) is a

measure used to assess knowledge related to suicide prevention. This quiz-like measure is

commonly used to assess the knowledge gained by participating in QPR training

(Quinett, 2009; Wyman et al., 2008). There are no psychometric properties reported on

this measure, but in the paper outlining the theoretical underpinnings of QPR training, the

items are stated to support key knowledge required to be effective at responding as a

gatekeeper (Quinett, 2005). Two items that required selecting multiple responses were

excluded due to errors in data collection. For the QPR knowledge scale, a sum scale was
 83

used in interpreting the data.

Self-efficacy (Wyman et al., 2008). The Wyman et al. (2008) Self-Efficacy

subscale was developed by Wyman et al. to evaluate the effectiveness of QPR training in

the residential housing center at varying colleges. This 7-item measure uses a 7-point

Likert scale containing confidence statements to evaluate perceived self-efficacy of

gatekeeping behaviors. Sample items include: “If a student experiencing thoughts of

suicide does not acknowledge the situation, there is very little that I can do to help”; “If

a student contemplating suicide refuses to seek help, it should not be forced upon

him/her.” Cronbach’s alpha of the seven items was reported as .796 (Wyman et al., 2008)

and .813 in the current sample.

Patient Health Questionnaire-9. The Patient Health Questionnaire-9 (PHQ-9;

Löwe, Unützer, Callahan, Perkins, & Kroenke, 2004) is a 9-item Likert questionnaire that

was used to assess depressive symptoms. The measure asks participants to respond on a

four-point Likert scale (Not at all = 0, Nearly every day = 3) to being bothered by a

variety of symptoms in the past two weeks. Symptoms included in the measure mirror

diagnostic criteria for major depressive disorder and include the following: “Little

interest or pleasure in doing things”; Feeling bad about yourself — or that you are a

failure or have let yourself or your family down.” Cronbach’s alpha of the scale was

reported to be .89 and test-retest reliability was reported at 0.84 (Kroenke, Spitzer, &

Williams, 2001) .88 in the current sample. The measure also has strong evidence for

construct validity and criterion validity (Kroenke et al., 2001). For the PHQ-9 a sum

scale was used in interpreting the data, with higher scores indicating more depressive
 84

symptoms.

Generalized Anxiety Disorder-7. The Generalized Anxiety Scale-7 (GAD-7;

Spitzer, Kroenke, Williams, & Lowe, 2006) is a 7-item Likert questionnaire that was

used to assess generalized anxiety. The measure asks participants to respond on a four-

point Likert scale (Not at all = 0, Nearly every day = 3) to being bothered by a variety of

symptoms in the past two weeks. Symptoms included in the measure mirror diagnostic

criteria for major depressive disorder and include the following: “Feeling nervous,

anxious, or on edge”; “Worrying too much about different things.” Cronbach’s alpha on

the scale was reported at .92 and was .92 in the current sample. The scale was reported

as having good procedural validity and diagnostic criterion validity (Spitzer, Kroenke,

Williams, & Lowe, 2006). For the GAD-7 a sum scale was used in interpreting the data,

with higher scales indicating more anxiety symptoms.

Analytic Approach

To address each of the research questions, a two-way mixed ANOVA analysis

was conducted. This analytic approach allows for analysis of two or more groups within

the independent variable while also having repeated measures on the outcome variable.

This approach simultaneously analyzes main (i.e., time) and interaction (i.e., time X

treatment, treatment X modality, and time X treatment x modality) effects on key

outcome variables. Prior to conducting main analyses, assumptions testing (normality of

data and equality of variances) was completed to determine the appropriateness of the

analytic technique. Results from tests of normality of variables (Shapiro-Wilk’s test)

identified several non-normally distributed variables (at time one and at time two), based
 85

on p-values that were less than .05. However, upon deeper review of the Shapiro-Wilk’s

statistics all values were greater than .90 or close to .90 (.78-.88) suggesting the two-way

mixed ANOVA is robust enough to handle the non-normality of the data on these

variables (Kim, 2012). Lastly, skewness and kurtosis of each variable was assessed. The

PHQ-9 and GAD-7 demonstrated both high skew and kurtosis as per the statistics (+/-

2.0). Upon further review it was determined to not complete any data transformations

because the values were within normal levels expected for individuals with depression or

anxiety in a college population.

Results

Research Question #1

Research question #1 asked: “Do Students That Participate in the MHAA

Curriculum Improve on Key Outcomes”?

Descriptive data for each condition and time point on key outcome variables are

provided in Table 3.4 and 3.6. Two-way mixed ANOVA examined Time X Condition

effects to address research question one. Results indicated significant Time X Condition

interactions on outcome measures where the treatment group improved significantly more

than the control group (see table 3.5 and figure 3.3). The significant interactions were on

the following outcome variables: MHAA-AT: Declarative Knowledge Identifying F(1, 151)

= 18.62, p = .00 , partial 𝜂𝜂2 = .11; MHAA-AT: Declarative Knowledge Locating F(1, 151) =

4.70, p = .03, partial 𝜂𝜂2 = .03; MHAA-AT: Self-Efficacy subscale F(1, 146) = 86.01, p =

.00, partial 𝜂𝜂2 = .37; QPR Knowledge Scale F(1, 153) = 3.92, p = .05, partial 𝜂𝜂2 = .03;
Table 3.4

Means and Standard Deviations with Available Data between Conditions on Outcome Measures

Control Treatment
──────────────────────── ────────────────────────
Measure Pre M SD Post M SD Pre M SD Post M SD
MHAA-AT
Declarative Knowledge Identifying In-person 4.60 2.25 5.06 2.38 5.30 2.35 7.63 1.41
Online 4.71 2.08 5.31 1.80 5.43 2.12 7.42 1.65

Declarative Knowledge Locating In-person 4.86 2.47 4.52 2.57 6.18 2.52 7.91 1.63
Online 5.26 2.09 5.31 2.11 6.09 2.24 7.81 1.93

Declarative Knowledge Responding In-person 3.02 1.42 4.06 1.83 3.70 1.98 5.33 1.31
Online 3.77 1.48 4.31 1.51 4.14 1.48 4.89 1.78

Self-Efficacy In-person 2.72 .82 3.15 .98 2.83 1.07 4.63 .76
Online 2.90 .92 3.29 1.04 3.01 .92 4.68 .72

Behavior In-person 1.04 .74 1.03 .78 1.19 .98 1.35 .87
Online 1.32 .92 1.20 .98 1.15 .77 .99 .65

QPR Knowledge In-person 11.21 1.23 10.06 1.35 11.03 1.00 10.47 .99
Online 11.11 .96 10.23 1.28 11.19 1.39 10.66 .79

Self-Efficacy (Wyman, et al.) In-person 3.76 1.00 4.24 .88 3.64 .99 4.97 .66
Online 3.89 .86 3.97 1.04 3.98 .92 5.07 .69
Note. The acquiring declarative knowledge, building self-efficacy, and applying skills (behaviors) items are microprocess subscales from the
MHAA-AT domains of identifying, locating, and responding. Knowledge items are broken down here by domain; Self-efficacy and Behavior are
total items across all domains. The QPR knowledge scale is used with permission from the QPR Institute. Self-Efficacy is a subscale from
Wyman et al., 2008 on gatekeeping behaviors used with permission from authors. N ranged from 150-157 across all scales.

86
 87

Table 3.5

Time X Condition Results of a Two-Way Repeated Measures ANOVA Analysis on Key

Outcome Variables

Source df (error) F p value Partial 𝜂𝜂 2

MHAA-AT
Declarative Knowledge Identifying 1 (151) 18.62 .00** .11

Declarative Knowledge Locating 1 (151) 4.70 .03* .03

Declarative Knowledge Responding 1 (150) 2.01 .16 .01
Self-Efficacy 1 (146) 86.01 .00** .37

Behavior 1 (146) .32 .58 .00

Other key outcome variables

QPR Knowledge 1 (153) 3.92 .05* .03

Self-Efficacy (Wyman et al.)
1 (152) 39.22 .00** .21
Note. Knowledge items are broken down here by each of the three domains; Self-efficacy and Behavior are
total items across all domains.
* p < .05 (2-tailed).
** p< .01 (2-tailed).

Table 3.6

Means and Standard Deviations with Available Data between Conditions on Mental
Health Outcomes

Control Treatment
──────────────────── ────────────────────
Measure Pre M SD Post M SD Pre M SD Post M SD
PHQ-9 In-person 5.19 5.06 5.09 4.69 6.76 6.00 7.11 7.73
Online 6.81 5.44 7.17 5.85 5.72 4.39 5.31 4.14

GAD-7 In-person 5.55 5.39 5.02 4.91 5.00 5.64 5.97 6.29
Online 6.56 5.82 7.64 6.78 5.25 4.39 4.83 3.71
 (A)MHAA-AT: D.K. Identifying (B) MHAA-AT: D.K. Locating (C) MHAA-AT: Self-Efficacy
10 10 6

5
8 8
7.55 7.86 4 4.66
6 6
6.15 3
5.42 5.06 4.91 3.22
4 5.19 4 2.82 2.92
4.62 2
2 2 1

0 0 0
Control Treatment Control Treatment Control Treatment

Pre-Test Post-Test Pre-Test Post-Test Pre-Test Post-Test

)
(D) QPR Knowledge Scale (E) Wyman et al. 2008, Self-Efficacy Scale
12 7
11
10 11.16 11.11 6
10.14 10.57
9
5
8 5.02
7 4
6 4.11
3.83 3.81
5 3
4
3 2
2 1
1
0 0
Control Treatment Control Treatment
Pre-Test Post-Test Pre-Test Post-Test

Figure 3.3. Mean plots of significant interactions for Time X Condition.

88
 89

for the MHAA-AT: Declarative Knowledge Identifying, MHAA-AT: Declarative

Knowledge Locating MHAA-AT: Self-Efficacy Subscale, and the QPR Knowledge Scale.

There was not, however, Time X Modality X Condition effects. Similar to the previous

analysis, the demographic variable of year in school was used as a covariate and the

analyses were conducted again. Including the demographic variable did not significantly

change the results of the analyses and for parsimony it was excluded from the results. For

full results of the two-way mixed ANOVA analysis for Time X Modality on each

outcome variable see Table 3.7 and for mean plots of significant interactions see Figure

3.4.

Table 3.7

Time X Modality Results of a Two-Way Repeated Measures ANOVA Analysis on Key

Outcome Variables

Source df (error) F p-value Partial 𝜂𝜂 2

MHAA-AT
Declarative Knowledge Identifying 1 (151) .40 .66 .00
Declarative Knowledge Locating 1 (151) .26 .61 .00
Declarative Knowledge Responding 1 (150) 6.11 .02* .04
Self-Efficacy 1 (146) .33 .57 .00
Behavior 1 (146) 3.29 .07 .02

Other key outcome variables

QPR Knowledge 1 (153) .39 .54 .00
Self-Efficacy 1 (152) 4.61 .03* .03
Note. Note. The acquiring declarative knowledge, building self-efficacy, and applying skills (behaviors)
items are microprocess subscales from the MHAA-AT domains of identifying, locating, and
responding. Knowledge items are broken down here by domain; Self-efficacy and Behavior are total
items across all domains. The QPR knowledge scale is used with permission from the QPR Institute.
Self-Efficacy is a subscale from Wyman et al., 2008 on gatekeeping behaviors used with permission
from authors. N ranged from 150-157 across all scales

* p < .05 (2-tailed).

** p< .01 (2-tailed).
 90

(A) MHAA-AT: D.K. Responding (B) MHAA-AT: Behaviors

10 6

5
8
4
6
3
4 4.52 4.51
4.04 2
3.54
2 1
1.26 1.27 1.07
1.03
0 0
Control Treatment Control Treatment

In-person Online In-person Online

(C) Wyman et al., 2008 Self-Efficacy

7
6
5
4
4.3 4.53
4 3.93
3
2
1
0
Control Treatment

In-person Online

Figure 3.4. Mean plots of significant interactions for Time X Modality.

Research Question #3

Research question #3 asked, “Do Students that Participate in the MHAA

Curriculum Improve Their Mental Health”?

Descriptive data for key mental health variables are provided in Table 3.8. Two-

way mixed ANOVA examined Time X Condition effects to address the third research

question. There were no significant Time X Condition or Time X Modality interactions

 91

for the PHQ-9 and GAD-7 mental health outcomes. For full results see Table 3.9.

Table 3.8

Means and Standard Deviations with Available Data Between Conditions on Mental
Health Outcomes

Control Treatment
──────────────────── ────────────────────
Measure Pre M SD Post M SD Pre M SD Post M SD
PHQ-9 In-person 5.19 5.06 5.09 4.69 6.76 6.00 7.11 7.73
Online 6.81 5.44 7.17 5.85 5.72 4.39 5.31 4.14

GAD-7 In-person 5.55 5.39 5.02 4.91 5.00 5.64 5.97 6.29
Online 6.56 5.82 7.64 6.78 5.25 4.39 4.83 3.71
N = 159.

Table 3.9

Time X Condition and Time X Modality Results of a Two-Way Repeated Measures

ANOVA Analysis on Mental Health

Source df (error) F p value Partial 𝜂𝜂 2

Time X Condition
PHQ-9 1 (155) .07 .79 .00
GAD-7 1 (155) .00 .99 .00

Time X Modality
PHQ-9 1 (155) .06 .80 .00
GAD-7 1 (155) .04 .84 .00
* p < .05 (2-tailed).
** p< .01 (2-tailed)

Discussion

The primary purpose of this study was to evaluate the effectiveness of the Mental

Health Awareness and Advocacy (MHAA) curriculum in improving students’ knowledge,

 92

self-efficacy, and behaviors related to mental health literary. Results indicated that the

MHAA curriculum was effective in improving areas of students’ knowledge and self-

efficacy measured by both the MHAA-AT and other outcome measures (QPR

Knowledge scale and Wyman and colleagues (2008) Self-Efficacy subscale) used to

evaluate commonly implemented mental health literacy programs (e.g., QPR, MHFA,

etc.). More specifically, the results of the study indicated that the MHAA students

improved their knowledge related to identifying mental health issues and locating

evidence-based resources, and their self-efficacy as was measured by the MHAA-AT.

The MHAA curriculum participants did not improve on applying skills (MHAA-AT

behaviors) or key mental health outcomes of anxiety and depressive symptoms

(impacting anxiety and depressive symptoms were not part of the hypothesized outcomes

of the course, but results are reported none-the-less).

Results of the study suggest that the curriculum is effective in improving a

student’s ability to identify key facts that are needed to identity depression and anxiety in

a variety of populations and then recognizing accurate information about accessing

evidence-based resources. These findings suggest that students that participate in the

MHAA are finishing the course with an in-depth understanding of the specific criteria

and demographic information needed to understand and recognize depression and anxiety

in real-time. Additionally, students completing the curriculum were demonstrating an

increase in knowledge on identifying high-quality resources. This could potentially lead

to more effective and useful referrals by these students in the future. Somewhat

surprisingly, students did not improve their declarative knowledge related to responding
 93

to mental health issues. This could be related to a discrepancy between what is offered in

the curriculum (e.g., specific skills about responding to a suicidal student) versus the

specific content asked on the MHAA-AT related to responding to mental health issues

(e.g., age group of individuals most likely to die by suicide). Future editions of the

MHAA curriculum could incorporate more demographic information related to

suicidology rather than primarily focusing on skills needed to respond to crisis situations.

The MHAA curriculum demonstrated the large effect on student’s self-efficacy as

measured by the MHAA-AT and the Wyman and colleagues (2008) Self-Efficacy

subscale. This result is consistent with other studies evaluating mental health literacy

programs. A deeper exploration of the data detailed that students improved their self-

efficacy in each domain: identifying mental health issues, locating evidence-based

resources, and responding to mental health issues. These findings suggest that students

that participate in the curriculum are completing the course feeling confident in each of

the key areas of the course. Adding more nuances to this finding, students completing the

MHAA curriculum improved more as measured by the MHAA-AT comparatively to the

Wyman and colleagues (2008) Self-Efficacy subscale. This finding could suggest that the

MHAA-AT is a more useful measure of students’ self-efficacy in this context and is more

sensitive to change. This is a particularly important finding as the Wyman and colleagues

(2008) Self-Efficacy subscale is currently one of the most common measures to evaluate

self-efficacy related to mental health literacy in the literature. Overall, the MHAA

curriculum demonstrates sound evidence that it improves students’ self-efficacy which is

very important considering that an individual’s self-efficacy is often predictive of their

 94

future behaviors.

Being in the treatment group did not influence the average scores of participants

on the MHAAT-AT behaviors subscale. This was somewhat a surprising finding as the

MHAA-AT self-efficacy items and MHAA-AT behaviors subscales were positively

correlated at pretest (r = .16, p < .05) and the process-oriented nature of the measure

would hypothesize an increase in behaviors. The lack of this finding could be due to a

possible lack of sensitivity of the behavior items or the questions being asked too close

following the course or the possibility that students didn’t have any opportunity to

respond. To the first point, the MHAA-AT is still a new measure and future studies can

determine if measurement issues contribute to the behavior subscale. Likewise,

evaluation of future MHAA classes can determine if this null finding is a result of an

ineffective intervention in which case the MHAA curriculum might also need to be

refined in future iterations to more explicitly encourage purposeful action in

communities.

On the topic of modality (i.e., in-person versus online delivery), there were

significant mean differences on MHAA-AT Declarative Knowledge Responding, Wyman

and Colleagues (2008) Self-Efficacy scale, and there was a trend for the MHAA-AT:

Behaviors subscale. These findings suggest that in-person delivery seemed to positively

influence growth in the in-person delivery courses more than online delivery courses.

This finding should be interpreted with caution because of the lack of three-way

interaction term (Time X Condition X Modality) being insignificant. This suggests that it

is not the MHAA curriculum influencing these changes, but instead a component of the
 95

in-person format. One possible explanation could be attributed to the value of being able

to form strong interpersonal relationships via direct contact with students in the in-person

modalities. However, more evaluation is needed to determine if these findings remain

consistent across groups or if this finding was unique to sample being considered prior to

drawing more definitive conclusions.

The MHAA curriculum was not effective in improving or worsening students’

mental health outcomes. This finding isn’t necessarily surprising to either side of the

effect. There are several common arguments perpetuated in the media that suggest talking

to students more about mental health issues can expose them to negative effects and ergo

worsen their mental health outcomes (Rosenquist, Fowler, Christakis, 2011; Fowler &

Christakis, 2008; Boyles, 2008). Conversely, there are also arguments that suggest the

more mental health issues are discussed it can provide relief for those experiencing these

issues (Quinnett, 2009) or that talking and being around mental health has minimal

contagion effect (Eisenberg, Golberstein, Whitlock, & Downs, 2013). In this study,

neither arguments are supported because there was no change over time detected. That

being said, if future goals of the course add improved mental health of the students

themselves, the MHAA curriculum might benefit from adding components of direct

online psychological interventions, like web-based Acceptance Commitment Therapy

programs (Levin, Haeger, Pierce, & Twohig, 2017), to help improve key mental health

outcomes and help encourage more purposeful self-actions.

An initial strength of the MHAA curriculum was the unique process-based

approach to the delivery and evaluation. Another strength was offering it as a for-credit
 96

course in a degree seeking program. In sum, both of these unique strengths gained

support for their effectiveness in either influencing mean scores or verifying a proof of

concept. More specifically, the MHAA-AT evaluation of the MHAA curriculum provides

a unique toolset for educators to more accurately identify knowledge and self-efficacy

deficits in students’ abilities at pretest. This ability could help future deliveries of the

curriculum by adjusting content throughout the 16-week course to better meet the average

needs of the students participating. Additionally, the MHAA curriculum operated well as

a course and has been continually offered for two years, suggesting the feasibility to

maintain a course on a college campus. These two points provide exciting opportunities

for future growth of the MHAA curriculum.

Limitations

There are several limitations of the current study addressed here. As was indicated

in the preliminary analyses section, the data failed several assumptions tests related to

normality in distribution prior to running the two-way mixed ANOVA. Despite this, there

is consistent evidence that suggest this analytic technique is robust enough to handle

these data issues. As additional samples are tested with the MHAA curriculum, it is

possible that issues of normality will improve. There is also a need to consider the utility

of the MHAA-AT and its use in evaluating an intervention. While this assessment tool

has provided strong psychometric properties (see Study 1 in this dissertation), there is not

yet evidence of pretest/posttest analyses beyond this initial curriculum evaluation. The

fact that other established measures, such as the XXXX improved in the treatment group
 97

suggest that measurement via a new tool (MHAA-AT) is not artificially driving these

effects. The results of the study should also be interpreted within the context of the

sample. For instance, the course was offered as an elective credit in the degree suggesting

that students that took the course might be unique, or there may be a selection effect due

to those that are participating in the treatment are actively choosing to take the course

(despite limited significant differences between students as per statistical tests explained

previously). Lastly, the course was highly homogenous in both sex and ethnicity.

Accordingly, future research is needed to determine the utility and consistency of the

assessment tool. Likewise, the course should be taught in other universities to identify

effectiveness of the course across more diverse regions and cultures.

Conclusion

The MHAA curriculum demonstrated strong initial evidence in this preliminary

study as being effective at improving students’ mental health literacy. While future

research is needed to replicate these findings, the MHAA curriculum provides a unique

and important intervention point for college campuses. Future efforts evaluating the

MHAA curriculum should seek to expand the reach of the curriculum by assessing it in

varying college settings including community colleges, smaller liberal-arts colleges, and

for-profit institutions. By finding ways to expand the scope and utility of the MHAA

curriculum, the tools by which a college campus can address the growing concern of

students’ mental health issues is addressed.

 98

References

American College Health Association (2015). American College Health Association-

National College Health Assessment II: Reference Group Executive Summary
Spring 2015. Hanover, MD: American College Health Association.

Auerbach, R. P., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C., Cuijpers, P., … WHO
WMH-ICS Collaborators (2018, September 13). WHO World Mental Health
Surveys International College Student Project: Prevalence and Distribution of
Mental Disorders. Journal of Abnormal Psychology. doi:
dx.doi.org/10.1037/abn0000362

Bandura, A. (1978). The self-system in reciprocal determinism. American

Psychologist, 33(4), 344-358.

Bandura, A. (1982). Self-efficacy mechanism in human agency. American

Psychologist, 37(2), 122-147.

Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: Macmillan.

Bandura, A. (2005). The primacy of self‐regulation in health promotion. Applied

Psychology, 54(2), 245-254.

Battaglia, J., Coverdale, J. H., & Bushong, C. P. (1990). Evaluation of a mental illness
awareness week program in public schools. The American Journal of
Psychiatry, 147(3), 324-329.

Becker, M. H. (1974) The Health Belief Model and personal health behavior. Health
Education Monographs, 2, 324-473.

Boyles S. (2008). Happiness is contagious: Social networks affect mood, study shows.
Retrieved from www.webmd.com/balance/news/20081204/happiness-is-
contagious

Burns, J. R., & Rapee, R. M. (2006). Adolescent mental health literacy: young people's
knowledge of depression and help seeking. Journal of Adolescence, 29(2), 225-
239.

Center for Collegiate Mental Health. (2018, January). 2017 Annual Report (Publication
No. STA 18-166). University Park, PA: Publisher.

Champion, V. L., & Skinner, C. S. (2008). The health belief model. Health behavior and
health education: Theory, Research, and Practice, 4, 45-65.
 99

Conley, C. S., Durlak, J. A., & Kirsch, A. C. (2015). A meta-analysis of universal mental
health prevention programs for higher education students. Prevention Science,
16(4), 487-507.

Czyz, E. K., Horwitz, A. G., Eisenberg, D., Kramer, A., & King, C. A. (2013). Self-
reported barriers to professional help seeking among college students at elevated
risk for suicide. Journal of American College Health, 61(7), 398-406.

Dahlberg, K. M., Waern, M., & Runeson, B. (2008). Mental health literacy and attitudes
in a Swedish community sample-Investigating the role of personal experience of
mental health care. BMC Public Health, 8(1), 8-18. doi: 10.1186/1471-2458-8-8

Davies, E. B., Morriss, R., & Glazebrook, C. (2014). Computer-delivered and web-based
interventions to improve depression, anxiety, and psychological well-being of
university students: A systematic review and meta-analysis. Journal of Medical
Internet Research, 16(5), e130.

Dumesnil, H., & Verger, P. (2009). Public awareness campaigns about depression and
suicide: a review. Psychiatric Services, 60(9), 1203-1213.

Eisenberg, D., Golberstein, E., & Gollust, S. E. (2007). Help-seeking and access to
mental health care in a university student population. Medical Care, 45(7), 594-
601.

Eisenberg, D., Golberstein, E., Whitlock, J. L., & Downs, M. F. (2013). Social contagion
of mental health: evidence from college roommates. Health Economics, 22(8),
965-986.

Eisenberg, D., Hunt, J., & Speer, N. (2013). Mental health in American colleges and
universities: variation across student subgroups and across campuses. The Journal
of Nervous and Mental Disease, 201(1), 60-67.

Eisenberg, D., Hunt, J., & Speer, N. (2012). Help seeking for mental health on college
campuses: Review of evidence and next steps for research and practice. Harvard
Review of Psychiatry, 20(4), 222-232.

Evans Research. (1999). Report on the review of mental health information brochures
produced under the Community Awareness Program (CAP). Canberra, AUST:
Commonwealth Department of Health and Aged Care, Mental Health Branch.

Furnham, A., Cook, R., Martin, N., & Batey, M. (2011). Mental health literacy among
university students. Journal of Public Mental Health, 10(4), 198-210.

Fowler, J.H., Christakis, N.A. (2008). Dynamic spread of happiness in a large social
network: Longitudinal analysis over 20 years in the Framingham Heart Study.
BMJ, 337, a2338.
 100

Francis, C., Pirkis, J., Dunt, D., Blood, R. W., & Davis, C. (2002). Improving mental
health literacy: A review of the literature. Melbourne, Australia: Centre for
Health Program Evaluation, University of Melbourne.

Goldney, R. D., Fisher, L. J., & Wilson, D. H. (2001). Mental health literacy: an
impediment to the optimum treatment of major depression in the
community. Journal of Affective Disorders, 64(2-3), 277-284.

Gulliver, A., Griffiths, K. M., & Christensen, H. (2010). Perceived barriers and
facilitators to mental health help-seeking in young people: a systematic
review. BMC Psychiatry, 10(1), 113-122. doi: doi.org/10.1186/1471-244X-10-
113

Hanisch, S. E., Twomey, C. D., Szeto, A. C., Birner, U. W., Nowak, D., & Sabariego, C.
(2016). The effectiveness of interventions targeting the stigma of mental illness at
the workplace: A systematic review. BMC Psychiatry, 16(1), 1-11.

Hadlaczky, G., Hökby, S., Mkrtchian, A., Carli, V., & Wasserman, D. (2014). Mental
Health First Aid is an effective public health intervention for improving
knowledge, attitudes, and behaviour: A meta-analysis. International Review of
Psychiatry, 26(4), 467-475.

Hickie, I. (2004). Can we reduce the burden of depression? The Australian experience
with beyondblue: The national depression initiative. Australasian Psychiatry,
12(sup1), s38-s46.

Holmes, E. P., Corrigan, P. W., Williams, P., Canar, J., & Kubiak, M. A. (1999).
Changing attitudes about schizophrenia. Schizophrenia Bulletin, 25(3), 447-456.

Hunt, J., & Eisenberg, D. (2010). Mental health problems and help-seeking behavior
among college students. Journal of Adolescent Health, 46(1), 3-10.

Jorm, A. F. (2012). Mental health literacy: Empowering the community to take action for
better mental health. American Psychologist, 67(3), 231-244. doi: 10.1037/
a0025957

Jorm, A. F., Barney, L. J., Christensen, H., Highet, N. J., Kelly, C. M., & Kitchener, B.
A. (2006). Research on mental health literacy: What we know and what we still
need to know. Australian & New Zealand Journal of Psychiatry, 40(1), 3-5.

Jorm, A. F., Korten, A. E., Jacomb, P. A., Christensen, H., Rodgers, B., & Pollitt, P.
(1997). “Mental health literacy”: a survey of the public's ability to recognise
mental disorders and their beliefs about the effectiveness of treatment. Medical
Journal of Australia, 166(4), 182-186.
 101

Jorm, A. F., Nakane, Y., Christensen, H., Yoshioka, K., Griffiths, K. M., & Wata, Y.
(2005). Public beliefs about treatment and outcome of mental disorders: a
comparison of Australia and Japan. BMC medicine, 3(1), 12-26. doi:
doi.org/10.1186/1741-7015-3-12

Kauer, S. D., Mangan, C., & Sanci, L. (2014). Do online mental health services improve
help-seeking for young people? A systematic review. Journal of Medical Internet
Research, 16(3), e66.

Kelly, C. M., Jorm, A. F., & Wright, A. (2007). Improving mental health literacy as a
strategy to facilitate early intervention for mental disorders. Medical Journal of
Australia, 187(S7), S26-S30.

Kern, A., Hong, V., Song, J., Lipson, S. K., & Eisenberg, D. (2018). Mental health apps
in a college setting: openness, usage, and attitudes. mHealth, 4(20), 1-12.

Kim H. Y. (2012). Statistical notes for clinical researchers: Assessing normal distribution
(1). Restorative Dentistry and Endodontics, 37(4), 245-248. doi:10.5395/rde.
2012.37.4.245

Kitchener, B. A., & Jorm, A. F. (2002). Mental health first aid training for the public:
Evaluation of effects on knowledge, attitudes and helping behavior. BMC
Psychiatry, 2(1), 10-16. doi: doi.org/10.1186/1471-244X-2-10

Kitchener, B. A., & Jorm, A. F. (2006). Mental health first aid training: review of
evaluation studies. Australian and New Zealand Journal of Psychiatry, 40(1), 6-8.

Kitzrow, M. A. (2009). The mental health needs of today's college students: Challenges
and recommendations. NASPA Journal, 46(4), 646-660.

Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ‐9: validity of a brief
depression severity measure. Journal of General Internal Medicine, 16(9), 606-
613.

Lancaster, P. G., Moore, J. T., Putter, S. E., Chen, P. Y., Cigularov, K. P., Baker, A., &
Quinnett, P. (2014). Feasibility of a web‐based gat ekeeper training: Implications
for suicide prevention. Suicide and Life‐Threatening Behavior , 44(5), 510-523.

Lipson, S. K., Speer, N., Brunwasser, S., Hahn, E., & Eisenberg, D. (2014). Gatekeeper
training and access to mental health care at universities and colleges. Journal of
Adolescent Health, 55(5), 612-619.

Löwe, B., Unützer, J., Callahan, C. M., Perkins, A. J., & Kroenke, K. (2004). Monitoring
depression treatment outcomes with the Patient Health Questionnaire-9. Medical
Care, 1194-1201.
 102

Mackenzie, C. S., Erickson, J., Deane, F. P., & Wright, M. (2014). Changes in attitudes
toward seeking mental health services: A 40-year cross-temporal meta-
analysis. Clinical Psychology Review, 34(2), 99-106.

Mitchell, S. L., Kader, M., Darrow, S. A., Haggerty, M. Z., & Keating, N. L. (2013).
Evaluating question, persuade, refer (QPR) suicide prevention training in a
college setting. Journal of College Student Psychotherapy, 27(2), 138-148.

Mehta, N., Clement, S., Marcus, E., Stona, A. C., Bezborodovs, N., Evans-Lacko, S., ...
& Koschorke, M. (2015). Evidence for effective interventions to reduce mental
health-related stigma and discrimination in the medium and long term: systematic
review. The British Journal of Psychiatry, 207(5), 377-384.

National Alliance on Mental Illness. (2017, January). Mental Health by the Numbers.
Retrieved from https://nami.org/

Pescosolido, B. A., Martin, J. K., Lang, A., & Olafsdottir, S. (2008). Rethinking
theoretical approaches to stigma: A framework integrating normative influences
on stigma (FINIS). Social Science & Medicine, 67(3), 431-440.

Pickett-Schenk, S. A., Cook, J. A., & Laris, A. (2000). Journey of Hope program
outcomes. Community Mental Health Journal, 36(4), 413-424.

Pinto-Foltz, M. D., Logsdon, M. C., & Myers, J. A. (2011). Feasibility, acceptability, and
initial efficacy of a knowledge-contact program to reduce mental illness stigma
and improve mental health literacy in adolescents. Social Science & Medicine,
72(12), 2011-2019.

Quinnett, P. (1995). QPR: Ask a Question, Save a Life. Spokane, WA: The QPR Institute.
Retrieved from: www.qprinstitute.com.

Quinnett, P. (2007). QPR gatekeeper training for suicide prevention: The model,
rationale, and theory. Retrieved from https://qprinstitute.com/research-theory

Reavley, N. J., McCann, T. V., & Jorm, A. F. (2012). Mental health literacy in higher
education students. Early Intervention in Psychiatry, 6(1), 45-52.

Rosenquist, J.N., Fowler, J.H., Christakis, N.A. (2011). Social networks determinants of
depression. Molecular Psychiatry, 16, 273-281

Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1988). Social learning theory and the
health belief model. Health Education Quarterly, 15(2), 175-183.

Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief measure for
assessing generalized anxiety disorder: the GAD-7. Archives of Internal
Medicine, 166(10), 1092-1097.
 103

Tompkins, T. L., Witt, J., & Abraibesh, N. (2010). Does a gatekeeper suicide prevention
program work in a school setting? Evaluating training outcome and moderators of
effectiveness. Suicide and Life-threatening Behavior, 40(5), 506-515.

Wasserman, D., Hoven, C. W., Wasserman, C., Wall, M., Eisenberg, R., Hadlaczky, G.,
… Carli, V. (2015). School-based suicide prevention programmes: The SEYLE
cluster-randomised, controlled trial. The Lancet, 385(9977), 1536-1544.

Wolff, G., Pathare, S., Craig, J., & Leff, J. (1996a). Community knowledge of mental
illness and reaction to mentally ill people. The British Journal of Psychiatry,
168(2), 191-198.

Wolff, G., Pathare, S., Craig, T., & Leff, J. (1996b). Community attitudes to mental
illness. The British Journal of Psychiatry, 168(2), 183-190

Wolff, G., Pathare, S., Craig, T., & Leff, J. (1996c). Public education for community
care: a new approach. The British Journal of Psychiatry, 168(4), 441-447.

Wyman, P. A., Brown, C. H., Inman, J., Cross, W., Schmeelk-Cone, K., Guo, J., & Pena,
J. B. (2008). Randomized trial of a gatekeeper program for suicide prevention: 1-
year impact on secondary school staff. Journal of Consulting and Clinical
Psychology, 76(1), 104-115.

Zalsman, G., Hawton, K., Wasserman, D., van Heeringen, K., Arensman, E.,
Sarchiapone, M., … Zohar, J. (2016). Suicide prevention strategies revisited: 10-
year systematic review. The Lancet Psychiatry, 3(7), 646-659.
 104

CHAPTER 4

GENERAL DISCUSSION

Currently, the most common approaches to mental health issues prevention

programming on college campuses harness the model of mental health literacy (Jorm,

2012). More specifically, schools have implemented the community-based programs of

Mental Health First Aid (Kitchener & Jorm, 2006) and Question Persuade Refer trainings

(Quinnett, 2007; Wyman et al., 2008). These programs target students as gatekeepers and

help them develop valuable skills to prevent and refer students with mental health issues

to treatment. These programs are supported as being effective and often specifically

evaluate students’ improvement in declarative knowledge (Wyman et al., 2008) and self-

efficacy in the five components of mental health literacy (Jorm et al., 1997; O’Connor &

Casey, 2015). While these approaches are useful, these assessment strategies largely

ignore the processes involved in developing the identity of a mental health advocate.

Additionally, these prevention programs are not traditionally offered as part of degree-

seeking programs and as a result are not reaching as many students as possible.

Collectively, between studies one and two, the primary purpose of this

dissertation was to address these two gaps in the literature. Study one focused on

strengthening current assessment techniques by integrating past strategies with

developmental theory. This led to developing a process-based mental health literacy

assessment: The Mental Health Awareness and Advocacy: Assessment Tool (MHAA-

AT). Study two sought to address the gap of mental health literacy programming as part

of a degree seeking programs on college campuses. Accordingly, the Mental Health

 105

Awareness and Advocacy (MHAA) curriculum was created and evaluated for its

effectiveness in helping students acquire declarative knowledge, build self-efficacy, and

apply skills (behaviors) within the larger domains of identifying mental health issues,

locating evidence-based resources, and responding to mental health issues.

A Process-Based Approach to Assessing Mental Health Literacy

There are several high-quality measurement devices that evaluate participants’

mental health literacy in the literature base (O’Connor & Casey, 2015; Wyman et al.,

2008). The primary area of growth needed in these measures is to emphasize the process-

based components of development and learning, specifically the Health Belief Model

(Becker, 1974) and components of Social Cognitive theory (Bandura, 1982, 2005).

Excitingly, psychometric results from an MHAA-AT across a wide range of college

students garnered support for the process-oriented approach (outlined in Figure 4.1). This

assessment tool will allow for researchers to target more specific outcomes (e.g.,

declarative knowledge within locating evidence-based resources) and help identify

specific intervention points when working with college populations. More specific

information related to each of the item types are described below.

Declarative Knowledge

Arguably the most exciting component of study one is related to the Item

Response Theory analyses. These analyses indicate that the microprocess of acquiring

declarative knowledge has appropriate item, person, and trait level characteristics that fit

into each macroprocess (i.e., Identifying, Locating, and Responding) that makes up
 106

Identifying Domain

a) Declarative Knowledge
b) Self-Efficacy
c) Behaviors

Locating Domain Responding Domain

a) Declarative Knowledge a) Declarative Knowledge

b) Self-Efficacy b) Self-Efficacy
c) Behaviors c) Behaviors

Figure 4.1. Process-based model of mental health awareness and advocacy.

mental health literacy. This is possibly the most unique advantage of MHAA-AT, in and

above prior declarative knowledge assessments in existing mental health literacy

measures (Jung, von Sternberg, & Davis, 2016; O’Connor & Casey, 2015). There is,

however, need to evaluate the content of several items (see study one of this dissertation)

and the content of the Responding domain to ensure that the true intent of the domains is

being achieved.

Self-Efficacy

The self-efficacy subscale of the MHAA-AT had the strongest psychometric

properties and detected the largest posttest effects in the evaluation of the curriculum in

study two. Developing the self-efficacy items for the MHAA-AT was guided by theory

from studies of self-efficacy in other contexts (Bandura, 1982, 2005) and is commonly
 107

used in other measures (Wyman et al., 2008). It is interesting that theory on self-efficacy

posits that as self-efficacy increases, behaviors should also increase. This effect is slightly

supported in the bivariate correlations of study one of this dissertation and the pretest

bivariate correlations of study two. The relationship between these variables was not

identified in the bivariate correlations between the posttest measurement of study two.

This relationship and the nature of the behavior questions needs to be explored more in

future research, perhaps with the inclusion of open-ended responses in future posttest

evaluations of MHAA, or in focus groups after course completion. For example, the

researcher could ask, “if referrals were not made, can you explain why?” to see if there is

reduced need to refer once students better understood the mental health needs of their

friends and family, or if they simply did not have scenarios where referrals were

necessary. This will be further discussed below.

Behaviors

The behavior subscale of the MHAA-AT had the most unexpected findings (null

findings) of the new assessment tool, in terms of response to the MHAA course.

Currently, the statistical analyses indicate that the measure is sound and can be used to

evaluate students’ behaviors related to the three subdomains of mental health literacy

(i.e., identifying mental health issues, locating evidence-based resources, and responding

to mental health issues). Despite these sound psychometric properties, the assessment

tool did not detect effects in the evaluation of the curriculum. While this could indicate

that the intervention is not effective at increasing direct behavior there is also need to

consider if the assessment tool is sensitive to behavioral change. For instance, the
 108

questions ask ‘within the last three months’ have you participated in a particular

behavior. This time frame could be skewing the results and the questions might

potentially be better asked at a 3-month follow-up following the administration of the

curriculum. Additionally, measuring direct behaviors via self-report is a traditionally

challenging approach comparatively to using direct, trained observers and there are sound

arguments to not use self-report measurements for behavioral outcomes (Baumeister,

Vohs, & Funder, 2007). Future iterations of the assessment tool will need to explore these

issues in more depth to ensure the accurate strengthening of the assessment tool and

curriculum to help achieve behavioral change.

Can Mental Health Literacy be Offered in Course Format as Part

of a Degree-Seeking Program?

In short, yes, mental health literacy can be offered as part of a degree-seeking

program. Study two provided a quasi-experimental proof-of-concept for a college-based

mental health literacy curriculum that can be used as part of a degree seeking program at

a university. The study provides a framework for a curriculum that can be taught at the

upper-division level at a college or university. Additionally, the curriculum could be

easily used as an elective to fit the needs of a general education requirement course. As

with any college course, the content can be adapted and updated over time. Reading

assignments could be changed to reflect updated trends, and/or to be more specific to a

discipline (Social Work, Education, etc.).

Despite not being included in the analysis study one or two of this dissertation,
 109

IDEA teaching evaluations garnered through the course have been very positive and

further support the proof of concept, namely that the course was a positive learning

experience across domains not assessed in the MHAA-AT. For instance, one student

stated the following:

I feel that this class should be something that is required for everyone to take. I
have learned so many things that are helpful to me as a community member and
that I will be able to use for the rest of my life. I think that [if] everyone was
trained in recognizing when a mental health issue is potentially present in
someone they spend a lot of time with, so many college students would not have
to suffer alone or feel that they are crazy for feeling a way that a lot of other
people do as well.

Similarly, themed comments are common throughout the delivery of the course.

There are also additional feedback points where students detail specific scenarios

of them responding to family members or classmates that are experiencing mental

health issues. One particularly meaningful example has been the countless

qualitative points in class where students have explained that the course material

have helped, they themselves, receive services and feel more supported as they

pursue their education.

Does the Curriculum Improve Mental Health Literacy?

Similar to other evaluative studies of mental health literacy programs (Jorm,

2012; Lipson, Speer, Brunwasser, Hahn, & Eisenberg, 2014), the MHAA curriculum is

effective at increasing knowledge and self-efficacy related to mental health literacy

outcomes. More specifically, the curriculum has measurable influences in increasing

students’ knowledge related to identifying mental health issues and locating evidence-
 110

based resources. The curriculum also positively influenced students’ self-efficacy related

to mental health literacy in each microprocess described in the MHAA-AT (see Figure

3.1). There is not, however, a detectable effect related to acquiring knowledge related to

responding to mental health issues.

The null findings in both the microprocesses of declarative knowledge and

behaviors related to the macroprocess of Responding to mental health issues requires

more attention. A post-hoc speculation is that current questions emphasize demographic

traits of suicidality (refer to the Appendix). In reviewing the MHAA curriculum content

surrounding this macroprocess, most of the content currently emphasizes declarative

knowledge of skills, rather than descriptive factors of responding to mental health issues.

This is largely guided by students’ request during the delivery of the curriculum to

identify more appropriate skills in responding to their suicidal peers. Following the above

line of logic, I would anticipate an increase in behaviors by students related to responding

to mental health issues, but as described there was a null finding. Another possible

explanation for lack of findings in the microprocess of behaviors, mentioned above

relates to the timing of the questions. The posttest may be too proximal to exposure to the

curriculum: students may not have had an opportunity to yet interact with individuals in

their communities that are experiencing mental health issues and thus react. Qualitative

IDEA course evaluation feedback and comments from students in their assignments,

stated that they have felt more comfortable interacting with peers and have even made

referrals during the course. However, the opportunity to react and refer may be limited to

just a few students who were provided that opportunity. Of note, at both time one and
 111

time two on the behavioral questions, dispersion of response is limited: on a 0-6 scale,

most answers are around 3-4. This may mean reduced variability, or little room to

improve over time. In short, more work is needed to sort out these issues and the extent to

which these findings reflect the current sample or broader issues in course content or

measurement.

Does Modality Influence Mental Health Literacy Outcomes?

The results of study two of this dissertation suggest that there are modality

differences (in-person/online X time) on several outcomes (e.g., MHAA-AT: Declarative

Knowledge Responding) when there are not Time X Condition effects or significant three-

way interaction effects (in-person/online X treatment/control X time). This finding is

curious as I hypothesized the interaction to be significant for the three-way interaction,

but it is not. One possible explanation for these types of findings is explored in the

literature and is related to students in in-person classes staying more engaged with

content than they are in online courses (Kemp & Grieve, 2014). Speaking to these factors

qualitatively as an instructor, there seems to be consistent utility in both courses. While

the in-person course allows for more in-depth report building with students and more

personal confidence in delivering feedback to students on skill development, there were

not significant statistical differences between the modalities. This largely suggests that

the modality does not have a significant effect and MHAA can be offered both in-person

and online equally effectively. This is an important finding as it could possibly allow for

the scaling of the MHAA curriculum in a more rapid and cost-effective manner (online
 112

delivery may be less expensive for the university and may offer an even wider reach of

students). There is, however, more research needed to explore the modality effects in the

MHAA curriculum.

Future Directions

To strengthen the MHAA-AT, replication studies need to evaluate the knowledge

questions and pretest/posttest analyses to determine retest reliability. The self-efficacy

and behavior domains of the measure need to be examined using confirmatory factor

analyses approaches to ensure that current factor structures that were identified in study

one remains consistent. Lastly, there is need to examine the specific nature of the

behavior questions to better determine their utility and sensitivity to change. Once these

steps are completed, the MHAA-AT could be expanded to additional college populations

to ensure the strength and consistency of the assessment tool.

To strengthen the MHAA curriculum, there is need to determine how to better

address declarative knowledge pertaining to responding to mental health issues. The

course curriculum could better address demographic factors related to mental health

issues, specifically suicidology (see the Appendix for questions of the MHAA-AT:

Declarative Knowledge Responding). Additionally, there is need to evaluate if and how the

course can encourage students to make more purposeful action related to mental health

literacy. For instance, there is not currently a statistical explanation of why students are

not having considerable measurable effects on the behavior items. Is this an assessment

issue? Or, are students facing other barriers that prohibit them from taking action that
 113

could be addressed in the curriculum? Lastly, there is a possibility to incorporate other

psychological interventions (e.g., online ACT interventions) into the curriculum to

directly address students’ own mental health issues, allowing the course to impact student

anxiety and depressive symptoms.

Conclusion

The results of study one indicates that the MHAA-AT is a sound measurement

and can be used to evaluate the effectiveness of mental health literacy programs (e.g.,

mental health first aid, MHAA programs, and other gatekeeping trainings). The findings

from study two indicate that the MHAA curriculum is effective at improving students’

key outcome variables related to mental health literacy. The findings of both studies

provide exciting opportunities for both future research and the potential for future

prevention programming on college campuses. More specifically, these studies open the

door to offer targeted interventions on college campuses across the nation. In the future,

work should emphasize developing a deeper evidence-base for the Mental Health

Awareness and Advocacy Assessment Tool and the Mental Health Awareness and

Advocacy curriculum by purposefully disseminating it to universities that are attempting

to prevent college students’ mental health issues.

References

Bandura, A. (1982). Self-efficacy mechanism in human agency. American Psychologist,

37(2), 122.

Bandura, A. (2005). The primacy of self‐regulation in health promotion. Applied

Psychology, 54(2), 245-254.
 114

Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007). Psychology as the science of
self-reports and finger movements: Whatever happened to actual behavior?
Perspectives on Psychological Science, 2(4), 396-403.

Jorm, A. F. (2012). Mental health literacy: Empowering the community to take action for
better mental health. American Psychologist, 67(3), 231-243.

Jorm, A. F., Korten, A. E., Jacomb, P. A., Christensen, H., Rodgers, B., & Pollitt, P.
(1997). “Mental health literacy”: A survey of the public's ability to recognize
mental disorders and their beliefs about the effectiveness of treatment. Medical
Journal of Australia, 166(4), 182-186.

Jung, H., von Sternberg, K., & Davis, K. (2016). Expanding a measure of mental health
literacy: Development and validation of a multicomponent mental health literacy
measure. Psychiatry Research, 243, 278-286.

Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates' opinions

and test performance in classroom vs. online learning. Frontiers in Psychology, 5,
1278. doi: 10.3389/fpsyg.2014.01278

Kitchener, B. A., & Jorm, A. F. (2006). Mental health first aid training: Review of
evaluation studies. Australian and New Zealand Journal of Psychiatry, 40(1), 6-8.

Lipson, S. K., Speer, N., Brunwasser, S., Hahn, E., & Eisenberg, D. (2014). Gatekeeper
training and access to mental health care at universities and colleges. Journal of
Adolescent Health, 55(5), 612-619.

O’Connor, M., & Casey, L. (2015). The Mental Health Literacy Scale (MHLS): A new
scale-based measure of mental health literacy. Psychiatry Research, 229(1-2),
511-516.

Quinnett, P. (2007). QPR gatekeeper training for suicide prevention: The model,
rationale, and theory. Retrieved from https://qprinstitute.com/research-theory

Wyman, P. A., Brown, C. H., Inman, J., Cross, W., Schmeelk-Cone, K., Guo, J., & Pena,
J. B. (2008). Randomized trial of a gatekeeper program for suicide prevention: 1-
year impact on secondary school staff. Journal of Consulting and Clinical
Psychology, 76(1), 104-115.
 115

APPENDIX

MENTAL HEALTH AWARENESS AND ADVOCACY ASSESSMENT TOOL

 116

Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT)

The Mental Health Awareness and Advocacy Assessment Tool l(MHAA-AT)

consists of three types of items: 1) declarative knowledge items (30 items); 2) self-

efficacy items (20 items); and 3) behavior items (15 items). These items are then divided

into the three micro-processes that define mental health literacy: a) identifying mental

health issues; b) locating evidence-based resources; and c) responding to mental health

issues (see Figure 1 below). The items and corresponding sections are detailed below:

Figure 1. Process-Based Model of Mental Health Awareness and Advocacy

Identifying Domain
d) Declarative Knowledge
e) Self-Efficacy
f) Behaviors

Locating Domain Responding Domain

d) Declarative Knowledge d) Declarative Knowledge
e) Self-Efficacy e) Self-Efficacy
f) Behaviors f) Behaviors
 117

Item Breakdown and Scoring

Declarative Knowledge Items: Item 1-30. Total score of 30.

Identifying Mental Health Issues: Item 1-10. Total score of 10.

Locating Evidence-Based Resources: Item 11-20. Total score of 10.

Responding to Mental Health Issues: Item 21-30. Total score of 10.

Self-Efficacy Items: Item 1-20. Total score of 120, converted to average on each item.

Identifying Mental Health Issues: Item 1-7. Total score of 42, converted to

average score on each item.

Locating Evidence-Based Resources: Item 8-14 Total score of 42,

converted to average score on each item.

Responding to Mental Health Issues: Item 15-20. Total score of 36,

converted to average score on each item.

Behavior Items: Item 1-15. Total score of 90.

Identifying Mental Health Issues: Item 1-5. Total score of 30, converted to

average score on each item.

Locating Evidence-Based Resources: Item 6-10. Total score of 30,

converted to average score on each item.

Responding to Mental Health Issues: Item 11-15. Total score of 30,

converted to average score on each item.

 118

Declarative Knowledge Items

The following section will ask you questions about your understanding of issues
regarding mental health awareness and advocacy. Please select the response that most
accurately reflects your current understanding of the question. If you do not know the
answer, please select “I don’t know the answer” rather than guessing.

1. All the following symptoms are required for a person to be diagnosed with Major
Depressive Disorder EXCEPT for which one of the following?

a) Depressed mood most of the day

b) Diminished interest in regular activities
c) Inability to fall asleep, daily
d) Difficulty in controlling worry
e) I don't know the answer

2. All the following symptoms are required to be diagnosed with Major Depressive
Disorder EXCEPT for which one of the following?

a) Feeling keyed up or on edge

b) Feelings of worthlessness
c) Significant weight loss or gain
d) Recurrent thoughts of death
e) I don't know the answer

3. Individuals are more likely to experience symptoms of depression when they are
between the ages of:

a) 6-17 years old (1)

b) 18-29 years old (2)
c) 30-41 years old (3)
d) 41-52 years old (4)
e) I don't know the answer

4. Francis shows a lack of interest in school, consistent laziness, and is regularly

procrastinating his homework assignments. These behaviors could be likely indicators of
what mental health issue:

a) Major Depressive Disorder

b) Agoraphobia
c) Bipolar Disorder
d) Borderline Personality Disorder
e) I don't know the answer
 119

5. According to research on major depressive disorder (MDD), which statement is most

true?

a) Men are more likely to experience MDD

b) Women are more likely to experience MDD
c) Men and women are equally likely to experience MDD
d) There is no research about this difference
e) I don't know the answer

6. Which of the following regions has higher proportions of people experiencing

generalized anxiety disorder?

a) Europe
b) Asia
c) Latin America
d) Africa
e) I don't know the answer

7. All the following symptoms are required to be diagnosed with generalized anxiety
disorder EXCEPT for which one of the following?

a) Diminished interest in regular activities

b) Difficulty in controlling worry
c) Excessive anxiety and worry
d) Muscle tension
e) I don't know the answer

8. All the following symptoms are required to be diagnosed with generalized anxiety
disorder EXCEPT for which one of the following?

a) Sleep disturbance
b) Feeling keyed up or on edge
c) Easily fatigued
d) Feelings of worthlessness
e) I don't know the answer

9. Sage tells you that she often experiences her hands shaking, often is sweaty, and says
she is 'always worried about everything.' If she is diagnosed with a mental health
disorder, which of the following best fits her symptoms?

a) Major Depressive Disorder

b) Generalized Anxiety Disorder
c) Panic Disorder
d) Bipolar Disorder
 120

e) I don't know the answer

10. According to research on Generalized Anxiety Disorder, which statement is most true
about the age at which the disorder occurs?

a) The disorder is most likely to occur before the age of 12

b) The disorder is equally likely to occur at all ages, with the exception of infancy
c) The disorder is most likely to occur during puberty
d) The disorder is most likely to occur after the age of 40
e) I don't know the answer

11. Which of the following mental health providers cannot prescribe medications to treat
mental health issues?

a) Licensed Clinical Social Worker

b) Psychiatrist
c) Psychologist
d) Family Practice Doctor
e) I don't know the answer

12. All the following treatments have been supported by research as effective treatments
for generalized anxiety disorder EXCEPT?

a) Cognitive Behavioral Therapy

b) Acceptance Commitment Therapy
c) Rebirthing Therapy
d) Psychopharmacological (medication) treatment
e) I don't know the answer

13. Which of the following has been identified by research as being the most effective
treatment for severe major depressive disorder?

a) Talk therapy
b) Self-help books
c) Herbal supplements
d) Exercise
e) I don't know the answer

14. Which of the following mental health providers cannot provide talk therapy as a
treatment?

a) Clinical Social Worker

b) Marriage and Family Therapist
c) Licensed Practical Nurse
 121

d) Psychologist
e) I don't know the answer

15. Which of the following organizations does not provide community resources to help
prevent suicide?

a) American Foundation for Suicide Prevention

b) National Alliance on Mental Illness
c) World Health Organization
d) National Organization for Women
e) I don't know the answer

16. According to research, one of the biggest factors keeping college students from
seeking treatment for a mental health issue is:

a) Not having a supportive friend to help seek treatment

b) Not knowing their issues is severe enough for treatment
c) Not knowing where to get help for their issue
d) Not having the financial resources to pay for treatment
e) I don't know the answer

17. All of the following are examples of effective ways to combat stigma except:

a) Mass media campaigns

b) Educational courses
c) Public policy
d) All of these are examples of effective strategies
e) I don't know the answer

18. Which of the following is the most accurate about insurance companies and mental
health treatments?

a) Insurance companies always pay for all costs of mental health services
b) Insurance companies typically pay for a percentage of mental health services
c) Insurance companies never pay for mental health services
d) Insurance companies have not begun to discuss mental health service coverage
e) I don't know the answer

19. John says to his friend that his mom is crazy because she often stays in bed all day
and has to go to therapy every week. Which response provided below would be the most
effective at helping decrease the negative stigma of mental health issues expressed by
John?
 122

a) "Wow, crazy seems kind of harsh. At least she is getting help for her issues."
b) "Oh, my gosh. I had no idea your mom had a problem like that. How sad!"
c) "It seems like your mom may really struggle with a serious condition. Have you ever
thought how hard that would be for her to handle?"
d) "I don't even know how you handle it, John!"
e) I don't know the answer
20. Anne tells you that she is looking for someone to help her manage her medications
and receive talk therapy. Who is the most appropriate mental health provider to refer her
to?

a) Psychiatrist
b) Marriage and Family Therapist
c) Clinical Social Worker
d) Family doctor
e) I don't know the answer

21. According to research, one of the most important factors in predicting the
improvement of a mental health issue is:

a) The individuals gender

b) The individual's quality of social support
c) The individual's family history of a mental health issues
d) The individual's age
e) I don't know the answer

22. Jane arrives late to class and she tells you that she just doesn’t want to keep trying.
Jane then explains that she thinks everyone would be better off if she just wasn’t around
anymore. Jane said she would prefer to just end her life. Based on what Jane has said,
what is the most likely conclusion to be made about Jane?

a) She is currently experiencing symptoms of generalized anxiety disorder.

b) She is currently experiencing symptoms of major depressive disorder
c) She is currently experiencing symptoms of suicidality.
d) She is currently experiencing major depressive disorder with suicidal thoughts.
e) I don't know the answer

23. According to research, which of the following factors is most important to consider if
you are trying to intervene with someone that is suicidal?

a) If they have past, unsuccessful suicide attempts

b) If they have a plan to attempt suicide
c) If they have the means to complete a suicide
d) These factors are all important to consider together
e) I don't know the answer
 123

24. According to research, who is most likely to attempt suicide?

a) Males
b) Females
c) They are equally likely
d) There is not a clear answer provided by research
e) I don't know the answer

25. According to research, who is at a higher risk to die by suicide?

a) Males
b) Females
c) They are equally likely
d) There is not a clear answer provided by research
e) I don't know the answer

26. According to research, what age group is at the highest risk to die by suicide?

a) Childhood (0-12 years old)

b) Adolescents (12-24 years old)
c) Middle age (45-64 years old)
d) Older adults (85+ years old)
e) I don't know the answer

27. According to research, what race/ethnicity is at a higher risk to die by suicide?

a) White
b) Black
c) American Indian
d) Hispanic
e) I don't know the answer

28. According to research, asking someone directly if they are suicide has what effect?

a) Increases the likelihood they will attempt suicide

b) Decreases the likelihood they will attempt suicide
c) Neither increases or decreases the likelihood they will attempt suicide
d) There is not a clear answer provided by research
e) I don't know the answer

29. Who is the most appropriate person to work with an individual that is suicidal?

a) Family practice doctor

b) Registered Nurse
 124

c) School administrator
d) Clinical psychologist
e) I don't know the answer

30. What is the first step you should take when someone tells you they are suicidal?

a) Leave the person and immediately contact a therapist

b) Persuade the person to stay with you until you can find help
c) Ask the person if they have been suicidal in the past
d) Only worry about the individual if they have a specific plan
e) I don't know the answer

Self-Efficacy Items

The following statements describe situations regarding mental health issues. Read each
statement and then respond by indicating your current (at this moment) level of
confidence with completing the task described in each statement.

1. I can identify each of the diagnostic criteria for major depressive disorder.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

2. I can identify each of the diagnostic criteria for generalized anxiety disorder.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

3. I can identify when someone is experiencing signs of depression based on their

behaviors and thoughts they are sharing with me.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
 125

e) Almost completely confident

f) Completely confident

4. I can identify when someone is experiencing signs of anxiety based on their behaviors
and thoughts they are sharing with me.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

5. I understand the clinical symptoms that indicate when someone may be experiencing
more severe than 'normal' feelings experienced in life.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

6. I understand the differences between regular sadness and nervousness compared to

major depressive disorder and generalized anxiety disorder.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

7. I can explain the difference between depression and anxiety accurately.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident
 126

8. I know at least three national organizations that work to prevent mental health issues or
suicide.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

9. In my experience, having conversations about mental health issues could help to

decrease stigma attached to mental health.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

10. I can identify the evidenced-based treatments that are most effective at treating
mental health issues.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

11. I can have conversations about mental health issues based on factual information.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

12. I can tell the difference between an empirically supported treatment and a non-
empirically supported treatment.

a) Not at all confident

b) A little confident
 127

c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

13. I can determine if a specific insurance plan covers the expenses of accessing mental
health resources.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

14. I can identify who to contact in my community and state to advocate for increased
resources for mental health issues.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

15. I can identify and access mental health resources in my community.

a) Not at all confident
b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

16. I can identify when someone needs professional help due to emotional or behavioral
problems.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident
 128

17. I can talk to someone about accessing mental health resources for depression or
anxiety issue in a kind and empathetic manner.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

18. I understand how to make appropriate referrals to mental health services when
someone needs help for a mental health issue.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

19. I know how to set healthy boundaries with someone when they are experiencing
consistent mental health issues that help keep us both safe.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Mostly confident
e) Almost completely confident
f) Completely confident

20. I know how to ask questions to better understand someone's current mood and
thoughts and if they pose a threat of harm to themselves or others.

a) Not at all confident

b) A little confident
c) Somewhat confident
d) Almost completely confident
e) Completely confident

Behavior Items

The following statements will describe a situation regarding mental health issues that you
may have encountered in the past three months. Read each statement and then indicate
 129

the frequency by which you have personally participated in the described behavior.

1. How often in the past three months have you recognized in someone that you know
reasonably well, symptoms that could be indicative of a diagnosable mental health issue?

a) Not applicable; No one I know has mental health issues

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ times

2. How often in the past three months have you recognized that someone you know
reasonably well is exhibiting symptoms or behaviors that are diagnosable characteristics
of depression?

a) Not applicable; No one I know has depression

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 times
g) 6+ Times

3. How often in the past three months have you recognized that someone you know
reasonably well is exhibiting symptoms or behaviors that are diagnosable characteristics
of anxiety?

a) Not applicable; No one I know has anxiety

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

4. How often in the past three months have you recognized that someone that you know
reasonably well has experienced a mental state (e.g., sadness, nervousness, depression,
anxiety) that has affected their relationships with others (e.g., friends, family members,
co-workers)?

a) Not applicable; No one I know has had this experience

b) 0 Times
 130

c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

5. How often in the past three months have you recognized that someone that you know
reasonably well has had a mental state (e.g., sadness, nervousness, depression, anxiety)
that has affected their ability in school, their quality of work, or their home life?

a) Not applicable; No one I know has had this experience

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

6. In the past three months have you engaged someone you know reasonably well in a
conversation about the importance of professionally treating their mental health issues?

a) Not applicable; No one I know has mental health issues

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

7. In the past three months, have you talked with someone that you know reasonably well
about the negative effects of not treating a mental health issue as soon as symptoms
arise?

a) Not applicable; No one I know has mental health issues

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times
 131

8. How often in the past three months have you contacted a mental health provider to
help someone that you know reasonably well access mental health resources?

a) Not applicable; No one I know has needed these resources

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

9. How often in the past three months have you researched or called a mental health
provider to find the best treatment option available for a mental health issue that someone
you know reasonably well is experiencing?

a) Not applicable; No one I know has a mental health issue

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

10. How often in the past three months have you researched or contacted an insurance
agency for someone that you know reasonably well to see if they will pay for mental
health services?

a) Not applicable; No one I know has needed these resources

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

11. How often in the past three months have you asked someone that you know
reasonably well who showed signs/symptoms of a mental health issue if they are doing
'okay' or if they needed help?

a) Not applicable; No one I know has had this experience

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
 132

f) 4-5 Times
g) 6+ Times

12. How often in the past three months have you encouraged someone that you know
reasonably well who was experiencing emotional or behavioral problems to seek help
from a professional?

a) Not applicable; No one I know has had this experience

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

13. How often in the past three months have you helped someone that you know
reasonably well who was experiencing symptoms of depression or anxiety receive help
from a professional?

a) Not applicable; No one I know has had depression or anxiety

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

14. How often in the past three months have you told someone that you know reasonably
well, who was considering suicide, to get help from a professional?

a) Not applicable; No one I know has had this experience

b) 0 Times
c) 1 Time
d) 2 Times
e) 3 Times
f) 4-5 Times
g) 6+ Times

15. How often in the past three months have you helped someone who was considering
suicide to get help from a professional?

a) Not applicable; No one I know has had this experience

 133

b) 1 Time
c) 2 Times
d) 3 Times
e) 4-5 Times
f) 6+ Times
 134

CURRICULUM VITAE

TY B. ALLER, LMFT
545 West 465 North, Suite 130
Providence, UT 84332
435-890-0193 [email protected]

_________________________________EDUCATION__________________________

Utah State University, PhD Aug. 2019

Human Development and Family Studies
Dissertation: Mental Health Awareness and Advocacy:
Assessment Tool Development and Curriculum Evaluation
Advisor: Elizabeth Fauth, PhD & Scot Allgood, PhD, LMFT

Utah State University, M.M.F.T. May 2015

Emphasis: Marriage and Family Therapy
Advisor: Lori A. Roggman, PhD & Ryan Seedall, PhD, LMFT

Utah State University, B.S. May 2012

Majors: Psychology & Political Science
Cum Laude

_______________________CLINICAL EXPERIENCE_________________________

License Status: Marriage and Family Therapist Oct. 2018

Ty B. Aller, LMFT, PLLC Oct. 2018-Present

Individual, couples, & family counseling

The Center for Person’s with Disabilities Mar. 2019- Present

Training Coordinator
Supervisor: Dr. Mathew Wappett

The Family Place Aug. 2017-Mar. 2019

Individual, couples, & family counseling
Supervisor: Reece Neilson, PhD, LMFT & JaNae Sorenson, LCSW

Kent W. Anderson, PhD P.C. Mar. 2018-Oct. 2018

Individual, couples, & family counseling
Supervisor: Kent W. Anderson, PhD
 135

Life Directions Private Practice Aug. 2016-Mar. 2018

Individual, couples & family counseling
Supervisor: Ryan Seedall, PhD, LMFT & Pamela King, LMFT

Cache Valley Community Health Center Aug. 2014-Apr. 2015

Individual, couples & family counseling in collaborative health care setting
Supervisor: Dave Robinson, PhD, LMFT

Life-STAR Sept. 2014-Feb. 2015

Co-facilitator of couples support group for pornography addiction
Supervisor: Megan Oka, PhD, LMFT; Tyler Patrick, LMFT

Family Life Center Aug. 2013-Apr., 2015

Individual, couples, & family counseling for community and students
Supervisor: Kay Bradford, PhD, LMFT; Ryan Seedall, PhD, LMFT

Sherwood Hills Recovery Resort Jun. 2014-Aug. 2014

In-patient individual, couples, & family counseling for substance dependency.
Supervisor: Dave Robinson, PhD, LMFT; Tami Curtis, LCSW

Youth Track Residential Treatment Center Oct. 2013-Jan. 2014

Co-facilitator of family group treatment for adolescent sexual offenders
Supervisors: Dave Robinson, PhD, LMFT; Kevin Barlow, LMFT

________________________TEACHING EXPERIENCE_______________________

CURRICULUM DEVELOPMENT

Mental Health Awareness and Advocacy Jan. 2016-Present

Supervisor: Scot Allgood, PhD; Elizabeth B. Fauth, PhD

Advanced Home Visiting Practices, FCHD 5550 Aug. -Dec. 2016

Supervisor: Lori A. Roggman, PhD

Sticky Situations: What you didn’t learn about ethics in Kindergarten Aug.-Dec. 2014
Supervisor: Kay Bradford, PhD

Effective Parenting Practices: A support group for parents of teens May-Aug. 2014
Supervisor: Dave Robinson, PhD

Parenting and Child Guidance, FCHD 2660 Aug.-Dec. 2012

 136

Supervisor: Kay Bradford, PhD

Drug and Alcohol Treatment Program Jan.-May 2010

Supervisor: Kent W. Anderson, PhD

GRADUATE INSTRUCTOR

HDFS/PSY 3700- Online (3 Credits): 110 Students Spring, 2019

Mental Health Awareness and Advocacy

HDFS/PSY 3700- Face-to-Face (3 Credits): 65 Students Spring, 2019

Mental Health Awareness and Advocacy

HDFS/PSY 3700- Online (3 Credits): 96 Students Fall, 2018

Mental Health Awareness and Advocacy

HDFS/PSY 3700- Face-to-Face (3 Credits): 56 Students Fall, 2018

Mental Health Awareness and Advocacy

FCHD/PSY 3700- Online (3 Credits): 98 Students Spring, 2018

Mental Health Awareness and Advocacy

FCHD/PSY 3700- Face-to-Face (3 Credits): 79 Students Spring, 2018

Mental Health Awareness and Advocacy

FCHD/PSY 3700- Online (3 Credits): 60 Students Fall, 2017

Mental Health Awareness and Advocacy

FCHD/PSY 3700- Face-to-Face (3 Credits): 44 Students Fall, 2017

Mental Health Awareness and Advocacy

FCHD 2400 (3 Credits): 24 Students Fall, 2016

Marriage and Family Relationships

GRADUATE TEACHING ASSISTANT

FCHD 2200- Online (3 Credits) Fall, 2016-17

Introduction to Home Visiting

FCHD 2400 (3 Credits) Fall, 2015-17

Marriage and Family Relationship
 137

FCHD 1500 (3 Credits) Fall, 2015-Spring, 2016

Development Across the Lifespan

FCHD 3570- Online (3 Credits) Fall, 2015- Spring 2016

Youth and Adolescence

FCHD 2660 (3 Credits) Fall, 2012-Spring, 2015

Parenting and Child Guidance

INVITED ACADEMIC LECTURES

Crafting Mental Health Policy: Do’s and Don’ts of Utah Feb. 2018
FCHD 7230: Family and Social Policy
Utah State University

Family Intervention: Emotionally Focused Therapy Apr. 2017

FCHD 2400: Marriage and Family Relationships
Utah State University

Family Intervention: History of Family Therapy Apr. 2017

FCHD 2400: Marriage and Family Relationships
Utah State University

Parenting: Baumrind’s Typologies Mar. 2017

FCHD 2400: Marriage and Family Relationships
Utah State University

Parenting: Using the PICCOLO Measure Mar. 2017

FCHD 2400: Marriage and Family Relationships
Utah State University

Parenting Discipline: A Strengths Based Approach Mar. 2017

FCHD 2400: Marriage and Family Relationships
Utah State University

From Science to Policy: College Mental Health Issues Feb. 2017

FCHD 7230: Family and Social Policy
Utah State University

Social-Emotional Development in Early Childhood Oct. 2015

FCHD 1500: Lifespan Development
Utah State University
 138

A Role Play for Circular Questioning Oct. 2014

FCHD 6310: Foundations in Marriage and Family Therapy
Utah State University

Fathering: Issues to Consider Aug. 2014

FCHD 2660: Parenting and Child Guidance
Utah State University

COMMUNITY PRESENTATIONS

College Students’ Mental health: Building a community of Support Apr. 2017

Oasis Community Group
Logan, Utah

A Student Perspective on the College Mental Health Crisis Feb. 2017

Legislative Spouses Luncheon
Salt Lake City, Utah

Mental Health Resources for Students in Need Dec. 2016

School of Graduate Studies: Research Faculty Training
Utah State University

Enriching the Couple Relationship Nov. 2016

Student Housing, ‘Night with the Expert’
Utah State University

Mental Health Toolbox for Couples Oct. 2016

Student Housing, ‘Night with the Expert’
Utah State University

Parenting through Depression Oct. 2016

Student Housing, ‘Night with the Expert’
Utah State University

Mental Health Resources for Graduate Students Aug. 2016

Graduate Training Series, School of Research and Graduate Studies
Utah State University

Developmental Parenting Jun. 2016

Student Housing Night with the Expert
Utah State University

Using Your Dreams as Motivation Mar. 2016

Adolescent High School Retreat
 139

Utah State University

Finding your passion: Identifying goals to propel you to success. Mar. 2015
Adolescent High School Retreat
Utah State University.

Managing Your Time Effectively: Tools to Beat the Clock Apr. 2014
Gray Matters Alzheimer’s Prevention Project
Authorship: Aller, T.B., Lachmar, E.M., & Robinson, W.D.

Parenting: Finding tools to decrease stress. Apr. 2014

Gray Matters Alzheimer’s Prevention Project
Authorship: Aller, T.B., Lachmar, E.M., & Robinson, W.D

Mindfulness and Stress Management Apr. 2014

Gray Matters Alzheimer’s Prevention Project
Authorship: Robinson, W.D., Aller, T.B., & Lachmar, E.M.,

Stress and Cognitive Health Apr. 2014

Gray Matters Alzheimer’s Prevention Project
Authorship: Robinson, W.D., Aller, T.B., & Lachmar, E.M.

INVITED WORKSHOPS

Channeling your Voice as a Student Leader Nov. 2018

Upstander Conference
Utah State University

Making University Policy: A step-by-step guide Mar. 2017

Student Involvement and Leadership Cente
Utah State University

Facilitating University-Wide Change: A team effort Mar. 2017

Student Involvement and Leadership Center
Utah State University

Youth Leadership: Channeling your voice to facilitate change Mar. 2017

Utah Youth Council Association
Utah State University
 140

PROFESSIONAL DEVELOPMENT

Graduate Instructor’s Forum Spring, 2017

Instructor: Troy Beckert, PhD

Graduate Instructor’s Forum Fall, 2016

Instructor: Troy Beckert, PhD

Graduate Instructor’s Forum Spring, 2016

Instructor: Elizabeth Fauth, PhD

________________________RESEARCH EXPERIENCE_______________________

REFEREED RESEARCH GRANTS

Aller, T.B., Novak, J. (2017). Mental Health Awareness and Advocacy: Measurement
Development in a Community Sample. Graduate Research and Collaborative
Opportunities Grant, Utah State University. Award amount: $1000.00

REFEREED PUBLICATIONS

Aller, T.B., Tekarli, N., & Rex, J. (2017). ‘What we Wish we Had Known’: Experiences
of student leaders and their motivations to grow. Journal of Student Leadership,
1(2), 48-60.

Aller, T.B., (2017). Student Leaders as Advocates: A collaborative approach to the

college mental health problem. Journal of Student Leadership, 1(1), 1-12.

Evans, C., Higgins, J.P., Aller, T.B., Chavez, J., Piercy, K. (2017). Role balance and
Leisure Activities with Newlywed Couples: A phenomenological study. Marriage
and Family Review, 54(2), 105-127, DOI:
http://dx.doi.org/10.1080/01494929.2017.1297756

Aller, T.B, Piercy, K., Roggman, L. (2014). ‘Helping us find our own selves’:
Exploring father role construction and early childhood program engagement.
Early Child Development and Care, 185(3), 360-376, DOI:
https://doi.org/10.1080/03004430.2014.924112
 141

REFEREED EXTENSION AGENCY PUBLICATIONS

Aller, T.B., Hall, K., Olson, T. (2017). Factsheet: Stepping Stones to Developmental
Success: Affectionate Parenting. Utah State University Extension Agency.

REFEREED CONFERENCE PRESENTATIONS

Aller, T.B., Piercy, K. (2017). Social Policy: A Guide for Social Scientists in the Academy.
Paper presented at the annual Utah Academy of Science, Arts, and Lectures. Utah
Valley Univeristy: Orem, Utah.

Aller, T.B., Dymock, J., Roggman, L.A., Seedall, R. (2014). Nonresidential Fathers and
Children: Implications for Therapy. Poster presented at the annual conference of
the American Association of Marriage and Family Therapy, Milwaukee, WI.

Aller, T.B., Seedall, R., Roggman, L.A. (2014). Depression in Families: Using the
PICCOLO Measure in Therapy. Poster presented at the annual conference of the
American Association of Marriage and Family Therapy, Milwaukee, WI

Seedall, R., Aller, T.B., Lachmar, M., Barker, C., (2014). Understanding Disability from
a Social Justice Perspective. Poster presented at the annual conference of the
American Association of Marriage and Family Therapy, Milwaukee, WI

Seedall, R., Barker, C., Lachmar, M., Aller, T.B., (2014). The Role of Attachment During
Positively-Themed Interactions. Poster presented at the annual American
Association of Marriage and Family Therapy, Milwaukee, WI

Aller, T.B., Roggman, L.A. (2014). Predictors of Caregiving by Nonresident Fathers.

Poster presented at the biannual conference of the Society for Research in Human
Development, Austin, TX.

Aller, T.B., Olsen, T., Williams, R., Hill, A., Gurko, K., Broome, M., Roggman, L.A.
(2014). Building Blocks: A Case Study of Project Based Learning in Human
Development Research. Poster presented at the biannual conference of the Society
for Research in Human Development, Austin, TX.

Roper, S. W., Seedall, R. B., & Aller, T. B. (2014). The relationship effects of parental
divorce. Poster presented at the Utah Council on Family Relations, Provo, UT.

Aller, T.B., Anderson, S., Roggman, L.A. (2012). Early family environment and
children’s 5th grade language and literacy outcomes. Poster presented at the
annual National Conference of Undergraduate Research, Ogden, UT.
 142

NONREFEREED CONFERENCE PRESENTATIONS

Evans, C., Aller, T.B., Roggman, L.A. (2014). Paternal Depression and Children’s
Developmentof Emotion Regulation. Poster presented at the annual Utah State
University Student Showcase, Logan, UT.

Broomé, M., Aller, T.B., Roggman, L.A. (2013). Involving Undergraduates in Human
Development Research: A case study. Poster presented at the annual Utah State
University Department of Family, Consumer, and Human Development Student
Showcase, Logan, UT.

Aller, T.B., Anderson, S., Skogrand, L., Roggman, L.A. (2013). Head Start and Early
Head Start Responsiveness to Culture: A Case Study of a Latino Father. Paper
presented at the annual Utah State University Student Showcase, Logan, UT.

Aller, T.B., Anderson, S., Roggman, L.A., (2012). Early father language interactions
and children's 5th grade reading achievement. Poster presented at the annual
Utah State University Student Showcase, Logan, UT.

COMMUNITY OUTREACH PUBLICATIONS

Aller, T.B. (2017). Communication time: Engaging Your Children. Conference for Moms.
Aller, T.B. (2017). Mindful Mom: Three steps to a mindful day. The Hatmaker’s Suitcase.
Aller, T.B. (2017). Parenting Strengths, The four domains of success. The Hatmaker’s
Suitcase.
Aller, T.B. (2017). Every Parent has Strengths, what are yours? The Hatmaker’s Suitcase.
Aller, T.B. (2017). Vote. Vote. Vote. Utah Statesman: Utah State University
Aller, T.B. & Maners, M. (2016). A vision for the future: Mental health awareness. Utah
Statesman: Utah State University.
Aller, T.B., (2014). Depression: A shadow in our lives. Utah Statesman: Utah State
University.
Aller, T.B., (2014). Perfectionism- walking the tightrope. Utah Statesman: Utah State
University.
Aller, T.B. (2014). It’s time to make a decision. Utah Statesman: Utah State University.
Aller, T.B. (2014). How to succeed during finals. Utah Statesman: Utah State University.

GRADUATE RESEARCH ASSISTANTSHIP

Home Visiting Observation Measure Jan. 2016-May 2017

Supervisor: Lori A. Roggman, PhD

Parenting Research Review Aug. 2014-Aug. 2015

Supervisor: Lori A. Roggman, PhD
 143

____________________________SERVICE ROLES___________________________

ELECTED POSITIONS

Graduate Studies Senator (Two Terms) Mar., 2015-Mar., 2017

Utah State University Student Association
Utah State University

Initiatives, 2016-17:

Initiative Outcome
Improve the quality of the Graduate Passed student legislation amending the
Research and Collaborative Opportunities GRCO process to include oversight by the
Grant Office of Research and Graduate Studies.
Awarded students from each of the eight
Increase the diversity of the portfolio for
colleges at Utah State University, and from
the Graduate Enhancement Award
12 different departments.
1. Sponsored university legislation
declaring mental health issues a crisis.
2. Co-authored state resolution declaring
Increase awareness and access to Mental
mental health issues a crisis in the Utah
Health Resources for students.
System of Higher Education.
3. Continued the University Sponsored
Mental Health Week programming.
Create a University-wide graduate student Drafted and passed initial legislation
expectations document to inform students outlining the Graduate Student Rights and
of rights and work expectations. Expectations legislation.
For the first time in eight years, the
Graduate Studies Senator position has a
Increase the competitiveness of the
contested election. This was accomplished
Graduate Studies Senator Election.
by increased advertising and direct
encouragement of involved students.

Initiatives, 2015-16:

Initiative Outcome
Office of Research and Graduate Studies
Foster a richer graduate student social life
created and sponsored the monthly
on campus.
“Graduate Student Social.”
1. Increased partnerships for the annual
Increase awareness and access to Mental
mental health week to include direct,
Health Resources for students.
yearly university sponsorship.
 144

2. Raised $5,000 for the American

Foundation of Suicide Prevention.
Office of Research and Graduate Studies
Improve quality and efficiency of overhauled teaching assistant training that
Teaching Assistant training. then received improved teaching
evaluations by one standard deviation.
Founded the Graduate Student Council and
Increase transparency of Graduate student
mandated each university department have
representation.
a graduate student representative.

Media Coverage of Initiatives and Outcomes:

Mental Health State Resolution:

The Statesman: USUSA submits official mental health crisis resolution
The Herald Journal: Mental health bill from USU student government enters
Legislature

Mental Health a Crisis on College Campuses:

The Statesman: USUSA moves to declare a mental health crisis in Utah
The Herald Journal: USU student leaders declare campus mental health crisis
UPR: USU Student Government Declares University-Wide Mental Health Crisis

General Mental Health Advocacy:

The Herald Journal: Mental Health Club In Works At USU
The Statesman: ‘Mental Health is No Joke’ aims to combat stigma

APPOINTED POSITIONS

Student Regent Jun., 2016- Jun., 2017

Utah State Board of Regents

Initiatives, 2016-17:

Initiative Outcome
Increase communication between Utah Established weekly meeting with Utah
Student Association and Board of Regents. Student Association to communicate
initiatives that were then delivered to the
board.
Increase Awareness of Student Mental Established the Mental Health Working
Health Problems. group to consider possible policy
solutions.
 145

Media Coverage of Initiatives and Outcomes:

Utah State Board of Regent and Mental Health:

The Herald Journal: USU student appointed member of state higher education
board
USHE: Student mental health on college campuses becoming a significant policy
issue
USHE: Regents establish working group on student mental health

Student Conduct Board, Chairperson May, 2015- May, 2018

Vice President’s Office of Student Affairs
Utah State University

Director of Graduate Research May, 2014- Mar.,2015

Utah State University Student Association
Utah State University

Graduate Student Vice President, Student Council May 2013- May, 2014
Emma Eccles Jones College of Education
Utah State University

Graduate Student Council May, 2013- May, 2014

Department of Family Consumer and Human Development
Utah State University

__________________________________AWARDS_____________________________

Top 50 Most Influential on Campus, #25 Most Influential Apr. 2019

The Statesman
Utah State University

Top 50 Most Influential on Campus, #10 Most Influential Apr. 2017

The Statesman
Utah State University

Top 50 Most Influential on Campus, #12 Most Influential Feb. 2016

The Statesman
Utah State University

Description: The Statesman, the school newspaper, has open nominations for the most
influential person on campus. The Statesman’s editorial board then selects and rank-
orders 50 people from these nominations and any additional nominations deemed
appropriate from the board. The final group of the Top 50 Most Influential on Campus
 146

consists of students, staff, faculty, and upper administration and rank-orders who they
deem as the most influential people on campus

Bill E. Robbins Memorial Award Apr. 2017

Robbins Awards
Utah State University

Description: This award is an open nomination process for both undergraduate and
graduate students and is presented to one student a year at Utah State University. This
award is presented to the student who represents the best youth has to offer. This student
has excelled academically, displayed outstanding leadership ability, shown dedication to
Utah State, and possesses traits that set him or her apart as a rare individual. This award,
unlike other Robins Awards, is based on total collegiate achievement.

Man of the Year, Finalist Apr. 2017

Robins Awards
Utah State University

Man of the Year Apr. 2016

Robins Awards
Utah State University

Description: This award is an open nomination process for both undergraduate and
graduate students and is presented to one male student a year at Utah State University.
The recipient of the Man of the Year award will have made a significant impact at the
University during this year and has contributed to his and his classmates learning
experience.

USUSA Student Body Officer of the Year Apr. 2016

Utah State University Student Association
Utah State University

Description: This award is presented to one student body officer a year that has
demonstrated excellence in their elected term. The award is selected through private voting
from each student body officer in the Utah State University Student Association.

USUSA Academic Senate Outstanding Officer of the Year Apr. 2016

Utah State University Student Association
Utah State University

Description: The Chairperson of the Academic Senate selects one student body officer each
year as the Academic Senate Outstanding Officer that has demonstrated excellence in
collaboration and work-ethic in representing their constituency.
 147

Undergraduate Research Scholar May, 2012

Office of Research and Graduate Studies
Utah State University

_______________________________SCHOLARSHIPS_________________________

SA Gary Chambers Student Leadership Endowment

Leah D. Widstow Scholarship
USUSA Graduate Studies Senator Award
Leah D. Widstow Scholarship
USUSA Graduate Studies Senator Award
USUSA Director of Graduate Research Involvement Scholarship
Phyllis R. Snow Memorial Scholarship
Graduate Student Enhancement Award
Lawson Fellowship
Leah D. Widstoe Scholarship
Ferne Page West Scholarship
New Century Scholarship
Utah State University Merit Scholarship

________________________CONTINUING EDUCATION______________________
Focused Acceptance Commitment Therapy (6 hours) Apr. 2019
Anxiety Workshop CE’s (8 hours) Feb. 2019
Trauma-Focused Cognitive Behavioral Therapy (22 Hours) Jan. 2019
AAMFT Ethics Reading (2.5 Hours) Jun. 2018
Utah’s Crime Victim Conference (7.5 Hours) Apr. 2018
Cognitive Processing Therapy (10 Hours) Jan. 2018