31st Annual

Visitor Studies Association

Conference Abstracts
Joint Conference with the
Association of Midwest Museums

Fostering Transparency,
Strengthening Public Trust

Chicago, Illinois
July 19-21, 2018
 Visitor Studies Association - 2018 Conference Abstracts

Introduction
Welcome to the 2018 Visitor Studies Association Annual Conference Abstracts!

The Abstracts serve as a preview of the vibrant conversations that will take place
this year in Chicago as we explore transparency and public trust through visitors
studies. The Abstracts also serve an important role in recording the conversations
for the future. Previous Conference Abstracts are available online at
http://www.visitorstudies.org/past-conferences​.

The 2018 VSA Conference Abstracts were compiled by Michelle Lentzner.

1
 Visitor Studies Association - 2018 Conference Abstracts

Table of Contents
Introduction 1

Thursday, July 19 3

10:45-12:00 PM - Concurrent Sessions 3

Larger Than Life: Evaluating Museum-Wide Initiatives 3
Recognizing Youth and Educator Identities Around Computational Thinking 5
Study and Improve Visitor Experience Using Timing and Tracking 7
Museums, Evaluation, and the Application in Design Thinking 9
Look Up at the Stars: Integrating a User-Centered Design Process into a Natural History Museum 11
Relevancy, Transparency, Trust: SCIENCES Project for Community Engagement in Chicago 12
Becoming Affect Aware: Ethics, Empowerment, and Emotion Detection 15

1:45-3:00 PM - Roundtables, Papers, and Fireside Chats 17

Deborah Perry Fireside Chat 17
Exploring Methods for Measuring Visitor Motivation, Expectations, and Satisfaction 17
Blending Both Worlds: Walking the Evaluator-Practitioner Path 19
Constructivist Dialogue Mapping: Evaluating Learning During Play of Ant Adaptation, a Complex
Interactive Tabletop Museum Game 20
Doomsday at the Science Museum: Measuring Visitor Understanding in an Exhibit About Nuclear
Weapons & Climate Change 22
Formative Evaluation of Facilitated Play Engagement with Early-Childhood Learners 23
The F-Word: Failure and Failing Towards Success 25

Friday, July 20 27

10:15-11:30 AM - Concurrent Sessions 27

What's Next? Visitor Journeys at the Clyfford Still Museum 27
What Research Tells Us About Supporting Engagement in Science 28
Reconciling Public Trust with Museums, Non-Profits, and Evaluation 30
Walking the Talk: Visitors' Understanding of Zoos/Aquariums' Conservation Efforts 33
The Evolution of Evaluation to Support Community Engagement 35
Program Evaluation Practice and Capacity in the Canadian Art Museum Sector: Findings from a National
Study of Museum Practitioners 36
Eliciting Teen Voices and Stakeholder Values Using Most Significant Change 38

1:45-3:00 PM - Round Tables, Papers, and Fireside Chats 40

Judy Koke Fireside Chat 41
Say the Unspoken: Fostering Dialogue for Emerging Professionals 41
Collective Insight: Strategies for Culturally Responsive Evaluation Capacity Building 42
Talking About Teens: Evaluating Museum Programs that Target Teens 44
Big Questions About Food Security - The BigPicnic Project 46
Exploring Empathy: Lessons from Challenging Exhibitions 48

2
 Visitor Studies Association - 2018 Conference Abstracts

4:00-5:15 PM - Concurrent Sessions 50

Engaging Undergraduate Students in Audience Research 50
Digital Experiences in Art Museums: Visitor Preferences, Participation, and Impacts 52
Bringing Our Values to Work 55
Who's Not Here? Defining Audiences, Broadening Methods, Changing Relationships 57
Parents as Partners in Supporting Early Childhood STEM Learning 60
Understanding and Assessing Visitor Attention 63

Saturday, July 21 65

9:45-11:00 AM - Concurrent Sessions 65

Gathering Data to Inform Equity-Focused Organizational Change: Methodological Implications 65
Researching with a Field-Wide Perspective 67
What's the Benefit of Single-Visit Art Museum Programs to Students? 69
Outside In: Learning from Community Voices in Visitor Studies 72
Critical Appraisal: What's the Value? 74
The "Unsafe" Museum: Negotiating Difficult Heritage 77

11:15-12:30 PM - Concurrent Sessions 78

How Informal Learning and Science Communication Define and Assess Identity 78
Design-Based Research in Informal Education: Examples, Promise, and Pitfalls 79
Visitor Perceptions on the Well-Being of Animals in Captivity 82
Visitor Insights Driving Change and Practice 84

3
 Visitor Studies Association - 2018 Conference Abstracts

Thursday, July 19

10:45-12:00 PM - Concurrent Sessions

Larger Than Life: Evaluating Museum-Wide Initiatives

Gretchen Haupt, Science Museum of Minnesota
Alex Lussenhop, Museum of Science
Allison Cosbey, Conner Prairie

Purpose

This session is intended to share methods, findings, and insights from three differently scaled studies
tackling questions beyond measuring the impact of a single exhibit or program. Along with different
approaches and methods for evaluating museum-wide initiatives, attendees will have an opportunity to
explore how evaluation can help informal learning institutions define or reshape their identity based on
the perceptions of the institution by various stakeholders and consider how differing internal agendas
can impact these kinds of evaluations (and how evaluators can manage these differing agendas when
designing evaluations for these projects).

Abstract

Create.Connect

The development of the Create.Connect exhibition at Conner Prairie was an integral part of our initiative
to integrate STEM with the history-focused experiences that we are best known for. The project team
was particularly interested in exploring visitors’ perceptions of the inclusion of STEM- would they find it
confusing, or a poor fit with our mission or identity? Visitors overwhelmingly responded that the
experience fit, and while many noticed the historical aspects of Create.Connect, others said the
experience fit because it was interdisciplinary and focused on learning, broadly defined- both
descriptors they would apply to Conner Prairie as a whole. Others appreciated the inclusion of new
topics and acknowledged that Conner Prairie was changing to expand the types of learning offered.
Beyond providing support for this particular museum-wide initiative, the evaluation of Create.Connect
helped us see Conner Prairie through the guests’ eyes as a place for learning about a broader set of
topics, through the lens of history.

Gridiron Glory/CTE Content

In Fall 2017, the Museum of Science, Boston hosted the NFL Hall of Fame’s Gridiron Glory traveling
exhibit. Around that time, several news stories about chronic traumatic encephalopathy (C.T.E.) in NFL
players broke, and MOS decided to create a small suite of programming about CTE to run concurrently
with the exhibit. The Research and Evaluation Department at MOS conducted a study on awareness and
perception of the content, and found that while there was high awareness among visitors of CTE,
concussions, and their connection to the NFL, few visitors were looking for CTE or concussion
information during their visits. Visitor exit surveys also showed there was low to moderate overall
awareness of MOS’s concussion/CTE content. These and other study findings raise more questions: who

4
 Visitor Studies Association - 2018 Conference Abstracts

was the audience for the CTE content? What do visitors see as the role of MOS in addressing
current/controversial topics like CTE and concussions? Does that perception differ from how MOS sees
its own role? Even if visitors were aware of individual content offerings, did they understand that MOS
was making a coordinated effort to address CTE?

Year of the Engineer and Illumination

Since 2016, the Science Museum of Minnesota (SMM) has shifted away from mostly one-off project and
exhibit evaluations to also include more integrated approaches considering questions from the visitor
perspective. This is accomplished by large complex studies like our ongoing exit study utilizing the
COVES protocol, but it also involves structuring smaller efforts traditionally associated with programs
and exhibits to connect findings across and beyond the Museum to answer three overarching questions:
“How can we understand and improve visitor experience?”; “How can we understand and improve the
ways visitors relate to SMM?”; and “How can we deepen relationships with communities we have
underinvested in?” SMM strives to answer these questions part by asking multiple audiences how their
expectations about the experience influenced their decision to visit, and how their actual experience
compared to their expectations by developing a consistent set of visitor experience and relationships
questions to apply across the Museum.

Importance

Museum-wide initiatives are not new, but thinking about them holistically is often overlooked. As
museums strive to broaden audiences beyond their core visiting demographics while continuing to
engage those with an existing relationship, evaluation that considers the organization’s intentions as a
whole by reaching beyond the scope of a single exhibit or program and building connections between
efforts is vital to understanding the agendas, expectations, and experiences of our visitors regardless of
it is their first or fiftieth visit. Coordinating and enacting these types of studies can seem overwhelming,
so beyond insights from the findings of the three studies described in this panel, importance also lies in
learning about the processes the presenters used to address the challenges --both methodological and
theoretical) at their own institutions.

Additional Links

About: What is COVES?. (n.d.). Retrieved from ​http://www.understandingvisitors.org/about/

Create.Connect. (n.d.). Retrieved from ​http://www.createconnect.org/

Illumination. (2017). Retrieved from ​https://vimeo.com/247379405

Year of the Engineer. (n.d.). Retrieved from ​https://www.smm.org/engineer

Recognizing Youth and Educator Identities Around Computational Thinking

Leigh Ann Mesiti, Coordinator of Research and Evaluation, Museum of Science, Boston
Alana Parkes, Supervisor of Exhibit Content Development, Museum of Science, Boston
Denise Huynh, Research and Evaluation Associate, Science Museum of Minnesota

5
 Visitor Studies Association - 2018 Conference Abstracts

Purpose

The purpose of this session is to look at how informal and formal settings can affect youth and educator
perceptions and ways of relating to computational thinking. It will also highlight challenges in conducting
this research, such as defining computational thinking from different perspectives and constraints
around collecting participant data. By sharing strategies used in an exhibit context, as well as a
programmatic context, participants will learn how different types of museum offerings can affect
students’ and educators’ understanding and relationship to computational thinking. As part of this
session, participants will be able to engage with sample activities and exhibits from these two programs
and hear recent findings from the field, as well as think about ways to incorporate a variety of strategies
into their own work.

Abstract

This session will look at how youth and educators learn about and identify around computational
thinking practices. Professionals will discuss findings from NSF-funded research, outlined below, around
computational thinking capacity and engagement in informal and formal learning environments. Each
institution will share how it defined computational thinking, its primary audiences, research context,
methods, and findings.

Building Computational Thinkers through Informal Exhibit Experiences is research project at the
Museum of Science that explored how educators and designers can effectively support the development
of computational thinking capacity and how these learning experiences can be customized to meet the
needs of learners from diverse backgrounds. This study focused on design strategies used in exhibits
that were part of The Science Behind Pixar travelling exhibition about STEM content and practices
behind Pixar’s innovative films. Project research was conducted in two phases, where the first phase
explored how students in grades 6-12 and experienced computer programmers/scientists engaged in
different types of computational thinking-focused exhibits. The second phase focused on how museums
can afford diverse, novice learners the opportunity to develop computational thinking capacity and
become initiated into contextualized use of computational thinking. This research helped researchers
and designers understand the impacts of exhibit design strategies for developing computational thinking
capacity in relation to learner background and identity. Findings suggested that these exhibits
embedded supports that made it easy for groups to identify with the work and people required to
create believable scenes in Pixar movies. By using symbols and language, relaying authentic challenges,
and revealing underlying mechanisms of the program, novices and experts connected to how Pixar
professionals approach a problem and work toward a solution.

LinCT is an intergenerational, all-female community structured around near-peer mentorship aimed at

building confidence around computational thinking. Through shared experience, collaborative learning,
and distributed leadership high school youth and pre-service teachers gain technology skills, practice
growth mindset, and develop and deliver tech-based camps and classes in formal and informal settings.
This project at the Science Museum of Minnesota (in collaboration with several other institutions)
explored how a program can be built around computational thinking practices and discovered how
important community and relevance is to the program model. Research for this project investigated
aspects of educator learning, such as how different forms of participation and engagement impact
teachers’ ability and willingness to incorporate technology use in their classroom, and how teachers’
motivation, confidence, and self-¬efficacy in technology integration can change as a result of their role

6
 Visitor Studies Association - 2018 Conference Abstracts

and experience in LinCT. It also investigated programmatic effects on youth, as to how the project model
effectuates change within the youth interns’ motivation, confidence, self¬-efficacy, and technology
knowledge over the duration of the program, as well as how summer camp and classroom students
grew in their computational thinking. By documenting how participants come to define computational
thinking over time and measuring perceptions and confidence related to this work, the museum gained
a better understanding of how to better support youth and educators to engage with technology and
computational thinking in a meaningful way.

Importance

Computational thinking is a hot topic in education across the nation, particularly in the museum field.
How can we support students to systematically engage in the process of solving complex problems? The
construct of computational thinking has been characterized as the “thought processes involved in
formulating problems and their solutions so that the solutions are represented in a form that can be
effectively carried out by an information-processing agent” (Wing, 2011). Informal learning
environments, such as museums, can support diverse learners by providing opportunities to develop
computational thinking capacity and provide an introduction to a contextualized use of computational
thinking that complements the work of formal, K-12 settings. In doing this work, museums are starting
to investigate how different types of learners understand and relate to computational thinking, as well
as how they perceive these practices.

References

Cahill, C., Mesiti, L.A., Paneto, S., Pfeifle, S., Todd, K. (2018). The Science Behind Pixar summative
evaluation report. Retrieved from:
http://informalscience.org/science-behind-pixar-summative-evaluation-report

Additional Links

http://informalscience.org/science-behind-pixar-summative-evaluation-report

http://sciencebehindpixar.org/

http://wesharescience.com/pin/8531

Study and Improve Visitor Experience Using Timing and Tracking

Beverly Serrell, Director, Serrell & Associates, serrellassociates.com
Karen Wise, Principal, Wise Strategic Advising, wisestrategic.com

Purpose

Big Idea

When staff understand patterns of visitor behavior and movement at their organizations they are able
to create and continually improve great visitor experiences. One of the easiest and most efficient ways
to gather actionable information is through timing and tracking, which can be done by staff and even
volunteers at almost any size organization.

7
 Visitor Studies Association - 2018 Conference Abstracts

Tracking and timing (T&T) data provide valuable information for exhibition planning and evaluation, and
benchmarking success (in-house or field-wide), as well as for general operations from amenities to
exhibits to staff hours. Most aspects of the visitor experience are influenced by audience movements
and stay time, and understanding these patterns can provide important metrics as well as supporting
improvements in the visitor experience and facility operations.

Abstract

Background

Tracking and timing (T&T) is one of the oldest research methods for learning about visitor behavior in
exhibitions and throughout a museum. Although T&T has gained importance within exhibitions, it is also
useful for understanding patterns of visitor behavior across an entire facility or building. T&T is a
relatively simple method that can be used by virtually anyone. It can also be used to make comparisons
both within an organization and across organizations. Practitioners will benefit from having more
rigorous, standardized, and detailed methods of presenting T&T data that allows for making such
comparisons.

Activities and Issues to be Discussed

After a brief presentation to define terms, attendees will work in small groups to analyze real timing and
tracking data sets from different exhibits in different kinds of museums, and generate their own
hypotheses, questions and sets of potential next steps. Each group will
derive average time spent, the % of stops made, and review visual expressions of the distribution of time
data (histograms) and visualizations of the distribution of time and stops by the whole sample
(scattergrams), and see examples of the attraction rates of exhibit elements within an exhibition (heat
maps).

The entire group will then have a facilitated discussion exploring patterns observed in the data, and
discussing the significance of their findings and the implications of patterns seen for running a museum.
Attendees will leave with the capacity to conduct T&T, and with free access to electronic and paper
versions of forms to collect and analyze data, as well as with a means to share and compare findings,
particularly the metrics of sweep rates and the percent of diligent visitors in exhibitions.

Issues likely to be encountered and discussed include:

o What can the shape of a histogram of time spent tell you?
o Which shapes or trends are common or rare?
o Is there such a thing as "the average visitor"?
o How big does the sample have to be?
o What can the distribution of data points on a scattergram tell you?
o What is an "outlier"? Are they common?
o What does the "percent of diligent visitors" mean?
o What does "sweep rate" mean? Are these metrics meaningful or useful?
o How do you make a "heat map" and what does it show?
o What is considered a high attraction rate for an element?
o Do high attraction rates for an element tell you about the audience or the time spent?
o What is considered a low attraction rate? If just one visitor stopped does that make it good?

8
 Visitor Studies Association - 2018 Conference Abstracts

Importance

There is much discussion about having shared methods of conducting visitor studies so that the data can
be shared and compared across time and institution types and sizes, yet there are few examples of
methods capable of producing field-wide data trends. Tracking and timing, while widely known and
often referenced, needs a more rigorous set of shared definitions and better methods of displaying the
data to make it a more useful tool for improving exhibitions, visitor experiences, and museum
operations for all types of museums. This session will provide free access and resources to methods and
forms that can be used to create data which are not only useful to individual institutions or projects, but
which will be comparable to data collected elsewhere, including a number of reports that are already
posted on ​informalscience.org​.

References

Bitgood, Stephen. 2014. "Exhibition Design that Provides High Value and Engages Visitor Attention."
Exhibitionist 33 (1): 6–14.

Bitgood, Stephen. 2014. Engaging the Visitor, on-line resource,

http://museumsetc.com/products/engaging

Borun, Minda, et al. 1998. Family Learning in Museums: Family Learning in Museums: The PISEC
Perspective, Association of Science-Technology Centers, Washington DC.

Detroit Institute of Arts. 2013. Tracking and Timing Report. In-house document.

Serrell, Beverly. 2015. Exhibit Labels: An Interpretive Approach, Second Edition. Rowman & Littlefield.

Serrell, Beverly. 2010. "Paying More Attention to Paying Attention."

http://www.informalscience.org/news-views/paying-more-attention-paying-attention

Serrell, Beverly. 1998. Paying Attention: Visitors and Museum Exhibitions. AAM.

Yalowitz, Steven and Kerry Bronnenkant. 2009. "Tracking and Timing: Unlocking Visitor Behavior." Visitor
Studies 12(1): 47–64.

Additional Links

Many of the reports whose data have been referenced can be found at informalscience.org, such as:
(order of information = Exhibition Name, Museum Name, Report Author)

Expedition Health, Denver Museum of Nature & Science, Patty McNamara

Secrets of Circles, San Jose Children's Museum, Sue Allen
Weaving a Revolution, Utah Museum of Natural History, Beverly Serrell
Denaina Way of Living, Anchorage Museum, Beverly Serrell
Search for Life, New York Hall of Science, Randi Korn
Amazing Feats of Aging, Oregon Museum Science & Industry, Patty McNamara
Star Wars, Museum of Science Boston, Carrey Tisdal

9
 Visitor Studies Association - 2018 Conference Abstracts

Rubens Gallery, Detroit Institute of Arts, Matt Sikora

Silent Witness, Shedd Aquarium, Beverly Serrell

Museums, Evaluation, and the Application in Design Thinking

Scott Burg

Purpose

This paper will provide an overview of the evaluation of three museums programs applying design
thinking in different contexts for diverse audiences. Attendees will learn how museums are using
human-centered design concepts to develop and test internal organizational capacities, improve
interdepartmental collaboration, create programs and experiences to better meet audience needs,
enhance outreach with local constituents, and serve as a resource for formal and informal educators
across the country. Attendees will also become familiar with methods for assessing design thinking
programs in museums, and with museum constituents.

Abstract

This paper will provide a summary of findings and methodologies for research and evaluation projects
with three museums, the Cooper Hewitt Design Museum, the New York Hall of Science and the
Computer History Museum, each adopting design thinking strategies for use with organizational
planning, or for training educators in their community, and elsewhere across the country.
The Computer History Museum collaborated with IDEO, a leading design, and innovation firm, to
conduct design workshops with CHM staff and community stakeholders to research needs and generate
creative concepts for CHM’s new Education Center. CHM has also adopted design thinking for use in
organizational planning, exhibitions, and education programs. Our evaluation documented the design
thinking processes and its impact on CHM staff and design of the Center.

The Cooper Hewitt Museum’s Design in the Classroom (DITC) workshop provided 30 K-12 educators
from across the country methods for integrating design thinking in the classroom. The museum wanted
to learn if the workshop met teachers’ needs, how teachers transferred skills and ideas from the
workshop into practice, and what effects the experience had on teachers and students. A mixed
methods evaluation was conducted before and during the workshop, and at participating schools to
observe and assess teachers’ design activities.

The New York Hall of Science's Design Lab, a multi-faceted STEM education initiative resulted in an
exhibition learning space, field trip activities, in-class curriculum projects, and teacher professional
development workshop all incorporating the use of design-based learning. A mixed-methods evaluation
assessed the application and alignment of design-based learning for museum staff, teachers, and
students across the project’s 4 interconnected activities. A 3-year retrospective study was conducted to
assess long-term sustainable impacts of design thinking methods with program alums.

Findings from these three projects provided researchers with a deeper understanding and appreciation
of the growing importance and impact of design thinking for museum professionals and their
professional education audiences. We observed that with design thinking that there is no single,
definitive way to move through the design thinking process.

10
 Visitor Studies Association - 2018 Conference Abstracts

Design thinking is messy. There are many structured and unstructured elements to consider when
assessing the effectiveness of the process itself and the impact of any resulting implementation.
Exploring the application of design thinking in three different museum settings and contexts provided an
opportunity to employ a number of different yet complementary evaluation methods, including
structured observations, participant reflections, assessment of program artifacts, peer evaluation, and
analysis of video footage.

Researchers also applied the design thinking method itself to define program outcomes and as a
framework to plan the evaluation across each of the three projects. For each project researchers
assessed four primary dimensions. These included:

● Changes in hard measurable outcomes

● Measurable changes in perception
● Changes in conversation and participant interaction
● Changes in how people think

Importance

Museums are taking on an increasingly important role in the training and dissemination of design
thinking for their own staff as well as with educators in both formal and informal education
environments. These workshops and outreach activities provide museum staff and educators with
opportunities to explore design thinking, to integrate the design process into their practice, or their
curriculum, and to transform the teaching and learning processes in classrooms. In our work with these
three museums, we have witnessed how design thinking serves as a catalyst for team building and
collaboration amongst peers, promotes risk-taking and incremental innovation, encourages internal and
external partnerships, and enhances the museum's standing with diverse local and national
communities.

This research provides insight on how design thinking can provide museums new methods to invigorate
their own internal planning, while also reaching new audiences to become more efficient and responsive
21st-century institutions.

Look Up at the Stars: Integrating a User-Centered Design Process into a Natural

History Museum
Barry Joseph, American Museum of Natural History

Purpose

The American Museum of Natural History (AMNH) is developing a deeper understanding about the
visitors in our permanent Halls and exploring what happens when the power of emerging media is
leveraged to connect the two. This multi-year investigation has been driven by a range of overarching
questions. The study being presented looks at a series of prototypes that surfaced information
pertaining to the following questions:
- Can we turn AMNH scientist’s digital data into Hall-based interactives?
- How can we create a social experience around digital specimens?

11
 Visitor Studies Association - 2018 Conference Abstracts

Abstract

“Remember to look up at the stars

and not down at your feet.”
― Stephen Hawking

To be better prepared for the key challenges presented by our new century, the American Museum of
Natural History developed an internal skunk works. It focused on tackling how to design interactives to
engage visitors with the digital specimens produced and studied by our more than 200 scientists. Using
VR, AR and other forms of emerging media, we developed prototypes and tested them with over 1,000
visitors to identify challenges and opportunities for the Museum visit of the future; virtual weevils and
sharks cavorted with augmented constellations and moon-scapes.

This presentation will start with an introduction to the Museum and the challenges it faces. It will then
introduce the department developing public-facing prototypes, its mandate to bring the digital
specimens studied by museum scientists to the public, and the overarching questions that informed its
work.

The presentation will then turn to the user-centric or human-centric design practices that informed this
prototype development process, the related evaluation techniques that were deployed, and how we are
learning by studying how visitors in our halls interact with and respond to these new experiences. It will
add to our understanding on techniques of iterative design, prototyping, and design sprints.

A use case will show what this looks like up close, focusing on a series of prototypes that turned the
most complete and accurate 3-D atlas of the universe (Digital Universe) into an interactive that
highlights how stars in constellations sit in a 3-D space. The different demographics reached, the
observational and interview evaluation methodologies deployed, and lessons learned about visitor
engagement with emerging media will be shared. The iterations included the following:

1. Google Tango was offered to allow visitors to “look around” their constellation.
2. A laptop was used that invited visitors to “map” their face into a constellation.
3. Microsoft’s Hololens immersed visitors in a starfield they could explore with their body.
4. The Hololens experience was integrated into a gamified, social context (an escape room).
5. A design sprint was held to rapidly advance the development process.
6. A Microsoft Kinect was attached to a large video screen that invited visitors to position their body
over various constellations.

The use case will highlight how this design process informed the final visitor experience.

Finally, the presentation will share the type of findings which emerged across the first year of these
projects.

Importance

This presentation advances our understanding of how design practices can inform the work of museum
visitor research. This session highlights a variety of approaches that can be adapted by a wide range of
museums.

12
 Visitor Studies Association - 2018 Conference Abstracts

Relevancy, Transparency, Trust: SCIENCES Project for Community Engagement in

Chicago
Lisa-Anne DeGregoria Kelly, Chicago Zoological Society
Shelly Hope, Chicago Zoological Society
Kate Livingston, ExposeYourMuseum, LLC

Purpose

“Supporting a Community's Informal Education Needs: Confidence and Empowerment in STEM,” or

“SCIENCES,” is a project that engages a community on the southside of Chicago in science learning
through a partnership of Chicago Zoological Society, which operates Brookfield Zoo, and a community
based nature center, Eden Place. The project provides an opportunity to consider both what relevant
community engagement means for the community and what it means for the partnering organizations.

This session offers a variety of SCIENCES project perspectives from the project’s Community Relations
Coordinator and co-Principal Investigator, who will address what it means to work within a community
with the focus on that community, the need to be transparent and relevant, and how mutual trust
developed over time; and the external evaluator, who along with the co-PI will address collaboration
with the lead organization’s internal evaluator, and how evaluation strategies play a role in transparency
and trust.

Abstract

Informal learning organizations increasingly recognize a responsibility to build relationships, particularly

in traditionally underserved communities. Expanded participation in science and science learning is
essential in addressing increasingly complex societal needs. Towards this goal of supporting lifelong
learning for a diversity of audiences, the SCIENCES Project was created (Supporting a Community's
Informal Education Needs: Confidence and Empowerment in STEM).

This NSF funded project partners Chicago Zoological Society and Eden Place, a community based nature
center in Chicago’s southside Fuller Park, which has long been one of Chicago’s most under-resourced
African American neighborhoods.

SCIENCES project partners created a “STEM learning ecosystem,” reaching a variety of audiences
through multiple learning touch points in the urban community surrounding Eden Place. This five-year
project (2013-2018) drew upon Brookfield Zoo and Eden Place program portfolios to increase
engagement of this community in science and conservation. A holistic model included education
programming for professional, school, and public audiences from young children to senior citizens.

ExposeYourMuseum, LLC conducts SCIENCES project front-end, formative, summative, and process
evaluation. Some SCIENCES project education program types had been previously offered in different
settings, and some had evaluation instruments designed and implemented by the zoo’s internal
evaluation team. Such instruments were drawn upon by the external evaluator, and required adaptation
to the SCIENCES context. In addition, new tools were designed and evaluation methods required
adaptation as programs evolved. The external evaluator further fostered relevancy by recruiting and

13
 Visitor Studies Association - 2018 Conference Abstracts

building capacity of a team of local, independent data collection contractors who dubbed themselves
“Community Ambassadors” – a strategy that had challenges and successes of its own.

Our VSA session represents one way we engage in dialogue about supporting learning in underserved
communities. We will discuss: What does it look like when informal science learning institutions work to
broaden participation and maintain relevancy with diverse audiences? How do we foster transparency
and strengthen public trust in a community-focused learning ecosystem? What roles do informal
learning and evaluation professionals working in partnership have in relevancy, transparency, and
mutual trust in community engagement?

The session will share perspectives and discussion points applicable to initiatives focused on the roles of
education programming and evaluation in community engagement endeavors. These perspectives
include:

● Developing an effective relationship between the external evaluator and the informal learning
organizations leading the project is an important element of transparency and trust, allowing
lessons learned to be put into action. Collaboration among PIs, program staff, and evaluators is
essential to ongoing program development and strategic decision-making. Such collaboration is
important for gaining community access for front-end evaluation, adapting evaluation needs as
projects evolve, finding relevant ways to share and discuss formative evaluation results, and
situating summative results within the context of a multi-year, complex project.

● Developing relevant, trusted community engagement takes transparency, time, patience,

persistence, and a consistent presence. Rethinking definitions of “community,” recognizing
community assets, and users of those assets by both neighborhood residents and others are
important considerations for community engagement and relevance within a
community-focused model in support of lifelong learning.

Importance

SCIENCES strives to increase community engagement. We will discuss lessons learned about the roles of
education programs and evaluation in expanding participation in our conservation missions, maintaining
relevancy with diverse audiences, and fostering transparency and community trust. Intended outcomes
for the VSA audience include:

● Increased awareness of an informal learning model that deepens community engagement in

science and conservation through a zoo-community partnership with education programs that
have multiple touch points across ages and audiences.

● Improved understanding of the importance of time, persistence, and patience; and

understanding varying definitions of community and community assets to relevancy,
transparency, and trust.

● Strengthened recognition of the role external evaluation can play in fostering and understanding
relevancy, transparency, and trust in a complex, community-focused project, including strategies
of adapting methods as programs evolve, collaborating with internal evaluators, and recruiting a
team of local “Community Ambassadors”—independent evaluation data collectors.

14
 Visitor Studies Association - 2018 Conference Abstracts

References

Mogerman, J.-E., Breen Bartecki, S., Kelly, L.-A.D., & Howard, M. (2017, April 18). SCIENCES: Supporting a
Community’s Informal Education Needs: Confidence and Empowerment in STEM. Retrieved
from
http://www.informalscience.org/news-views/sciences-supporting-community%E2%80%99s-info
rmal-education-needs-confidence-and-empowerment-stem

Additional Links

https://www.czs.org/Chicago-Zoological-Society/Community-Impact/Eden-Place

Becoming Affect Aware: Ethics, Empowerment, and Emotion Detection

Sarah May, Museum of Science, Boston
Sunewan Paneto, Museum of Science, Boston
Gabrielle Schlichtmann, EdTogether
Kevin Kent, CAST

Purpose

Emotion is a critical mediator of engagement, yet we rarely attend to it explicitly. Affective technologies
address this by attempting to detect emotion – and not without controversy. Researchers from the
Museum of Science, Boston, EdTogether, and CAST discuss issues of trust and transparency percolating
through three studies that employ affective technologies and the ethics surrounding leveraging
emotions on a broader level. This team shares insights related to affective technologies, while reflecting
on nuanced issues related to researching and designing for emotion experiences in a museum. What are
affordances and constraints of affective technologies? What roles should stakeholders play in making
meaning of emotion? What is the line between emotional design and manipulation? Through dialogue
related to these questions, descriptions of recent studies, and reflection on how emotion work could be
applied in other settings, this session highlights the complex nature of emotion in informal education.

Abstract

Background

The field of affective computing is exploding with new technologies that measure emotional
information. These technologies create opportunities for museum professionals to explore the nature of
emotional engagement, with implications that may support more effective design and research
practices. However, use of these technologies raises questions of empowerment, privacy, autonomy,
and trust (Cowie, 2015).

Since 2013, researchers from the Museum of Science, Boston (MOS), CAST, and EdTogether have
engaged in collaboration exploring how affective technologies can be ethically and effectively leveraged
in emotional design for informal learning. Three case studies on research investigating emotional
engagement illustrate how technologies designed to create emotional awareness for visitors, designers,
and researchers can be leveraged, with a focus on ethical considerations and implications for

15
 Visitor Studies Association - 2018 Conference Abstracts

transparency. Through this work, our research team has itself become more “affect aware.” Lessons
learned provide insight for museum professionals seeking to integrate methods of emotion
measurement and design into their work.

Case Study 1: Research on Emotional Engagement

The MOS and CAST collaborated on an NSF-funded project (Emotion and Thinking in Designed Informal
Science Environments, DRL #1222613) to understand emotional engagement during informal learning.
Participants engaged in an exhibition while providing data on their emotional experiences through
self-reports, video, eye-tracking glasses, and skin conductance wristbands. Public audiences were invited
to learn about this research and experience the emotion monitoring technologies through an educator
interpretation activity. Data collected helped identify emotional states not typically addressed in
informal learning contexts but are critical for learning. This work illustrated the need to engage in more
transparent conversations about emotional engagement with stakeholders.

Case Study 2: Designing for Emotional Engagement

With funding from NSF (Designing for Productive Struggle, DRL #1612577), we are conducting research
to develop a framework for supporting “productive struggle” during informal science learning. Emotion
measurement technologies are used during the research process to inform prototype development, and
we are taking a design-based research approach where stakeholders engage as co-designers in the
prototype development process, and consumers of the final exhibit components. We are exploring ways
to level power dynamics by fostering empathy between visitors, designers, and researchers to inform
development and address issues of emotional engagement in design.

Case Study 3: Empowering Learners through Emotional Awareness

The Empowering Learners through Effective Emotional Engagement project funded by the Argosy
Foundation seeks to enable museum professionals and visitors to leverage emotion information through
affective systems, while remaining critically responsive to issues of trust, transparency, and
empowerment. Front-end studies with visitors and staff offered insights into public trust in the museum
as well as general distrust of affect detection technologies. Insights informed the team’s approach to
technology integration that will support end-user autonomy and trust.

Altogether, we have grappled with ethical issues emergent in relevant literature (see Daily et al., 2013;
Heger et al., 2016; Hook et al., 2008) around emotion induction, visitor autonomy in emotion
construction, and power dynamics between researchers and visitors, while reflecting on practices that
give stakeholders the ability to experience empowerment and autonomy in learning experiences.

Importance

As trusted, public informal education institutions we have a stake in shaping conversations around the
use of ambient data collection technologies for public good, especially as these technologies might be
leveraged in support of research into visitor experience and exhibit design more broadly. In this domain,
emergent technologies for emotion measurement are some of the most promising and ethically
concerning. Several high level questions emerge from this work that require exploration and reflection.
How are emotional outcomes different or similar to cognitive outcomes? Does measuring emotion
“feel” more personal and intrusive? Why might measuring emotional outcomes feel more invasive than

16
 Visitor Studies Association - 2018 Conference Abstracts

other kinds of outcome measurement, especially when technology comes into play? How do we balance
rigorous research and evaluation designs with the ethical, transparent inclusion of stakeholders along
the way? What is the role of user empowerment and autonomy when thinking about integrating
affective technologies into informal learning experiences?

References

Cowie, R. (2015). Ethical issues in affective computing. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas
(Eds.), The Oxford Handbook of Affective Computing (334-348).New York, NY: Oxford Press.

Daily, S. B., Meyers, D., Darnell, S., Roy, T., & James, M. T. (2013, July). Understanding privacy and trust
issues in a classroom affective computing system deployment. In International Conference on
Distributed, Ambient, and Pervasive Interactions (pp. 414-423). Springer, Berlin, Heidelberg.

Heger, O., Kampling, H., & Niehaves, B. (2016). Towards a theory of trust-based acceptance of affective
technology. Association for Information Systems, 6(15).

Hook, K., Stahl, A., Sundstrom, P., & Laaksolahti, J. (2008). Interactional empowerment. CHI 2008
Proceedings. April 5-10, 2008.

Rappolt-Schlichtmann, G., Evans, M., Reich, C., & Cahill, C. (2017). Core engagement in informal science
learning. Exhibition, 36(1). 42-51. Retrieved from
https://www.name-aam.org/s/10_Exhibition_CoreEmotion.pdf

1:45-3:00 PM - Roundtables, Papers, and Fireside Chats

Deborah Perry Fireside Chat
Dr. Perry has conducted research on the role of intrinsic motivation in informal educational settings
focusing on the question, "What makes learning fun?" In addition to her accomplishments in museum
exhibit and program evaluation, she is the founder and director of Chicago-based Selinda Research
Associates, as well as a former VSA President and board member. Attend this session to participate in a
discussion about the role of visitor studies in supporting visitor-centered learning and translating
research to practice.

Exploring Methods for Measuring Visitor Motivation, Expectations, and

Satisfaction
Sheri Levinsky-Raskin, AVP of Research and Evaluation, Intrepid Sea, Air & Space Museum
Amanda Thompson Rundahl, Director of Learning and Engagement, Saint Louis Art Museum
Kanani Hoopai, Curator of Education, Mission Inn Foundation
Mariruth Leftwich, PhD, Director of Education, Senator John Heinz History Center

Purpose

17
 Visitor Studies Association - 2018 Conference Abstracts

This roundtable discussion explores a range of methodologies for collecting and documenting museum
visitor perceptions, motivations, expectations and levels of satisfaction. The big idea for the discussion
will present opportunities for session attendees to:

- Explore why museums / cultural institutions measure visitor motivations, expectations and levels of
satisfaction and how this information informs our work;
- Identify varied collection methodologies appropriate for institutional or programmatic goals and
intentions, and institutional environments; and
- Engage in discourse with colleagues about the opportunities and challenges associated with examples
presented by session presenters and audience members, and how collected information can be used
and inform decision-making.

An action planning handout will provide a physical take-away for attendees to reflect on current
evaluation practices at their institutions, why they need the data and how it can be used, as well as
consider new approaches.

Abstract

This roundtable discussion will investigate a range of methods to measure why visitors come to our
institutions, what they expect, and if they leave satisfied with their experience. Seeking to empower
professionals at diverse cultural institutions to be creative in the evaluation practices they employ, this
session will engage attendees in activities they can replicate with their visitors and in discussions about
best practices for institutional buy-in and cross-departmental collaboration.

The session builds on the history of visitor studies in museums through reflection of varied examples
from presenting institutions. Directions the conversation will take during the session include reviewing
the data collected, why the data was collected and why the approach was selected, how and why
approaches evolved during the course of studies, surprises revealed, and applications of the data.

Recognizing the essential need for research and evaluation across an institution, whether led internally
by museum professionals or through contracted researchers, discussions will reference the spectrum of
museum work that this data informs - from exhibition development, programming, marketing, and
development. Other key topics considered through presenter comments and attendee discussions will
be ethics, comparative data analysis, results and outcomes.

Importance

Linking to the conference theme, the importance of this session discussion addresses institutional
responsiveness to communities and visitors, realizing an institution's value to its communities, reflecting
visitor interests and needs through our presentation of content and experiences / programs, and making
decisions that fosters empathy and understanding about the audiences we serve and seek to serve.

Implications for the field include the continually changing world in which we live, how it influences our
visitors, and in turn how we remain informed about their expectations, motivations and satisfaction,
while being agile, responsive, and relevant.

References

18
 Visitor Studies Association - 2018 Conference Abstracts

In addition to those listed below, other resources will be shared on the session handout.

Adams, M., Falk, J. and Dierking, L. (2003). ‘Things Change: Museums, Learning, and Research’. In M.
Xanthoudaki, L. Tickle and V. Sekules (eds), Researching Visual Arts Education in Museum and
Galleries: An International Reader. Dordrecht: Kluwer Academic Publishers.

Journal of Museum Education (JME). Empowering Museum Educators to Evaluate, 40:1, March 2015.

Pekarik, A. J., Schreiber, J. B. and Visscher, N. (2018). Overall Experience Rating – Measuring Visitor
Response in Museums. Curator. Doi:10.111/cura.12256

Additional Links

American Evaluation Association (AEA) - ​https://www.eval.org/

Committee on Audience Research and Evaluation (CARE) -

https://www.aam-us.org/professional-networks/committee-on-audience-research-and-evaluation-care/

“Evaluation Toolkit for Museum Practitioners.” (2008). -

http://visitors.org.uk/wp-content/uploads/2014/08/ShareSE_Evaltoolkit.pdf

Institute of Museum and Library Services (IMLS) Evaluation Resources -

https://www.imls.gov/research-evaluation/evaluation-resources

Blending Both Worlds: Walking the Evaluator-Practitioner Path

Patience Baach, Audience Insights and Research Manager, The Field Museum
Elizabeth Bolander, Director of Audience Insights and Services, Cleveland Museum of Art
Marcus Harshaw, Director of Education and Community Engagement, Indiana Historical Society

Purpose

This roundtable discussion will focus on the opportunities and challenges of being an
evaluator-practitioner. The goal of this session is to offer an open forum for professionals who must take
on dual roles of being someone who must conduct research and evaluation activities as well as someone
who utilizes these findings to guide their work. As more museum professionals must take on evaluation
activities as a part of an existing role, this timely discussion, led by three presenters trained in evaluation
who are taking on more expanded practitioner roles, is expected to offer attendees a chance to learn
new strategies for managing their time and efforts and ways to communicate the challenges of being an
evaluator-practitioner to supervisors.

Abstract

This roundtable discussion will investigate the place where evaluator and practitioner worlds meet. The
three leaders of this roundtable discussion will discuss how their own roles are evolving to encompass
more than just typical “evaluator” functions. Elizabeth Bolander and Patience Baach will discuss how
their roles have evolved to incorporate more visitor experience elements. Marcus Harshaw will bring
insights from his role in education.

19
 Visitor Studies Association - 2018 Conference Abstracts

Panelists and participants will discuss the opportunities and challenges associated with these role
enhancements. Some examples of this include maintaining independence and being free from bias when
analyzing data; managing peer relationships; ways of communicating this divide to supervisors; and
carving out time for more in-depth studies when facing practitioner role demands. Specific discussion
prompts include “What are some of the best things about being an evaluator-practitioner?,” “What are
some of the challenges of being an evaluator-practitioner?,” “How do you find balance between the
evaluation and practitioner functions with which you are charged?,” and “How to do you maintain your
independence as an evaluator when you also have a practitioner role?”. Attendees will engage in active
dialogue about strategies and tactics for managing these blended roles and being a visitor advocate
under varied circumstances.

Importance

As more museum professionals come to evaluation through dual evaluator-practitioner roles, it is

important to have conversations about the positive and challenging elements of these positions. This
discussion will foster this community and provide ideas and insights that participants can use to be
better evaluators and visitor advocates.

Constructivist Dialogue Mapping: Evaluating Learning During Play of Ant

Adaptation, a Complex Interactive Tabletop Museum Game
Kit Martin

Purpose

One can only learn so much in a short interaction with a game in a museum. This session will detail how
to create a constructivist dialogue map to show this learning in these contexts. Constructivist dialogue
mapping is a evaluation methodology, I developed that is specifically designed for the short, informal
interactions typical with digital interactives. At the heart of this work is the notion of museum visitors as
active theorists. They develop their understanding in interaction with the context during their visit. I
develop constructionist dialogue maps, a type of concept maps (Miles, Huberman & Saldana, 2014; Chi
and Koeseke, 1983), as they emerge through transcript data. These maps represent what players say. As
shown in figure 1, at this level, I captured ideas about the understanding of science players demonstrate
during play in a constructivist dialogue map.goal of your session.

Abstract

I developed constructivist dialogue mapping, a way to study learning in an informal environment based
in on constructivist theory. In constructivist theory, a learner’s mental model drives his or her
construction of an understanding. This process includes accommodation and assimilation (Piaget, 1952)
and maintaining a balance between stability and change, continuity and diversity, and closure and
openness when exploring the world (Ackerman, 2001). For Piaget, children are not just incomplete
adults. Their ideas function very well for their current context and as a result, their mind changes
through experience. As Ackerman (2001) said, children’s’ conceptual changes are like those of scientists:
they happen through “action-in-the-world” (p. 3) to accommodate for experiences, and most likely
through a host of internal cognitive infrastructures. “Knowledge is not merely a commodity to be

20
 Visitor Studies Association - 2018 Conference Abstracts

transmitted, encoded, retained, and re-applied, but a personal experience to be constructed”

(Ackerman, 2001, p. 7).

Piles of theories of how the world works woven together form mental models. DiSessa and Cobb (2004)
argue that from Newton, to Einstein, and Darwin, theories embody generalizations to organize overly
abundant data that is subsequently viewed as part of a new theory. In this way diSessa and Cobb (2004)
posit theory as a lens, “teaching us how to see” (p. 4). How we see the world, is the crucial bit about
these theories since the lenses constructed through experience actual form our view. Just as “[t]he
world is not just sitting out there waiting to be to be uncovered, but gets progressively shaped and
transformed through the child's, or the scientist's, personal experience” (Ackerman, 2001),
constructionist thought highlights transformation and molding as the work of mental models. In order to
foster transparency in accounting for learning during interactions with museum interactions, I provide a
system to evaluate these theories as they develop through short interaction.

Methods: I applied a mixed methods approach to the observations I made (Clampet-Lundquist, Edin,
Kling & Duncan, 2011). In addition to observing visitors’ interactions with the display and taking
ethnographic field notes, I conducted a pre-post semi-structured interview protocol with participant
groups. The protocol asked about groups’ background knowledge, specifically about complex system
notions in traffic, and feelings about the role of change in social life. This protocol provided a wider view
of Ant Adaptation (Martin, Horn and Wilensky, 2018) to better understand what players learned about
the complex system through playing with the interactive tabletop.

Data Analysis: I defined learning as elaboration of how players talk about the entities in the game. As a
result, I analyzed the transcript data. As shown in figure 2, to build the map I would look at how the
person changed from describing ants as six legged (Figure 2a) to six legged ants that carry 50x their
weight and follow trails to find food (Figure 2d). I compare the change in processes and subprocesses
the constructivist dialogue maps track before, during, and after play. I track these elaborations of
entities and their actions to demonstrate players’ learning.

Importance

One can only learn so much during a 2-6-minute interaction in a museum. My constructivist dialogue
mapping is a method designed to show how much, through evaluating participants’ discussions. The
method, as developed, allows for a transparent accounting of what people discuss while interacting with
museum artifacts. This account is both useful in evaluating our exhibits and when talking to the public
because we can engender trust through providing information on what learning we can expect through
our public informal learning institutions. The session is an opportunity to share back with the community
the methods I developed through evaluating my own design in a museum. In the session, I will present
examples of the framework so we can actively discuss its potential use in evaluation and research to
foster interest in exploring the method and discussing its limitations.

References

Ackermann, E. (2001). Piaget’s Constructivism, Papert’s Constructionism: What’s the difference?

Chi, M. T., & Koeske, R. D. (1983). Network representation of a child's dinosaur knowledge.
Developmental psychology, 19(1), 29.

21
 Visitor Studies Association - 2018 Conference Abstracts

Clampet-Lundquist, S., Edin, K., Kling, J. R., & Duncan, G. J. (2011). Moving teenagers out of high-risk
neighborhoods: How girls fare better than boys. American Journal of Sociology.

diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments.
Journal of the Learning Sciences, 13(1), 77–103.

Martin, K., Horn, M, & Wilensky, U. (2018). Ant Adaptation: A complex interactive multitouch game
about ants designed for museums. Constructionism 2018 Conference, Vilnius, Lithuania.

Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis: A method sourcebook. CA,
US: Sage Publications.

Piaget, J. (1952). The origins of intelligence in children. International Universities Press, Inc.

Doomsday at the Science Museum: Measuring Visitor Understanding in an

Exhibit About Nuclear Weapons & Climate Change
Jana Greenslit, Research & Evaluation Coordinator, Museum of Science and Industry

Purpose

This session focuses on the evaluation of an exhibit about nuclear weapons, climate change, and the
other ways in which humanity has brought itself closer to or farther away from global catastrophe ever
since the dawn of the nuclear age. It utilizes interviews with visitors to determine how they view their
own role and ability to impact the issues of nuclear weapons and climate change after experiencing the
exhibit. The majority of the session will cover ways in which to design and implement an interview
protocol when talking to visitors about such politically charged and potentially unsettling topics, and
how to attempt to separate their prior feelings with these topics from their experience with the topic at
the museum.

Abstract

Turn Back the Clock is a temporary exhibit at the Museum of Science and Industry that focuses on the
history and enduring relevance of the Bulletin of Atomic Scientists and their Doomsday Clock, which
uses the concept of “minutes to midnight” as a metaphor for how close the world is to global
catastrophe from threats such as nuclear weapons and climate change. Its timeline covers “the dawn of
the nuclear age, how the Clock serves as a metaphor for the global challenges we face today, and the
potential applications of 21st century emerging technologies” (Museum of Science and Industry, 2017).

The evaluation of this exhibit focused in part on how visitors view their own role and ability to impact
the issues of nuclear weapons and climate change. Evaluation of this exhibit poses several challenges
–first, the relevance and global implications of these topics means that visitors may have prior feelings
on the subject, making it difficult to determine to what extent a visitor’s post-visit thoughts can be
attributed to the exhibit itself. Second, several ideas that the exhibit is trying to communicate, such as
the importance of collaboration between scientists and policy makers to keep the world safe, are
complex in that they draw upon multiple narratives throughout the exhibit, as opposed to being stated
explicitly at every point. Lastly, the concept of global catastrophe could make people nervous. How do

22
 Visitor Studies Association - 2018 Conference Abstracts

you capture the idea that visitors may not necessarily feel “good” about nuclear weapons and climate
change after seeing this exhibit, but could be nonetheless instilled with a greater sense of hope about
the future of our planet?

This presentation focuses on the process of designing an interview protocol that measures for visitor
understanding of this topic. It will cover lessons learned throughout the development and
implementation of an interview protocol as well as analysis of the interview transcripts. The final
evaluation results demonstrated that most participants did interpret the message of the exhibit as
positive. However, it was clear from the interview transcripts that individuals’ background knowledge
and attitudes strongly influenced their interpretation of the exhibit and subsequently their interview
responses.

This session will provide a launching point for attendees to think about how to measure understanding
in museum exhibits with similarly charged but nonetheless relevant topics. In what ways can evaluators
structure conversations with guests to help separate guests’ prior feelings about a topic from their
experience with that topic while at a museum? Lessons learned from this evaluation will be shared, but
attendees are also encouraged to come with their own ideas for how to collect data around these kind
of topics and share with the group.

Importance

As museums increasingly elect to focus on current, politically charged topics, it is important for
evaluators to understand how to facilitate these types of conversations and discern meaning from them.
While topics such as these may add a layer of complexity to an evaluation, they also add a layer of
depth, as guests make immediate connections back to their own lives and even to their futures.
Museum exhibits or programs that tackle vital issues can and should be highlighted through evaluation,
which only pushes the field forward in terms of what we consider to be appropriate or relevant issues
for our audiences.

References

Museum of Science and Industry, 2017. Iconic Doomsday Clock is the Focus of the Museum of Science
and Industry, Chicago’s Newest Exhibit. Retrieved from
https://www.msichicago.org/fileadmin/assets/press/turn_back_the_clock/TBTC_Press_Release.
pdf

Additional Links

For more information about the exhibit:

https://www.msichicago.org/press/exhibits-and-events/turn-back-the-clock/

Formative Evaluation of Facilitated Play Engagement with Early-Childhood

Learners
Brian Slattery, Lincoln Park Zoo
Erin Shoffstall, Lincoln Park Zoo

23
 Visitor Studies Association - 2018 Conference Abstracts

Purpose

This session will give participants insight into a collaborative relationship between two evaluation teams
working toward the same goal while learning more about how to approach an early childhood
evaluation project. We will share our methods, tools, and key findings as we share our experience in this
collaborative journey. Participants will learn about how a cycle of reflection and implementation informs
the development of an early childhood interpretation program. The aim is to prompt a discussion about
the advantages and challenges of working with multiple evaluation teams in the same organization.

Abstract

Background

A perennial concern of informal learning institutions is a lack of capacity to fully evaluate a range of
programs, given varying constraints on evaluation teams. This paper presents an on-going effort
between two evaluation teams at Lincoln Park Zoo, regarding the interaction between interpretive
volunteers and early-childhood visitors as part of a play facilitation program. Initial findings showed that
interactions between guest engagement volunteers and children at educational carts were largely led by
facilitators. But, early childhood pedagogy suggests play is most valuable to the learner when it is
child-led, which was an existing area of interest for the zoo’s Child & Family Learning (CFL) team.
Combined with the initial findings, in 2017, the CFL team developed the role of a Play Assistant (PA),
with the goal of facilitating play engagements for families at the zoo.

To assist in facilitating early-childhood play, PAs were equipped with a variety of techniques and tools
that are best suited for younger children. In order to understand all aspects of the PA
program—including outcomes for the PA participants as well as outcomes for the child visitor—the
Evaluation & Learning Research team and Audience Research & Evaluation team partnered to conduct
an interdisciplinary study. The goal of this effort was to support the formative, participatory evaluation
of interpretive programming and facilitative behavior, by engaging in cycles of reflection and
implementation that involved both evaluators and stakeholders. In this paper, we will discuss how this
cycle helped support PAs to engage in more child-directed interactions, and how evaluators were better
able to see and assess these changes through a collaborative, iterative process.

Methods

The two teams of evaluators used varying mixed-methods and analytic approaches to evaluate the
intersection of child engagement and interpretive strategies. Data collection consisted of observations,
interviews, surveys, and reflections at facilitated play stations at multiple locations, including the
play-oriented Main Barn at Farm-in-the-Zoo, and the Regenstein Center for African Apes. Data was
shared with stakeholders in the CFL team through data meetings, where both evaluators and
stakeholders highlighted important findings that would inform programmatic changes. Reflections from
evaluators and stakeholders also informed progressive phases of data collection and analysis, which
took place over summer 2017 and winter 2017-18.

Findings

While the initial educational cart findings largely indicated interpreter-driven conversations, the two
rounds of subsequent evaluations showed growth in PAs following, rather than directing, children’s play,

24
 Visitor Studies Association - 2018 Conference Abstracts

which was also accompanied by shifts in children’s engagement with the hands-on play activities.
Through the evaluation process, the evaluation teams worked together to develop the instruments and
methodology used to evaluate PAs and child visitors, as well as refine this approach through stakeholder
feedback. These interpretive data meetings led to clearer operationalization of the core phenomena
being observed, which informed the instrument design and evaluative approach of the second round.
The two teams of evaluators had the opportunity to reflect on their practices, and developed the second
round instruments as a result of the teams’ differing strengths.

Importance

This paper contributes to the ongoing work of improving the evaluation of early-childhood programs,
particularly the facilitation of child-directed play. These programs provide the opportunity for children
to decide the content and form of their interaction with facilitators, which reinforces the free-choice
nature of informal learning spaces such as zoos.
More broadly, as a field, evaluators often work amongst their own teams. This paper emphasizes the
power of bringing two evaluation teams together while approaching evaluation with a participatory lens.
As informal learning institutions have to make decisions as to what can and cannot be evaluated given
what resources are available, this conversation of “combining forces” with stakeholders and other
evaluators becomes essential to provide the most meaningful and rich data that tells the whole story.

References

Dockett, S. (2010). The challenge of play for early childhood educators. In Rethinking play and pedagogy
in early childhood education (pp. 40-55). Routledge.

Fetterman, D., Rodríguez-Campos, L., Wandersman, A., & O’Sullivan, R. G. (2014). Collaborative,
participatory, and empowerment evaluation: Building a strong conceptual foundation for
stakeholder involvement approaches to evaluation (A response to Cousins, Whitmore, and
Shulha, 2013).

Pattison, S. A., & Dierking, L. D. (2013). Staff-mediated learning in museums: A social interaction
perspective. Visitor Studies, 16(2).

Schön, D. A. (2017). The reflective practitioner: How professionals think in action. Routledge.

The F-Word: Failure and Failing Towards Success

Mary Jackson, Woodland Park Zoo
Taline Kuyumjian, Kuyumjian Consulting LLC
Kathryn Owen, Kathryn Owen Consulting LLC
Dr. Jennifer Rigney, Monterey Bay Aquarium

Purpose

The value of learning from failure is undeniable. Reflecting on and sharing our failures fosters
transparency and has been described as the key to innovation. Despite this acknowledgement,
embracing this concept in our field or in our individual practice is still not common.

25
 Visitor Studies Association - 2018 Conference Abstracts

The purpose of this session is to encourage an open, honest, and lively dialogue about our professional
failures. Through presentations, discussion and a lively, informal activity inviting attendees to share their
own professional blunders, we hope to foster a friendly, open dialogue as we strive to make failure an
acceptable and celebrated discussion.

In this session participants will:

• Understand both the value and barriers to examining mistakes and learning from failure.
• Take away key lessons to avoid similar mistakes in their own practice.
• Have increased comfort in openly discussing failure within our professional environments.

Abstract

It’s okay to fail. At least that’s a sentiment we often hear from today’s celebrated thought leaders and
innovators, usually through inspirational quotes such as “Fail fast, fail small, learn big” or “Failure is
success if we learn from it”. Although most would probably agree there is significant value in reflecting
on our mistakes, truly learning from failure is much more difficult to put into practice. There are
emotional, social and technical barriers that deter individuals and organizations from revisiting past
missteps. When examining individual failures we may feel negative emotions or a reduced
self-confidence that can inhibit learning. Most organizational systems have inherent properties that
inhibit managing failures. The skills that are beneficial in this reflective process, such as openness,
patience, or acceptance, are not often rewarded, unlike say decisiveness, efficiency and action.
Ultimately this inability to discuss these disappointments makes it more difficult to prevent larger
failures down the road.

Evaluators play a unique role in that we facilitate access to data that enables the people we work with to
identify and reflect on shortcomings. As such we are often adept at helping our clients or stakeholders
turn negative findings into valuable learning opportunities, but how often do we turn that lens upon
ourselves and our practice?

In this session we will briefly cover the three suggested processes for failing intelligently: 1) identifying
failure, 2) analyzing and discussing failure and 3) experimentation (Cannon & Edmonson, 2005). Then we
will put the first process into practice, hearing from four evaluation professionals as they reflect on their
own past blunders and how they found the lessons and hopefully the humor in these failures. Stories
include forgetting to check our own assumptions, losing sight of the goal while wasting valuable
resources, and failing to consider existing perspectives and organizational cultures in an attempt to
implement change.

Throughout the presentations the panelists will share their tips for separating failure from fault and how
to reframe failure as a valuable learning experience. Session attendees will also be invited to join the
discussion as we reflect on our field and the things we are doing to encourage or muffle frank
conversations about our professional mistakes.

During this interactive session we will strive to create a safe, open, comfortable, and fun environment
where admitting to our mistakes is not a sign of weakness, but of professionalism. After hearing from
our presenters, attendees will be invited to participate in a fun, informal activity where they can confess
to their own professional mistakes, big or small. Most importantly people will leave this session with an
enthusiasm to celebrate the rewards of failure.

26
 Visitor Studies Association - 2018 Conference Abstracts

Importance

As a field that is frequently involved in helping other stakeholders identify and learn from professional
failures, it is extremely important that we ask the same of ourselves and our practice. Demonstrating
our own ability to discuss our missteps in an open setting fosters greater learning and acceptance in the
field. We hope all those who attend or share their stories feel more empowered to own their failures,
learn from them, and improve our collective practice.

References

Cannon, & Edmondson. (2005). Failing to Learn and Learning to Fail (Intelligently): How Great
Organizations Put Failure to Work to Innovate and Improve. Long Range Planning, 38(3),
299-319.

Friday, July 20
10:15-11:30 AM - Concurrent Sessions
What's Next? Visitor Journeys at the Clyfford Still Museum
Sonia Rae, Clyfford Still Museum
Katherine Gean, Slover Linett Audience Research
Tanya Treptow, Slover Linett Audience Research

Purpose

In this panel presentation, representatives from the Clyfford Still Museum (CSM) in Denver and the
audience research firm Slover Linett will discuss an in-depth case study of our research partnership. We
will share key moments regarding how CSM determined that it was the right time for research, and how
we adapted innovative data collection methodologies from UX research to understand visitor journeys.
We will provide a behind-the-scenes look at how we brought non-visitors into the research—inviting
them to think-aloud as they explored the museum website, grounds, and galleries to illuminate a
first-time museum experience. We will guide attendees in a simple hands-on activity during the talk and
hope that they will leave with ideas to add to their evaluation toolkit and inspiration for how to translate
research findings into institutional change. Though situated in an art museum, the lessons learned
throughout apply to many kinds of cultural organizations.

Abstract

In early 2017, Clyfford Still Museum (CSM) in Denver began this research project with the aim of learning
more about how to continue to grow their audiences, reach more Denverites, and make their institution
more relevant to more people. The Clyfford Still Museum and Slover Linett set out to learn more from
non-visitors in Denver about how the museum could become a more top-of-mind destination and create
deeper, more fulfilling visitor experiences. In essence, CSM needed insight on how to tell the story of
itself as an institution and the single-artist on which it was built.

27
 Visitor Studies Association - 2018 Conference Abstracts

In this panel presentations, we will start with an overview of the questions and organizational strategies
that CSM began their research journey with, as well as the qualitative user-experience methods used by
Slover Linett to answer those questions. The session will outline the process by which Slover Linett
employed a User Experience research model to map the holistic journeys of non-museum visitors (i.e.
have never visited CSM before) to understand opportunities for better communication and a more
relevant museum experience. We’ll be including a fun, simple audience activity to give attendees a
chance to practice methods we used.

Research methods began with online interviews to understand visitor experience decision-making via
the CSM website and continued with follow-along cognitive interviews to unpack visitors’ first
experience at the museum. We will spend some time walking through these techniques to outline key
best practices for these methods. In particular, we will focus on ways to ask questions and truly listen,
will discuss the balance of observation versus diagnostic conversation, and how to recognize missed
opportunities for better communication and deeper experiences.

We will then share the key findings from the research before engaging in a dialogue with the CSM
Director of Audience and Community Engagement about the process of translating research findings
into a plan for implementation and change within the Museum. The Clyfford Still Museum Director will
share some of the key successes they’ve found since the research period and ongoing challenges or
questions they’ve run up against in making data-driven change at the Museum. Topics for discussion
may include setting up research for broad institutional buy-in, ways to incorporate change
incrementally, and how to push through challenges by reframing priorities. While the findings in case
study help bring more transparency to the Clyfford Still Museum’s interactions with its specific
audiences, the process we used can have resonance for many different types of cultural organizations.

Importance

We hope that this panel presentation will provide an important perspective on adapting UX research
methods to museum and cultural institution practices, as well as helping to think about the needs of
both current visitors and potential visitors to an institution. By the end of the presentation, we hope
that attendees will gain concrete insights on using research to build audience-driven experiences at their
own cultural institutions.

Additional Links

Clyfford Still Museum: ​https://clyffordstillmuseum.org/

Slover Linett Audience Research: ​www.sloverlinett.com/

What Research Tells Us About Supporting Engagement in Science

Marta Beyer, Museum of Science, Boston
Rachel Bonnette, University of Pittsburgh
Emily Howell, University of Wisconsin-Madison

Purpose

28
 Visitor Studies Association - 2018 Conference Abstracts

People encounter science through formal schooling and at multiple points in their daily lives, yet data
indicate more work needs to be done to encourage the growth of STEM-related careers and an
informed, science-literate public (Falk & Dierking, 2010; National Academies of Sciences, Engineering,
and Medicine, 2016). What can research tell us about ways to design programs and learning experiences
that will motivate people to engage with science? Through this session, conference attendees will leave
with deeper knowledge about the strategies that facilitate engagement in science and the types of
impacts that can be achieved. Conference attendees will also have a greater understanding of how to
measure engagement in science in different settings and will learn about new resources and frameworks
for assessing science learning and engagement.

Abstract

Findings from large-scale projects will provide conference attendees with a better sense of how to
implement and evaluate projects that undertake this important work. In this session, three panelists
who have studied this topic from different vantage points will share what they know about the public’s
associations with science and about ways to support successful science engagement and learning in
formal and informal settings and beyond.

Rachel Bonnette will begin by presenting information about the Learning Activation Lab. She will
describe the steps taken to develop a framework for understanding and measuring the various
dimensions that support science learning: fascination, values, scientific sensemaking, and competency
beliefs (see http://activationlab.org/tools; Dorph, Cannady, & Schunn, 2016). She will summarize how
the dataset was used in a recent Activation Lab study (Bonnette, Crowley, & Schunn, pre-submission) to
illustrate both the ways in which science activation measurements have been used and what the
resultant findings elucidate about challenges and potential solutions to supporting engagement in and
out of school, particularly motivation for girls and children who identify as members of racial or ethnic
groups that are underrepresented in STEM fields.

Marta Beyer will then present findings from the NSF-funded ChemAttitudes project. Research from this
project examined how particular content, program formats, and facilitation strategies used in informal
settings could support the public’s view of chemistry. In particular, this project analyzed the types of
design criteria that might lead visitors to have increased interest in chemistry, enhanced understanding
and perception of its relevance, and stronger belief in their self-efficacy with respect to the subject.
These measures, based on the ARCS motivational model, speak to the Learning Activation Lab’s
dimensions (Keller, 1987). By sharing data gathered from visitors through observations, videotaping, and
interviews collected at two science museums, this presentation will offer another perspective on the
types of experiences needed for supporting engagement and interest in chemistry.

Emily Howell will expand the conversation beyond informal and formal settings and share findings from
a second research study associated with the ChemAttitudes project. This study provides insight into how
members of the general public view various fields of science and, in particular, chemistry. Findings from
two large-scale quantitative surveys distributed through Amazon Mechanical Turk will be shared along
with considerations of how interest, relevance, and self-efficacy seem to relate to each other and factor
into whether or not the public is likely to engage with activities in science museums. The results will help
illuminate how factors that the ChemAttitudes and Learning Activation Lab have used for understanding
science engagement in schools and informal settings also provide insight into the general public’s
connections with science.

29
 Visitor Studies Association - 2018 Conference Abstracts

By hearing about these frameworks meant to support the design and measurement of science
engagement, attendees will enhance their understanding of how to support these types of experiences.
Specifically, findings related to key strategies and factors will be relevant to all those currently involved
in evaluating or creating programs meant to spark changes in attitudes related to science.

Importance

Information from this session will offer insights to all those currently designing and studying science
engagement in ISEs and other types of settings. In particular, experienced evaluators and researchers
along with designers and on-the-floor educators will benefit by learning about recent research for how
to measure and encourage engagement with science. Not only is it crucial for the field to continue
considering how to assess and design for the various dimensions of science engagement, but,
specifically, it is importance to include the topic of chemistry in these conversations, as it has been
underrepresented in informal learning institutions.

References

Bonnette, R. N., Crowley, K., & Schunn, C. D. (pre-submission). Out-of-school learning experiences stem
the middle-school slide in science fascination.

Dorph, R., Cannady, M. A., & Schunn, C. D. (2016). How science learning activation enables success for
youth in science learning experiences. Electronic Journal of Science Education, 20(8).

Falk, J.H. & Dierking, L.D. (2010). The 95 percent solution: School is not where most Americans learn
most of their science. American Scientist, 98(6), 486-493.

Keller, J.M. (1987). Development and use of the ARCS model of instructional design. Journal of
Instructional Development, 10(3), 2-10.

National Academies of Sciences, Engineering, and Medicine. (2016). Science Literacy: Concepts,
Contexts, and Consequences. Washington, DC: The National Academies Press.
https://doi.org/10.17226/23595

Additional Links

http://activationlab.org/tools

http://www.nisenet.org/chemattitudes

Reconciling Public Trust with Museums, Non-Profits, and Evaluation

Sarah Cohn, Aurora Consulting
Vidhya Shanker, Rainbow Research
Renae Youngs, Minnesota State Arts Board

Purpose

30
 Visitor Studies Association - 2018 Conference Abstracts

The trust of visitors, clients, and communities is important to both evaluators and museums. It lends our
work credibility, but we rarely question why, or even whether, we deserve it. This panel will address the
political and social histories of the evaluation, non-profit, and museum fields and how that history
relates to the existence or loss of public trust in those fields. In order to cultivate deeper trust between
professionals in these three fields and their key stakeholders, we must be aware of how our work is
intertwined with and sometimes complicit in perpetuating a broader context of unjust social and
political conditions. The panelists will briefly outline historical and contemporary conditions that hinder
trusting relationships in each field, then facilitate attendees’ discussion of justice- and trust-building
actions to incorporate into our professional practice.

Abstract

Visitor studies is a relatively young area of professional practice, and a core value of this work
(sometimes implicit, often explicit) is that we are serving the visitors of informal learning institutions.
Sometimes we deliberately take on the responsibility of being the “voice of the visitor,” while other
times our work indirectly benefits visitors by improving an institution’s practice. Through this lens, it can
be easy to presume that our work merits the public’s trust. Sadly, we must realize that the structures,
histories, and practices that underlie our work - whether in museums, nonprofits and philanthropy, or
evaluation and research/social science - can actively erode the trust we seek to build with those we
serve.

When we look to the history of museums, the field’s origins are rooted in the collecting habits of
aristocrats and colonizers - and the political, economic, and social conditions that made their
accumulation of wealth and objects possible. Practices of acquisition, classification, and didactic,
“neutral” interpretation have been intentional features of museum practice for centuries. Only in the
past 30 years have museums begun to embrace the idea that they can and should be of service to all via
an orientation toward free-choice learning.

The field of evaluation is similarly intertwined with unjust social and political systems. Its origins reflect
the accountability expected by those who hold an accumulation of wealth (such as government or
philanthropic entities) as a condition of distributing their resources for some social good. While
accountability is not fundamentally bad, it does highlight and reinforce power imbalances between
resource-holders and resource-seekers. Moreover, evaluation practice has often (but not always) drawn
on the wider enterprise of social science research to set its norms and practices. Expectations from
Western worldviews about objectivity, quantifiability, hierarchy, and certainty have thus crept into
evaluation practice and values, and this cultural paradigm that reinforces inequitable power dynamics
has often been invisible to evaluators themselves.

Overlaid on these historical issues with museums and evaluation are the wider contexts of the nonprofit
and philanthropic sectors in which most of us work. In the United States, most philanthropic entities
exist because of the wealth accumulation of their founders, which was sometimes made possible by
exploiting unjust social conditions and structures. Nowadays, all private foundations exist partly so that
accumulated wealth can accomplish a social good, but also partly so the wealth can be protected from
other means of redistribution for social good (i.e., taxation). Not-for-profit organizations in the U.S. must
exist for some charitable purpose, and often work to advance a social good or mitigate unjust social
conditions. But an organization’s commitment to doing good does not automatically translate into
adopting all the practices that ensure it is worthy of the public’s trust.

31
 Visitor Studies Association - 2018 Conference Abstracts

These factors create a messy, problematic context in which we all seek to earn public trust in our work,
but they are not often discussed in our professional communities. This session will name the big,
important challenges we face so we can discuss ways to mitigate them.

Importance

The broader evaluation field and non-profit sector in the U.S. are built on the same fraught, imperfect,
and unequal footing as the rest of our society. Institutions that benefit from or reinforce the
concentration of wealth, power, and authority in the hands of the few need to reckon with our place in
these wider, unjust systems. We do ourselves, our work, our organizations, and our communities a
disservice by remaining uncritical or unconcerned about whether our actions bolster or undermine
public trust. Instead of presuming to have the public’s trust, we must interrogate our actions, our
motivations, and what they communicate to others about our trustworthiness. Until we can recognize
how we may be undermining our own work, we will not be able to realize the potential of fully stepping
up to existing for the public good and because of public trust.

References
Abt, J. (2006). The origins of the public museum. In S. Macdonald (Ed.), A companion to museum studies
(115-134). Malden, MA: Blackwell.

Alexander, E.P. & Alexander, M. (2008). Museums in motion: An introduction to the history and
functions
of museums. Lanham, MD: AltaMira Press.

Cameron, D.F. (1971). The museum, a temple or a forum. Reprinted in G. Anderson (Ed.), Reinventing
the
museum: Historical and contemporary perspectives on the paradigm shift. (2004). (61-73).
Lanham, MD: AltaMira Press.

Hein, G.E. (2006). Museum education. In S. Macdonald (Ed.), A companion to museum studies (340-352).
Malden, MA: Blackwell.

Hirzy, E. (1992). Excellence and equity: Education and the public dimension of museums. Washington,
DC: American Association of Museums.

Hood, M.G. (1983). Staying away: Why people choose not to visit museums. Reprinted in G. Anderson
(Ed.), Reinventing the museum: Historical and contemporary perspectives on the paradigm shift.
(2004). (150-157). Lanham, MD: AltaMira Press.

Malaro, M.C. (1994). Museum governance: Mission, ethics, policy. Washington, DC: Smithsonian
Institution Press.

Museums as Sites for Social Action community of practice. (2017). MASS action toolkit. Retrieved from
​https://www.museumaction.org/s/TOOLKIT_10_2017.pdf

Mesa-Bains, A. (1992). The real multiculturalism: A struggle for authority and power. Reprinted in G.
Anderson (Ed.), Reinventing the museum: Historical and contemporary perspectives on the
paradigm shift. (2004). (99-109). Lanham, MD: AltaMira Press.

32
 Visitor Studies Association - 2018 Conference Abstracts

Paquet Kinsley, R., & Wittman, A. (2018). The Incluseum. Retrieved from ​https://incluseum.com/

Rand, J. (2000). The visitors’ bill of rights. Reprinted in G. Anderson (Ed.), Reinventing the museum:
Historical and contemporary perspectives on the paradigm shift. (2004). (158-159). Lanham,
MD:
AltaMira Press.

Stevens, G. (2017, June 28). Excellence and equity at 25: Then, now next. Alliance Lab blog post.
Retrieved from ​https://www.aam-us.org/2017/06/28/excellence-equity-at-25-then-now-next/

Sullivan, R. (1994). Evaluating the ethics and consciences of museums. Reprinted in G. Anderson (Ed.),
Reinventing the museum: Historical and contemporary perspectives on the paradigm shift.
(2004). (257-263). Lanham, MD: AltaMira Press.

The Empathetic Museum collaborative. (2018). The empathetic museum. Retrieved from
http://empatheticmuseum.weebly.com/

Weil, S. E. (2002). Making museums matter. Washington. DC: Smithsonian Books.

Walking the Talk: Visitors' Understanding of Zoos/Aquariums' Conservation

Efforts
Amy Niedbalski, Saint Louis Zoo
Jennifer Rigney, Monterey Bay Aquarium
Manda Smith, Lincoln Park Zoo

Purpose

Museums invest in a variety of activities beyond their onsite experience, including education, research,
and advocacy. However, museums vary in the extent to which they communicate these activities to
visitors. Most museum missions require collective action and visitors may not feel they can create
meaningful change towards such large societal problems. The purpose of this session is to explore ways
that museums can raise awareness of their work and discuss whether doing so can foster trust in the
organization, help visitors feel part of a community committed to social change, increase efficacy to act
and join the museum in acting for social change.
In this session, three zoo and aquarium evaluators will discuss their work to better understand how to
effectively communicate their organization’s work to visitors, discuss implications of doing so for
perceptions of the organization and discuss the conditions under which visitors feel ready to act for
change.

Abstract

Museums are more than educational institutions and typically work toward goals such as civic
engagement, social justice and conservation. One way museums work to achieve these mission-related
goals is through their public galleries. Effecting change requires not only raising awareness but crafting
exhibits that spark visitors to change their perspectives, attitudes and even behaviors. Furthermore,

33
 Visitor Studies Association - 2018 Conference Abstracts

museums often engage in other strategies to work toward their missions, such as providing educational
programs for children and teens, conducting research, and advancing policy.

The purpose of this session is to explore ways to interpret museums’ work to visitors in ways that
connect with their values and interests and discuss possible benefits to both the museums and visitors
of doing so. Museums might be concerned with communicating their work to visitors and the public at
large to increase favorability towards these organizations, foster trust, normalize target behaviors, and
help visitors feel part of a community committed to social change. Many of these potential outcomes
are preconditions for inspiring action. For example, perceiving an organization to hold similar values and
goals increases trust in that organization, which in turn serves as an important mediator of acting on
information provided by the trusted organization (Winter & Cvetokovich, 2010). Additionally, perceiving
that organizations are doing much to advance their goals may make personal engagement in those social
movements feel more efficacious, which is crucial for individual involvement in social movements
(Bamberg, Rees, & Seebauer, 2015; Jugert, et al., 2016). Thus, if museums want to inspire their visitors
to engage in action post-visit they should be concerned with perception of their research and advocacy
work.

In the current session, three zoo and aquarium evaluators will present their work to better understand
the importance of interpreting their organization’s research and advocacy work to visitors. Zoos and
aquariums engage in a variety of activities that advance conservation goals such as public education
efforts, wildlife research, sustainability recommendations, and environmental advocacy. Because of the
increasing public scrutiny of captive animals, zoos and aquariums have been particularly concerned with
public perceptions and interpreting our work may be one strategy for increasing both public trust and
bringing visitors along as we advance our mission.

In this session, Niedbalski will present results from an evaluation of Zoo member awareness and
understanding of the Saint Louis Zoo’s Institute for Conservation Medicine and discuss the implications
for the Zoo’s messaging around this work (Padda, Niedbalski, Tate & Deem, 2018). Rigney will discuss
results from a series of studies designed to evaluate whether the Monterey Bay Aquarium’s programs
that turn its wildlife conservation efforts into 15-minute stories are successful at increasing attendees’
understanding of how the Aquarium works to protect the ocean and whether that increased
understanding influences visitors trust in and affinity for the Aquarium. Finally, Smith will round out the
discussion by presenting results from a study exploring visitors’ and community members’ perceived
behavioral control around the Zoos’ calls to action and discuss the role of feeling part of a conservation
community in inspiring action.

Importance

Museums have increasingly focused on their public value in recent years as funding for museums has
declined and as the public has raised the bar for what they consider to be fulfilling museum experiences
(Korn, 2013). The public derives important personal benefits from visiting museum galleries (Packer,
2008) and museums enact their mission, in part, through these galleries. However, many museums
make progress towards their mission beyond working through their casual visitors—they provide
educational programs for children and teens; they conduct mission-relevant research; and they
advocate for policy changes. If the public has raised the bar on what they expect from museums,
perhaps helping visitors understand them as “more than a museum” can foster trust and garner public
support. Interpreting this important work to visitors might also help them feel part of a vast collective of
museum stakeholders who are ready to act with the museum for change.

34
 Visitor Studies Association - 2018 Conference Abstracts

References

Bamberg, S., Rees, J., & Seebauer, S. (2015). Collective climate action: Determinants of participation
intention in community-based pro-environmental initiatives. Journal of Environmental
Psychology, 43, 155–165. ​https://doi.org/10.1016/j.jenvp.2015.06.006

Jugert, P., Greenaway, K. H., Barth, M., Büchner, R., Eisentraut, S., & Fritsche, I. (2016). Collective
efficacy
increases pro-environmental intentions through increasing self-efficacy. Journal of
Environmental
Psychology, 48, 12–23. ​https://doi.org/10.1016/j.jenvp.2016.08.003

Korn, R. (2013). Creating public value through intentional practice. In C.A. Scott (Ed.), Museums and
Public Value (pp. 31-43). Burlington, VT: Ashgate Publishing.

Padda, H., Niedbalski, A., Tate, E., & Deem, S. L. (2018). Member Perceptions of the One Health Initiative
at a Zoological Institution. Frontiers in Veterinary Science, 5(February), 22.
https://doi.org/10.3389/fvets.2018.00022

Packer, J. (2008). Beyond Learning : Exploring Visitors ’ Perceptions of the Value and Benefits. Curator:
The Museum Journal, 51(1), 33–54. ​https://doi.org/10.1111/j.2151-6952.2008.tb00293.x

Winter, P. L., & Cvetkovich, G. T. (2010). Trust Mediates Conservation-Related Behaviors. Ecopsychology,
2(4), 211–219. ​https://doi.org/10.1089/eco.2010.0046

The Evolution of Evaluation to Support Community Engagement

Sarah Lee, President, Slover Linett Audience Research
Ken Morris, Manager of Evaluation and Research, Detroit Institute of Arts
Elizabeth Bolander, Director of Audience Insights and Services, Cleveland Museum of Art

Purpose

In recent years, museums and other informal learning organizations have been focusing more and more
attention on reaching and engaging communities that have typically been underrepresented among
their visitor bases—particularly communities of color or lower-income communities, but also those who
simply haven’t found traditional museum experiences to be relevant to them. In this session, we want to
consider the implications of this shift on the evaluation field: What new skills or ways of working does it
demand from evaluators? What new opportunities does it open up for evaluation and audience research
to contribute to the overall success of informal learning organizations? And how can VSA support the
field in taking full advantage of that potential?

Abstract

A recent evaluation of the Irvine Foundation’s New California Arts Fund, conducted by Slover Linett
Audience Research, suggests that cultural organizations that want to deepen their ability to
authentically and sustainably engage communities of color or lower-income communities need to do

35
 Visitor Studies Association - 2018 Conference Abstracts

more than just offer new kinds of programming (Lee & Gean, 2018). They need to build new internal
capacities and ways of working—including deepening their ability to gather input from their
communities and strengthening their evaluation and measurement muscles. This finding resonates with
the on-the-ground experience of our panelists at the Detroit Institute of Arts and the Cleveland Museum
of Art and, we suspect, with our colleagues throughout the VSA community. We’ve seen that community
engagement efforts require spending time and energy getting to know communities, building empathy
for the ways that they experience cultural and informal learning organizations, and understanding how
to develop programming that is interesting, enticing, and high impact.

Such work makes natural use of tools in the evaluator’s toolkit: observation, interviewing, focus groups,
pattern recognition, identifying actionable implications, and so on. But deploying those tools outside of
the institution and with communities that haven’t already chosen to visit may require some new
capabilities, from the very tactical (like recruiting non-visitors) to the more strategic (like cultural
competency). These ways of working can also change the relationship between evaluation and program
development—sometimes blurring the boundary between those two traditionally separate domains. In
this session, we want to leverage the knowledge and experience of the VSA community to better
understand: (1) how individual evaluators, both at informal learning institutions and in independent
practices, are currently connecting with community engagement work; (2) what new opportunities and
challenges they’re facing; and (3) how they’re building the capacity necessary to contribute meaningfully
to community engagement efforts.

We’ll kick the session off with a brief overview of the community engagement landscape and the role
that evaluators are playing within it. This overview will draw on the report for the Irvine Foundation
cited above, as well as panelists’ experiences in their institutions. We’ll then facilitate a conversation
among participants about the key challenges, questions, or opportunities that they’ve encountered in
the community engagement work they’ve been involved in, aiming to describe and prioritize the most
pressing ways in which the field is or ought to be changing in order to effectively support community
engagement.

From there, we’ll turn toward examining how VSA can support the field in growing in these areas. We’ll
select 4 or 5 key challenges to focus on, breaking into small working groups to brainstorm specific
resources or forms of support. Some of these may be existing resources (e.g., journal articles or other
reading, online communities, etc.), which we’ll compile into a list and work with VSA to distribute to the
community. Others may be resources that need to be developed; our goal will be to identify a few
people who would be interested in collaborating to develop critical resources on community
engagement for the VSA community.

Importance

As the museum and informal learning field invests more time, energy, and resources in community
engagement, the evaluation and visitor studies field may need to change too. This session will provide
space for an open, candid dialogue among evaluators involved in (or aspiring to be involved in)
community engagement efforts about how the profession is changing or ought to change. We see a
direct connection between this conversation and the conference theme: in order to support the field in
fostering and maintaining the trust of the public, evaluators must be able to contribute to institutions’
ability to effectively engage their communities.

References

36
 Visitor Studies Association - 2018 Conference Abstracts

Lee, S. & Gean, K. (2018). The Engagement Revolution: A study of strategic organizational
transformation
in 10 California arts nonprofits. Retrieved from ​https://www.irvine.org/arts/learning

Program Evaluation Practice and Capacity in the Canadian Art Museum Sector:
Findings from a National Study of Museum Practitioners
Agnieszka Chalas, Queen’s University

Purpose

In recent years, art museums have increasingly been called upon to engage in comprehensive
evaluations of their educational programs in an effort to both foster transparency and strengthen public
trust. Despite this trend, research on program evaluation in the context of art museums is scarce.
Consequently, little is known about what such museums are doing to evaluate their educational
programs or to what extent they possess the internal capacities to conduct evaluations and use
evaluation findings for both accountability and learning purposes. In this paper, I present the results of a
pan-Canadian study that was undertaken to fill this gap in the literature. Its objectives were to describe
what evaluation practice and capacity looks like across the Canadian art museum sector, and elucidate
specific examples of institutional successes for the purpose of sharing them with the growing number of
art museums interested in integrating evaluation into their organizational cultures.

Abstract

A qualitative interview study was used to address the purposes of this research. Specifically, a key
informant purposeful sampling strategy was employed to identify practitioners with responsibility for
program evaluation working in Canada’s largest publicly owned art museums. Using a semi-structured
interview protocol, an in-depth interview of around 60 minutes in length was carried out with each art
museum practitioner that agreed to participate in the study. The protocol consisted of 25 open-ended
questions grouped according to the dimensions of a literature-derived conceptual framework of
evaluation capacity that was developed to guide this research. Accordingly, interview questions asked
practitioners to report on both their art museum’s contextual characteristics and extant evaluation
culture. Practitioners were likewise asked about: (a) what drove program evaluations at their museums
(i.e., internal demands versus external pressure), (b) the type and amount of evaluation resources and
training supports that were available to them, (c) their current capacity to collect, reflect on, and
incorporate various forms of data into their educational program development, and (d) the specific
evaluation activities they engaged in to document programmatic impacts. All interviews were recorded
using a digital recorder, transcribed in full, and sent to interviewees for review. A framework approach
to qualitative content analysis (Ritchie & Spencer, 2002) was employed to manage, analyze, and
interpret the interview data collected during this study. Specifically, analysis involved: familiarizing
myself with the data; developing a data-coding system; coding the data using a combined
deductive-inductive approach; synthesizing the entire data set into structured matrix coding charts; and
reviewing the data across the charts to draw out descriptive and explanatory conclusions clustered
around each of the dimensions comprising the afore-mentioned conceptual framework. Overall, the
findings of this study shed new light on Canadian art museum’s program evaluation practices and their
current capacities for evaluation while also providing considerable insight into the factors and conditions

37
 Visitor Studies Association - 2018 Conference Abstracts

that can both promote and limit such capacity in these unique learning settings. The study’s findings
likewise highlight both the challenges Canadian art museums face with respect to implementing
program evaluation and their needs for assistance while also pinpointing distinct strategies that were
used to overcome limitations of resource scarcity. In addition to sharing the results of this research,
during this presentation I will also examine the implications of the study’s findings for visitor studies
theory, research, and practice, and will begin a discussion about how, as a field, we might both support
better program evaluation practices and strengthen capacity for evaluation in art museums and other
institutions of informal learning.

Importance

This study addresses a pressing literature gap by advancing knowledge on program evaluation practice
and capacity in the Canadian art museum sector while providing a seminal basis for future transnational
inquiry, thus initiating field-wide learning and comparison. It likewise supports a new conceptual
framework that can be used to describe program evaluation practice and capacity across varying
museum environments. Moreover, the study also offers both art and other types of museums assistance
in the form of institutional successes and a practical tool (framework operationalized as a rubric) that
can help practitioners reflect on their capacity for evaluation and identify appropriate activities to
further develop lower-capacity areas over time. Ultimately, the findings of this study will be used to
improve evaluation practices and inform capacity building efforts in museums. In doing so, the field can
reach a fuller understanding of the wide range of effects and outcomes attributable to museum
programs.

References

Bourgeois, I., & Cousins, B. (2013). Understanding dimensions of organizational evaluation capacity.
American Journal of Evaluation, 34(3), 299–319.

Chalas, A. (2016). Toward evaluating art museum education at the Art Gallery of Ontario. Canadian
Review of Art Education: Research and Issues, 43(1) [Special Issue], 60–69. Retrieved from
​http://crae.mcgill.ca/article/view/21

Cousins, J. B., Elliott, C., Amo, C., Bourgeois, I., Chouinard, J., Goh, S. C., & Lahey, R. (2008).
Organizational capacity to do and use evaluation: Results of a pan-Canadian survey of
evaluators.
The Canadian Journal of Program Evaluation, 23(3), 1–35.

Korn, R. (2007). New directions in evaluation. In P. Villeneuve (Ed.), From periphery to center: Art
museum education in the 21st century, (pp. 213–218). Reston, VA: National Art Education
Association.

Nielsen, S. B., Lemire, S., & Skov, M. (2011). Measuring evaluation capacity—Results and implications of
a
Danish study. American Journal of Evaluation, 32(3), 324–344.

Ritchie, J., & Spencer, L. (2002). Qualitative data analysis for applied policy research. In M. A. Huberman
& M. B. Miles (Eds.), The qualitative researcher’s companion, (pp. 305-328). Thousand Oaks, CA:
Sage.

38
 Visitor Studies Association - 2018 Conference Abstracts

Suarez-Balcazar, Y., Taylor-Ritzler, T., García-Iriarte, E., Keys, C., Kinney, L., Rush-Ross, H., …Curtin, G.
(2010). Evaluation Capacity Building: A Cultural and Contextual Framework. In F. Balcazar, Y.
Suarez-Balcazar, T. Taylor-Ritzler, & C. B. Keys (Eds.), Race, culture and disability: Rehabilitation
science and practice, (pp. 307-324). Sudbury, MA: Jones and Bartlett.

Additional Links

http://educ.queensu.ca/students/profiles/chalas

https://www.youtube.com/watch?v=zV6FV6Sbz0c&list=PLww1dvjSoO4PO1a17d5VDea_VDhdCYhck&in
dex=6

Eliciting Teen Voices and Stakeholder Values Using Most Significant Change
Cassandra Solomon-Filer, Shedd Aquarium

Purpose

This paper presentation addresses the use of the participatory method of Most Significant Change (MSC)
to engage teens and other stakeholders for youth informal education programming focusing on
STEM/STEAM and career exploration.
During the presentation I will discuss the qualitative approach of MSC as a potential evaluation tool for
teen programs that are complex or programs that do not have clearly delineated outcomes (Dart &
Davies, 2005). The presentation will center on the following questions:
● What is MSC?
● How is MSC implemented?
● What it the value of MSC with teens? With other stakeholders?
● How can MSC be used for program planning and evaluation?
● What did I learn using MSC for the evaluation of a drop-in teen space?
● What value do I see in using MSC for an informal science education teen program moving
forward?

Abstract

Most Significant Change (MSC) is a participatory evaluation method that originated from the monitoring
and evaluation of international development projects (Dart & Davies, 2003, 2005). MSC involves the
collection of stories from program participants of what each considers to be the most significant change,
positive or negative, they experienced through a program. The technique therefore encourages
authentic collaboration with participants by asking them to define their own program outcomes from
their own systems of values and ranges of program-related experiences. Groups of stakeholders then
discuss and prioritize stories in an effort to agree on a story or stories that best illustrate what they
believe the program should be accomplishing. This participatory coding and analysis of participant voices
is an important element of the technique, as the process leads stakeholders towards a collective
understanding of potential program outcomes valued by both participants and other stakeholders.
I piloted MSC at a teen learning lab structured towards informal education and self-directed learning.
The teen learning lab’s resources focus on aquatic science, technology, art, and the natural world, and

39
 Visitor Studies Association - 2018 Conference Abstracts

the space was conceived as a place for teens to socialize, work on projects, and explore careers. The
teen lab welcomes a diverse audience of teens from Chicago and its surrounding suburbs.
For this pilot study, I collected a total of 15 MSC stories from teens who had visited the teen lab at least
once. Teens submitted stories by responding to questions on Surveymonkey or through semi-structured
interviews. The next step was organizing small groups of stakeholders to discuss what stood out to them
in the stories, with each group ranking stories to narrow the pool for the next group of stakeholders.
At the time of submitting this abstract, I am concluding stakeholder discussions and starting the analysis.
I am approaching MSC from an interpertivist ontology and social constructivist epistemology (Guba &
Lincoln, 1989), and am using NVivo to conduct a thematic analysis from the different stories collected,
with codes and themes that emerge primarily from ongoing stakeholder discussions.

Limitations I have experienced using MSC include the challenge of convincing internal staff to try the
approach, the difficulty of collecting stories from a diverse sample of teens, and then also of finding the
time for teens to read through stories and analyze them. Like many qualitative approaches, the analysis
takes a lot of time.
I plan to conclude this pilot with a brown bag to share results with internal staff, including interested
teens. I hope to continue using MSC to explore how participants’ stories change or stay the same
longitudinally and across teen programs, and to incorporate teen participant evaluators to conduct MSC
interviews while expanding the interviewee sample. The repeated themes of confidence and self-worth
that are arising in the stories of teens who identified as females also suggest the utility of analyzing
stories through a feminist critical lens.

Importance

MSC provides a structure for teens to define their own outcomes and communicate what matters to
them most, allowing evaluators to hear “participants as they are” (Goodyear et al, 2014, p.251). It also
asks teens to reflect on their own learning and assess the how and why behind their decisions and
behaviors, encouraging the further development of critical thinking and reflection skills in participants
and other stakeholders. Participatory analysis brings in stakeholders, including teens that might not have
been included or wanted to participate in the initial story collection, to voice their own values and
viewpoints in a process that will be documented and shared widely with others. It also asks stakeholders
at different institutional levels to engage with teen voices by contextualizing teen stories within their
own value systems. MSC fosters transparency in the evaluation process with its collaborative and
participatory elements, and fosters “learning by all involved” (Goodyear et al, 2014, p.251).

References

Dart, J. and Davies, R. (2003). A Dialogical, story-based evaluation tool: The most significant change
technique. American Journal of Evaluation, 24(2), 137 – 155. Retrieved from
https://doi-org.proxy2.cl.msu.edu/10.1177/109821400302400202

Dart, J. and Davies, R. (2005). The 'Most Significant Change' Technique - A Guide to Its Use, Funded
by CARE International, United KingdomOxfam Community Aid Abroad, Australia | Learning to
Learn, Government of South AustraliaOxfam New Zealand | Christian Aid, United Kingdom |
Exchange, United Kingdom Ibis, Denmark | Mellemfolkeligt Samvirke (MS), Denmark Lutheran
World Relief, United States of America. Retrieved
from ​http://www.mande.co.uk/docs/MSCGuide.pdf

40
 Visitor Studies Association - 2018 Conference Abstracts

Guba, E.G., & Lincoln, Y.S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

Goodyear, L., Jewiss, J., Usinger, J. & Barela, E. (2014). Elements of quality in qualitative evaluation. In
Goodyear, L., Barela, E., & Jewiss, J. (Eds.). Qualitative inquiry in evaluation : from theory to
practice (251 – 276). Retrieved from ​http://ebookcentral.proquest.com​ Created from
michstate-ebooks on 2018-03-15 17:50:52.

Additional Links

Rick Davies describes how to code and analyze MSC stories:

https://www.youtube.com/watch?v=QKK39k2UYss

Download a facilitation guide for using MSC integrated with Photovoice:

http://www.comminit.com/global/content/facilitation-guide-integrated-evaluation-methodology-most-
significant-change-and-photovo

1:45-3:00 PM - Round Tables, Papers, and Fireside Chats

Judy Koke Fireside Chat
Throughout her career as a researcher and museum professional, Dr. Koke has challenged museums to
broaden and deepen their meaningful relationships with audiences. She has acted as a VSA board
member and held leadership roles at the Art Gallery of Ontario and The Nelson-Atkins Art Museum, and
she currently serves as Director of Professional Development at the Institute for Learning Innovation.
Attend this session to participate in a discussion about true collaboration between organizations and
their communities and the changing role of museums today.

Say the Unspoken: Fostering Dialogue for Emerging Professionals

Jason Allen, Project Manager and Evaluator, Eiteljorg Museum of American Indians and Western Art
Taline A. Kuyumjian, Experience Evaluator, Kuyumjian Consulting LLC
Lauren Wilson, Audience and Evaluation Specialist, Illuminated Ideas

Purpose

This roundtable discussion hopes to encourage an open and frank dialogue about the realities of being
an emerging professional in the Visitors Studies field. It can be difficult to encourage transparency with
our communities when we gloss over, and occasionally diminish, the challenges within our own field and
institutions. Through this open conversation, participants will share challenges faced and perspectives
on growing in and influencing our field. Topics like mentorship, networking, imposter syndrome, and
realities of entry-level evaluation positions will lend a critical lense to life as an emerging professional
and Visitors Studies as a field at large.

Abstract

As evaluators, working effectively and responsibly with communities requires transparency. Before
fostering trust and openness with those we serve, it must first be established within our institutions,

41
 Visitor Studies Association - 2018 Conference Abstracts

organizations, and careers. Say the Unspoken will explore ways emerging professionals in Visitor Studies
can chart their career path, underscoring transparency through mentorship and networking, examining
the structure and realities of entry-level evaluation positions, and discussing how involvement in the
field can shape its future.

Guidance through mentorship and networking empowers emerging professionals. During early stages of
a new job role or career path, newcomers may suffer from imposter syndrome, or feeling like a fraud
(Weir, 2013). Building a network helps emerging professionals overcome these fears, both by
establishing interpersonal connections and offering access to resources for continued development.

With emerging professionals as discussion hosts, issues and difficulties faced early in Visitor Studies
careers will be at the forefront of conversation. Often, early stage work involves tasks which may feel
irrelevant to or out-of-line with future career goals. However, work on less fulfilling projects may open
doors to valuable opportunities. Through understanding the current state of our field, emerging
professionals will be better equipped to navigate its challenges. In time, they will usher in a new era of
emergent professionals; it is today’s newcomers who will structure systems that promote equity and
value for our field in the years to come.

Finally, participants will feel empowered to share their own expertise and knowledge back to the field.
Involvement in VSA and other organizations offers a platform for contributing ongoing dialogue
surrounding our field. Established professionals will gain their fresh insights, while emerging
professionals will benefit from the experience and collective knowledge of the industry. Discussion will
encourage participants to capitalize on their unique experience to give back and establish a culture of
collaboration and camaraderie.

Importance

While the field of Visitor Studies readily tackles issues of diversity and inclusion within our visitorships,
we do not always approach systemic issues faced by emerging professionals with the same clarity or
drive. Dedicating space to discuss the realities of building a career in our field encourages open and
direct communication within our institutions around advocating for early stage professionals. In turn,
the ability to address difficult topics of systemic change internally helps institutions to be more
transparent with the communities we serve. With attention to the ways access to our field is granted or
restricted, we will continue to increasingly welcome diverse voices and perspectives into our staffs and
institutions.

References

Weir, K. (2013, November). Feel like a fraud? Retrieved from

http://www.apa.org/gradpsych/2013/11/fraud.aspx

Collective Insight: Strategies for Culturally Responsive Evaluation Capacity

Building
Patricia Montaño, Bridging Communities FIG Co-Chair
Kate Westaby, ¡Milwaukee Evaluation!, University of Wisconsin-Madison
Angie Ong, University of Washington

42
 Visitor Studies Association - 2018 Conference Abstracts

Jill Stein, Independent Consultant

Purpose

Join the Bridging Communities Focused Interest Group (FIG) to learn about building capacity for
culturally responsive evaluation (CRE) among emerging and established evaluation professionals.
Together we will brainstorm strategies for VSA professional development and capacity building in
culturally responsive evaluation, which will be shared with VSA’s Professional Development Work Group
and Board leadership to inform future directions.

In this working group session, presenters will share their reflections on efforts to build capacity for CRE
and diversity within the evaluation field. This will prompt participants to consider various types of
capacity building opportunities so that they can collectively generate ideas for how VSA can further its
own capacity building efforts in the area of CRE. The Bridging Communities FIG Co-Chairs will facilitate
and record the group discussions and lead post-conference actions with interested session participants.

Abstract

VSA’s Evaluator Competencies for Professional Development were developed in 2008 to support visitor
studies professionals in planning their own professional development and to guide the work of VSA in
building the capacity of its members (VSA, 2008). Since then, the American Evaluation Association
published its Statement on Cultural Competence in Evaluation (AEA, 2011) and culturally responsive
evaluation (CRE) (Frierson, Hood, Hughes, and Thomas, 2010) has increasingly gained mainstream
attention. Within VSA, there have been several presentations focused on themes relevant to cultural
competence and CRE in recent years; these have ranged from discussing implications of AEA’s statement
on cultural competence for VSA (Stein, Heimlich, Valdez, Reich, and Garibay, 2012), to pathways into
visitor studies for diverse professionals (Aichele, Benne, Bequette, Huerta Migus, and Valdez, 2013), to
practicing culturally responsive evaluation (Garibay, Garcia-Luis, and Gontan, 2015), to specific projects
and strategies. Furthermore, with the creation of the Bridging Communities FIG within VSA and the shift
from Committees to Work Groups within VSA’s management structure, there are new forums and
mechanisms that can be leveraged in still evolving ways to advocate for and facilitate member capacity
building.

At the beginning of the working group session, panelists will speak about developing CRE within
communities of emerging and professional evaluators. Attendees will be asked to consider what types of
capacity building VSA could provide its members in the area of CRE.

Kate Westaby from ¡Milwaukee Evaluation! and University of Wisconsin-Madison will discuss the work
of ¡Milwaukee Evaluation! (Wisconsin's professional evaluation association), including pipeline work to
foster workforce diversity, intentionally providing space for cultural differences at their hallmark Social
Justice and Evaluation Conference, and results from their recent Culturally Responsive Evaluation
Survey.

Angie Ong will speak about how CRE is incorporated into University of Washington’s Introduction to
Museum Evaluation syllabus to build awareness among emerging museum professionals, regardless of
whether they eventually become evaluators.

43
 Visitor Studies Association - 2018 Conference Abstracts

Jill Stein will share insights from her experience as an audience researcher and evaluator who focuses on
culturally responsive practices, as well as her position as an At-Large Member of VSA’s Board of
Directors.

In working groups, attendees will brainstorm ideas for adding CRE to the association’s list of professional
competencies as well as strategies for building capacity for VSA members. Each group will discuss one of
these topics:

● Skills and competencies for CRE. What are they, and how could they be incorporated into VSA’s
Evaluator Competencies?
● Professional development opportunities. What formats, topics, and entry points should VSA
consider?
● Emerging professionals. How can VSA and universities build emerging professionals’ cultural
competency skills and capacity to do CRE? What is the potential role of mentorship programs?
What support do graduate programs need?
● Organizational change. How can VSA support organizations to be inclusive, diverse, and
culturally competent within their workforce? With community interactions during audience
research and evaluation?

Following the conference, the Bridging Communities FIG will discuss these topics with VSA’s Professional
Development Work Group and other leadership to identify next steps and develop an action plan.

Importance

The conference theme Fostering Transparency, Strengthening Public Trust asks each of us to reflect on
how informal learning institutions can better serve and respect diverse communities. How can we do
better at building capacity and preparing informal learning professionals, particularly visitor studies
professionals, to understand and engage with various communities in ways that foster trust and support
culturally relevant processes and outcomes? The interest in CRE and cultural competency has grown
within VSA and it is time to act not as individuals, or as a FIG, but as a collective association. Through
learning from strategies and findings from ¡Milwaukee Evaluation!, higher education, and experiences of
VSA members, the collective insight from participants will illuminate a path forward for how VSA can
address capacity building regarding diversity, inclusion, and culturally responsive evaluation within
visitor studies.

References

Aichele, Benne, Bequette, Huerta Migus, and Valdez. (2013). Creating pathways for diversity in visitor
studies professions. Visitor Studies Association Conference: Milwaukee, WI.

American Evaluation Association. (2011). Public statement on cultural competence in evaluation.

Retrieved from ​http://www.eval.org/aea.culutrally.competent.evaluation.statement.pdf

Evaluator Competencies for Professional Development. (2008). Visitor Studies Association. Retrieved
from
https://visa.memberclicks.net/assets/docs/historical/eval_comp/evaluator-competencies.pdf

Frierson, H. T., Hood, S., Hughes, G. B., & Thomas, V. G. (2010). Chapter 7: A guide to conducting

44
 Visitor Studies Association - 2018 Conference Abstracts

culturally responsive evaluations. In Joy Frechtling (Ed.), The 2010 user-friendly handbook for
project evaluation (Revision to NSF Publication No. 02-057) (pp. 75-96). Arlington, VA: National
Science Foundation. Retrieved from ​http://www.westat.com/pdf/projects/2010ufhb.pdf

Garibay, Garcia-Luis, Gontan. (2015). Point of No Arrival: Cultivating culturally responsive/contextually

relevant research & evaluation practices. Visitor Studies Association Conference: Indianapolis,
IN.

Stein, Heimlich, Valdez, Reich, and Garibay. (2012). Forum: Applications of AEA’s statement on cultural
relevance in evaluation statement to VSA. Visitor Studies Association Conference: Raleigh, NC.

Talking About Teens: Evaluating Museum Programs that Target Teens

Jennifer Borland, Rockman et. al.
Preeti Gupta, American Museum of Natural History
Alison Mroczkowski, Museum of Science and Industry

Purpose

The vast majority of out-of-school time programs target youth ages twelve and younger. Similarly,
museum programming tends to focus on young children and adults, and has sometimes struggled to
attract and engage teen audiences. In this Roundtable discussion, we will talk about the challenges and
benefits of programs designed for teens and the unique issues and opportunities that go along with
evaluating these programs.

Abstract

This roundtable discussion is designed to explore the challenges and benefits of programs designed for
teens—including the unique issues and opportunities that go along with evaluating these programs.

Specific Outcomes and Objectives:

● Attendees will learn more about different types of programs being offered to teens in museum
settings
● Attendees will gain insights into best practices for evaluating teen programs
● Attendees will come to understand key differences between evaluating teen audiences in
comparison/contrast to other audiences

Proposed Method:

Our planned discussion will include a brief introduction of the co-presenters, wherein each will discuss
teen programming offered at or studied by their institutions—including Teen SciCafes and other
teen-oriented programming at the American Museum of Natural History and the long-running Science
Minors and Science Achievers program for teens at the Museum of Science and Industry. Roundtable
participants will also be invited to share examples of the work they are doing with teens as well. A series
of targeted discussion questions will be posed in order to explore strengths and weaknesses of programs
geared toward teen audiences and best practices (and/or potential pitfalls to avoid) when evaluating
teen programs.

45
 Visitor Studies Association - 2018 Conference Abstracts

Why the closer look at teen programming specifically?

Teen audiences are similar to adult audiences in many ways—including their ability to grasp more
advanced concepts and their verbal abilities to communicate information about their experiences.
However, there are unique cultural differences that should be taken into consideration by evaluators
and researchers seeking to work with teen audiences—e.g., social norms within groups of teens and
between teens and adult program facilitators, and different communication and technology
preferences.

Importance

Since the teen years are an important period in youth development in terms of emerging skills and
self-identity formation related to career interest and exploration, it is especially important for us to
consider programs aimed at youth in this age-group and seek to find ways to effectively study and
communicate findings about their impacts.

References

Borland, J. (2017) Human Health, Biodiversity, and Microbial Ecology: Strategies to Educate, Year 3
Evaluation Report. Available on request from [email protected].

Cole, S. E. (2012). The development of science identity: An evaluation of youth development programs
at
the Museum of Science and Industry, Chicago (Doctoral dissertation, Loyola University Chicago).

Falk, J. H., Semmel, M., Goldman, K. H., Ashooh, W., Beach, J. H., Brice, D., ... & Race, M. (2015). The
Science Learning Ecosystem. In Proceedings from Public Libraries & STEM–A National
Conference
on Current Trends and Future Directions: Denver, CO.

Golubchick, L. (2017) Teen SciCafe, Evaluation, and the Human Microbiome Presentation at the
Association of Science and Technology Centers Annual Conference. October 21, 2017.

Hall, M., & Mayhew, M. A. (2015, December). Teen Science Cafés: A Vehicle for Scientists Seeking
Broader Impacts. In AGU Fall Meeting Abstracts.

Hall, M., & Mayhew, M. A. (2017, December). Teen Science Cafés: A Model for Addressing Broader
Impacts, Diversity, and Recruitment. In AGU Fall Meeting Abstracts.

Price, C. A., & Kares, F. R. (2016). Researching Long-Term Impacts of an Out-Of-School Time Program.
Dimensions, July 2016.

Price, C. A., Kares, F., & Segovia, G. Gender differences in STEM career and educational choices of alumni
of an urban, museum-based after-school program. Transforming Learning, Empowering
Learners.

46
 Visitor Studies Association - 2018 Conference Abstracts

Price, C. A., Kares, F., Segovia, G., & Loyd, A. B. (2018). Staff matter: Gender differences in science,
technology, engineering or math (STEM) career interest development in adolescent youth.
Applied Developmental Science, 1-16.

Additional Links

AMNH Teen SciCafes: ​https://www.amnh.org/learn-teach/grades-9-12/teen-programs/teen-scicafe/

MSI Science Minors and Achievers:

https://www.msichicago.org/education/out-of-school-time/science-minors/

Teen SciCafe Network: ​https://teensciencecafe.org/

Big Questions About Food Security - The BigPicnic Project

Theano Moussouri, University College London (UCL)

Purpose

This presentation discusses how BigPicnic, a multi-actor collaboration and engagement EC-funded
project, provides a platform for dialogue about responsible research and innovation (RRI) on food
security. BigPicnic brings together the public, scientists, policy-makers and industry to help tackle the
global challenge of food security. The project aims to do this through the creation of a series of
exhibitions, science cafes and participatory events held and run by 14 botanic gardens spanning twelve
countries across Europe and one in Uganda. Co-creation, action research and organisation research are
at the heart of the project’s collaborative approach which brings botanic garden practitioners in
dialogue with local people as a way to build greater understanding of food security issues. Participants
will be involved in thought-provoking, reflective conversations about the use of co-creation approaches
together with participatory approaches to evaluation and research to tackle more complex questions
and engage local communities in global issues.

Abstract

As social and scientific institutions, botanic gardens hold the expertise to inform, raise awareness of, and
engage the publics with global environmental matters. Food security is one of the greatest challenges
facing society today and it is a concern not only at policy level but also for the general public. According
to the Special Eurobarometer 389 (European Commission, 2012) EU citizens are concerned that
sufficient food is produced to meet the needs of the world’s population and suggest that the EU should
help other countries to produce more food, should produce more food to reduce its dependence on
imports and to meet the rising demand in the EU and elsewhere. Research and innovation can be part of
the solution to the food security challenge and the EU citizens should be given the opportunity to
express their ideas and opinions on which direction these can be developed.
The BigPicnic project aims to create this opportunity at a large scale across Europe.

By acting as mediators, botanic gardens and other institutions taking part in BigPicninc facilitate
dialogue and develop a mutual understanding among researchers and visitors by engaging them in
co-creation activities. Both the co-creation process and the outcomes of the co-creation activities and
other events and exhibitions developed as part of BigPicnic are evaluated using Teams-Based Inquiry

47
 Visitor Studies Association - 2018 Conference Abstracts

(TBI). As a form of action evaluation, TBI gives professionals the opportunity to also reflect on the
process and the impact of the co-creation activity as well as the practice that went into its development.
This collaborative approach aims to give a voice to adults and young people on Responsible Research
and Innovation (RRI), communicating their views to policy-makers, sharing ideas, and encouraging
debate on the future of our food.

This panel presentation highlights the importance of using co-creation approaches together with
participatory approaches to evaluation and research to tackle more complex questions. This approach
helps understand the nuances of key issues around food security in their local context. One of the key
themes that has emerged so far is the societal and cultural significance of food. The idea of food as
heritage goes at the heart of people’s personal and collective identities, any policies around food and
food systems need to take this into account. This is a parameter that is omitted by the prevalent
European and global policies that deal with food and sustainable development but is strongly linked with
the growing awareness and recognition of intangible cultural heritage by UNESCO. The presentation will
also link the particularities of the Garden’s individual projects with the wider research themes provided
both by the EU, but also with the UN Sustainable Development Goals. For example, key findings which
directly link to Food 2030 policy priorities and Sustainable Development Goals include safe food,
community empowerment and gender issues, nutrition and combating hunger, food education, and
sustainable food production and consumption.

This account aims to inspire attendees to think through the challenges and considerations for this type
of participatory multi-organisation project. Co-creation case studies, and participatory approaches to
research and evaluation employed by different botanic gardens will be used as examples to discuss what
has worked and what has failed and why.

Importance

This panel​ ​session is closely connected to the themes of fostering transparency and strengthening public
trust by showing how botanic gardens use participatory approaches to programme and exhibition
development, to research and evaluation to foster dialogue with their communities and European
citizens in general. It directly promotes two VSA learning competencies, namely “Knowledge of and
Practices with Social Science Research and Evaluation Methods and Analysis” and “Professional
Commitment”.

References

Engeström, Y. (2010) Expansive Learning at Work: Toward an activity theoretical

reconceptualization, Journal of Education and Work, 14:1, 133-156.

Engestrom, Y. (2010) Activity theory as a framework for analyzing and redesigning

work,Ergonomics, 43:7, 960-974,

Engeström, Y. , Miettinen, R. and Punamaki, R.-L., eds (1999) Perspectives on Activity

Theory. Cambridge: Cambridge University Press.

BigPicnic Project website: ​https://www.bigpicnic.net/

Food 2030 website:

48
 Visitor Studies Association - 2018 Conference Abstracts

http://ec.europa.eu/research/bioeconomy/index.cfm?pg=policy&lib=food2030

Responsible Research and Innovation ​http://www.ucl.ac.uk/sts/research/projects/rri

Sustainable Development Goals website: ​https://unstats.un.org/sdgs

Team-Based Inquiry ​http://www.nisenet.org/catalog/team-based-inquiry-guide

UNESCO, Convention for the Safeguarding of the Intangible Cultural Heritage:

https://ich.unesco.org/en/convention

Examples:
● Safeguarding traditional foodways of two communities in Kenya
https://ich.unesco.org/en/projects/safeguarding-traditional-foodways-of-two-communi
ties-in-kenya-00176
● Mediterranean diet ​https://ich.unesco.org/en/RL/mediterranean-diet-00884

Exploring Empathy: Lessons from Challenging Exhibitions

Kirsten Madsen, National September 11 Memorial & Museum

Silvina Fernandez-Duque, United States Holocaust Memorial Museum
Juli Goss, Crystal Bridges Museum of American Art
Dr. Rebekah Sobel, United States Holocaust Memorial Museum (facilitator)

Purpose

In this session, panelists will share their experiences with conducting evaluations that center visitor
empathy and discuss the challenges of capturing empathy in museum settings. Participants will walk
away understanding how empathy has been defined, successful and unsuccessful tools for
measurement, and how these conversations may live on within organizations.

Abstract

Panelists from the National September 11 Memorial Museum, the United States Holocaust Memorial
Museum, and Crystal Bridges Museum of American Art will discuss their experiences with tackling
questions about empathy-building through evaluation. The discussion will focus on exhibitions and
exhibition components relating to current issues, including undocumented migration, the Syrian refugee
crisis, and 9/11-related health effects.

Silvina Fernandez-Duque (US Holocaust Memorial Museum) will discuss evaluation work done in
conjunction with a virtual reality (VR) experience featuring the story of a Syrian refugee. A short 360
documentary film called For My Son was shown on a VR headset as part of an exhibition about the crisis
in Syria. The evaluation of the program looked at whether the film and the immersive technology of VR
had an influence on empathy or learning. Silvina will share the results of the evaluation and some of the
challenges in trying to measure empathy.

49
 Visitor Studies Association - 2018 Conference Abstracts

Juli Goss (Crystal Bridges) will discuss findings from an exhibition evaluation as well as the resulting
institutional conversations that have come about since. Border Cantos was an exhibition which sought
to humanize the experience of undocumented immigrants crossing the border between the U.S. and
Mexico. Juli will provide background and definitions on how the concept of empathy functioned in an art
museum environment through this exhibition and beyond, and how her museum worked with
community groups and stakeholders to present the exhibition and foster empathy.

Kirsten Madsen (9/11 Memorial Museum) will focus on evaluation work relating to visitors’ awareness
of 9/11-related illness, which affects thousands of 9/11 survivors and first responders. She will reflect on
how the unique nature of the 9/11 Memorial Museum as a memorial at the authentic site of the 9/11
attacks impacts visitor experience and how evaluation at the institution has evolved to meet these
challenges. Because most visitors to the museum have memories of September 11, 2001, she will also
explore how visitors’ lived experience might affect their ability to connect with others’ stories and
experiences.

Based on their shared experiences, Silvina, Juli, and Kirsten will discuss strategies for considering visitor
emotion, opinion, and empathy in the course of museum evaluation. Questions under consideration
include: Is it possible to know how deeply an experience has affected someone’s understanding if
they’re overwhelmed or disassociated when intercepted? Can we formulate questions that help visitors
process their experience enough to reflect? Is evaluating in the moment ultimately unproductive? How
does this commitment to empathy live on in an organization beyond one project?

Importance

The concept of empathy is often discussed but rarely approached through an evaluative lens. Through
discussion of the panelists’ projects, we hope to have a meaningful discussion about why producing
empathy among museum visitors is a worthwhile goal, and how museum professionals can begin to
consider empathy as a visitor outcome in evaluation work. In discussing the outcomes of our projects
and our plans for future empathy-centered evaluation, we will begin to identify strengths and
weaknesses of existing evaluative tools and explore areas for growth in the field. We hope to give our
colleagues, especially those working with difficult history, the confidence to start conversations with
their colleagues about what visitor empathy looks like at their institution and how they can deploy
evaluation to determine whether their programs and exhibitions are successful in guiding visitors
toward empathy.

References

Gubkin, L. (2015). From Empathetic Understanding to Engaged Witnessing: Encountering Trauma in the
Holocaust Classroom. Teaching Theology and Religion, 18, 103-120.
https://onlinelibrary.wiley.com/doi/pdf/10.1111/teth.12273

Gehlbach, H., Brinkworth, M.E., & Wang, M.-T. (2012). The social perspective taking process: What
motivates individuals to take another’s perspective? Teachers College Record, 114(1), 197-225.

Reniers, R.L.E.P., Corcoran, R. Drake, R. Shryane, N.M., Vollm, B.A. (2011). The QCAE: a questionnaire of
cognitive and affective empathy. Journal of Personality Assessment, 93, 84-95.

Additional Links

50
 Visitor Studies Association - 2018 Conference Abstracts

The Border Cantos evaluation report can be found here:

https://crystalbridges.org/reports-and-research/

4:00-5:15 PM - Concurrent Sessions

Engaging Undergraduate Students in Audience Research
Fran Mast, Research and Evaluation Associate, Research and Evaluation, John G. Shedd Aquarium
Jennifer Matiasek, Research Program Coordinator, Chicago Zoological Society/Brookfield Zoo
Nadya J. Bennett, Audience Research and Interpretive Content Specialist, Columbus Zoo and Aquarium
Nicole Rivera, Assistant Professor of Psychology, North Central College

Purpose

Improved understanding of the strategies cultural institutions can employ to build the competence and
confidence of undergraduate students in audience research practices

Increased awareness of the ways cultural institutions can work with colleges and universities to offer
audience research experiences to undergraduate students

Increased awareness of strategies to integrate undergraduate students into workflow to support project
planning and resource management

Abstract

This session will explore structures and strategies for working with undergraduate students and
college/university partners to engage students in audience research. The presenters will share examples
from their own institutions, discuss methods for recruiting students with diverse academic backgrounds
and experience, and present perspectives on developing and maintaining relationships with colleges and
universities.

Undergraduate research experiences have many benefits for the participating students as well as the
institutions that offer research opportunities (Lopatto, 2009). Students demonstrate gains such as
personal development, growth of new skills, preparation for scientific careers, increased interest in
graduate work, and more positive perceptions of research (Crowe & Brakke, 2008; Lopatto, 2009;
Russell, Hancock, & McCullough, 2009).

The Research and Evaluation department at Shedd Aquarium has three different positions that allow
undergraduate students to build research and evaluation skills and participate in social science research.
These opportunities include volunteer positions for those wanting to develop applied skills such as data
collection and entry, internships that allow students to lead one project across all evaluation phases,
and part-time staff who assistant and coordinate day-to-day efforts of the department. Shedd’s
presentation will discuss the challenges and rewards of this multifaceted model in relation to student
recruitment, engagement and learning and institutional capacity building.

51
 Visitor Studies Association - 2018 Conference Abstracts

The Chicago Zoological Society’s Audience Research department recruits research assistants each
summer to conduct audience research at Brookfield Zoo. This team is typically made up of students from
a range of backgrounds who collect and manage data to address needs identified by the zoo. More
self-directed audience research is conducted by students as part of the zoo’s internship program. This
presentation will share approaches, successes, and challenges related to recruiting students and
creating learning opportunities as part of applied research projects and beyond.

The Audience Research Team at the Columbus Zoo and Aquarium offers seasonal internship
opportunities to undergraduate and graduate students from diverse fields of study. The program
provides interns with the opportunity to take an active role in the evaluation process from project
development and planning to writing reports and presenting results to stakeholders. This presentation
will highlight the structure of the program, how challenges turned into successes, and examples of
projects interns completed over the last three years.

Students at North Central College have opportunities to participate in a variety of informal course
experiences, internships, and engagement in audience research and evaluation projects with cultural
institutions. Informal course experiences occur through the college’s Verandah program and includes a
week-long class in which students visit five different cultural institutions to work with research and
evaluation teams. Students participate in research and evaluation projects as volunteers and/or to earn
course credit for a research practicum or internship.

Following the presentations, the panel will invite audience members to take part in a discussion of
challenges and benefits in working with undergraduate students. Finally, the group will consider
traditional and novel ways our institutions can maximize the learning potential of these types of
research experiences to stimulate new questions, ideas, and relationships.

Importance

From an institution’s perspective, having undergraduate students contribute to research can support
workflow and project management, contribute to the field of science by developing future professionals,
and provide research projects with enthusiasm and new perspectives. Additionally, grant projects that
engage undergraduate opportunities and academic partners may be stronger for funding consideration.
As museums and cultural institutions endeavor to grow capacity for research and evaluation,
undergraduate students are an important resource to support current practices and develop a pipeline
of future professionals. The John Shedd Aquarium, the Chicago Zoological Society, Columbus Zoo, and
North Central College each work with undergraduate students to offer opportunities to experience
applied audience research and help students to develop research skills. Staff from these institutions will
share their approaches to engaging undergraduate students and discuss successes and challenges they
encounter in their work and share ideas for fostering partnerships between cultural institutions and
universities/ colleges.

References

Johnson, W.B., Behling, L., Miller, P., & Vandermaas-Peeler, M., (2015): Undergraduate research
mentoring: Obstacles and opportunities. Mentoring & Tutoring: Partnership in Learning, DOI:
10.1080/13611267.2015.1126167

Crowe, M. & Brakke, D. (2008). Assessing the impact of undergraduate-research experiences on

52
 Visitor Studies Association - 2018 Conference Abstracts

students: An overview of current literature. CUR Quarterly, 28(4), 43-50. Available online at
http://www.cur.org/assets/1/7/summer08CroweBrakke.pdf

Lopatto, D. (2009) Science in solution: The impact of undergraduate research on student learning.
Tucson, AZ: Research Corporation for Science Advancement. Available online at
http://web.grinnell.edu/sureiii/Science_in_Solution_Lopatto.pdf

National Science Foundation. (2000). NSF GPRA Strategic Plan, FY 2002–2006. (NSF Publications 0104).
Retrieved from ​https://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf0104

Russell, S.H., Hancock, M.P., & McCullough, J. (2007). Benefits of undergraduate research experiences.
Science, 316(5824), 548-549. Available online at
http://www.sciencemag.org/content/316/5824/548.summary
 
Digital Experiences in Art Museums: Visitor Preferences, Participation, and
Impacts
Laura Brown, Manager, Evaluation & Visitor Research, The Nelson-Atkins Museum of Art
Shiralee Hudson Hill, Lead Interpretive Planner, Art Gallery of Ontario
Hannah Ridenour, Research Specialist, Cleveland Museum of Art
Maia Werner-Avidon, Principal, MWA Insights

Purpose

Using case studies from three art museums, this session will explore what recent research has revealed
about the incorporation of digital experiences in art museums. Presentations will focus on visitors’
preferences for analog versus digital experiences, digital engagement, and impacts of digital experiences
ranging from apps to augmented reality. During the session, participants and panelists will reflect on
best practices for incorporating digital experiences into art museum galleries, and participants will learn
about new approaches to measuring the impact(s) of digital experiences that have the potential to be
adopted by other museums.

Abstract

In recent years, art museums have increasingly embraced the inclusion of digital experiences inside their
galleries, with the hope that these experiences will attract new audiences, change visitors’ perceptions
of the museum, and offer new and engaging ways of interacting with art (see, for example, Cannell,
2015; Gamerman, 2015, and Lohr, 2014) . But how well do these digital experiences mesh with visitors’
expectations and preferences for their museum experience? To what extent are visitors choosing to
engage with technology? What impact does engaging with digital technologies ultimately have on the
visitor experience? And are there certain types of digital experiences that are more impactful than
others? This session will include case studies of three recent research studies that sought to answer
some of these questions.

Case Study 1: Print or pixel: Comparing visitor use and preferences for analog versus digital interpretive
components at The Nelson-Atkins Museum of Art

53
 Visitor Studies Association - 2018 Conference Abstracts

The Nelson-Atkins Museum of Art recently completed a renovation of its 19th- and early-20th-century
European art galleries (the Bloch Galleries). The renovation incorporated new visitor engagement and
interpretation strategies, including several digital components. Summative evaluation, conducted in
collaboration with MWA Insights, examined visitors’ use and depth of engagement with both digital and
analog interpretation and captured visitors’ opinions about the inclusion of digital experiences and
preferences for ways of getting information.

Case Study 2: Transporting you into the art: Visitors’ responses to augmented reality at the Art Gallery of
Ontario
In 2016, the AGO partnered with Toronto-based digital artist Alex Mayhew to realize ReBlink, an
exhibition and augmented reality (AR) digital project, which allowed visitors to download an app to see
an animated “remix” of specific paintings. The summative evaluation sought to discover who was using
the app, to what extent visitors were engaging with the technology as well as the project’s thesis about
change and contemporary life, what visitors found captivating about this experience, and whether the
AR experience encouraged close-looking at the artwork. This case study will explore how we can use our
research to think further about enhancing the visitor experience and connecting art, ideas and visitors
through AR.

Case Study 3: Defining Metrics and Discerning Change: Evaluating the impact of interactive technology at
the Cleveland Museum Art
The Cleveland Museum of Art was awarded a grant from the National Endowment for the Arts to better
understand the value of visitor engagement with digital interactives and the metrics that best measure
this value. The study, designed and implemented in collaboration with Rockman et al, is examining ways
in which the museum’s recently reopened interactive space, the ARTLENS Gallery, positively impacts the
visitor experience, and which metrics tested in the study appear to best measure the impact of
interactive technology in any art museum.

Importance

The featured case studies will provide takeaways related to best practices in digital use in art museums
and how to measure its impact. These studies have garnered new insights about the value of in-gallery
digital experiences, including reported preferences for printed over digital components and visitor
reactions when technology can achieve experiences not possible in typical viewing experiences. They
have explored the relationship of location and experience, and will reflect on the appropriate tone, type
of experience, and amount of content to provide in specific locations. They have worked to understand
the results of an interactive intervention against a traditional museum experience across various visitor
groups, and defined the metrics to measure its impacts and value. The panelists will share the ways in
which museums can assess digital impact even as technology continues to evolve and museums
continue to embrace new interpretive approaches.

References

Cannell, M. (2015, March 17). Museums Turn to Technology to Boost Attendance by Millenials. New
York
Times. Retrieved from
http://www.nytimes.com/2015/03/19/arts/artsspecial/museums-turn-to-technology-to-boost-a
ttendance-by-millennials.html

54
 Visitor Studies Association - 2018 Conference Abstracts

Ciecko, Brendan. (2018, March 27). Museums are the best place to find innovation in AR. Retrieved from
https://venturebeat.com/2018/03/27/museums-are-the-best-place-to-find-innovation-in-ar/

Gamerman, E. (2015, October 16). A Look at the Museum of the Future. Wall Street
Journal. Retrieved from
http://www.wsj.com/articles/a-look-at-the-museum-of-the-future-1444940447

Helal, D., Maxon, H., & Ancelet, J. (2013). Lessons learned: Evaluating the Whitney’s multimedia guide.
In
Museums and the Web 2013, N. Proctor & R. Cherry (eds). Silver Spring, MD. Museums and the
Web. Retrieved from
https://mw2013.museumsandtheweb.com/paper/lessons-learned-evaluating-the-whitneys-mul
timedia-guide/

Katz, Miranda. (2018, April 23). Augmented Reality is Transforming Museums. Retrieved from
https://www.wired.com/story/augmented-reality-art-museums/

Lohr, S. (2014, Oct. 23). Museums Morph Digitally. New York

Times. Retrieved from
http://www.nytimes.com/2014/10/26/arts/artsspecial/the-met-and-other-museums-adapt-to-t
he-digital-age.html

Mudhar, Raju (2017, July 6). The AGO puts the AR in art with ReBlink exhibit. Retrieved from
https://www.thestar.com/entertainment/2017/07/06/the-ago-puts-the-ar-in-art-with-reblink-e
xhibit.html

Ricci, Talia. (2018, Jan. 3). AGO's ReBlink exhibit combines augmented reality and classic paintings.
Retrieved from
http://www.cbc.ca/news/canada/toronto/art-exhibit-combines-augmented-reality-and-classic-p
aintings-1.4452158

Sternbergh, M. C., Fantoni, S. F., & Dejen, V. (2015). What's the point? Two case studies of
introducing digital in-gallery experiences. MW2015: Museums and the Web 2015. Retrieved
from
https://mw2015.museumsandtheweb.com/paper/whats-the-point-two-case-studies-of-introduc
ing-digital-in-gallery-experiences/

Additional Links

Nelson-Atkins Museum of Art Bloch Galleries Summative Evaluation Report

http://informalscience.org/nelson-atkins-museum-art-bloch-galleries-summative-evaluation
 
​Bringing Our Values to Work

Cecilia Garibay, Garibay Group

Kenneth Morris, Detroit Institute of Arts
Jeanine Pollard, Minneapolis Institute of Art
Kate Livingston, ExposeYourMuseum

55
 Visitor Studies Association - 2018 Conference Abstracts

Purpose

How do our values compel us? What are the risks and rewards of “living our values?” What happens
when personal values and museum values don't align? In this participatory workshop, attendees assess
their core values and explore ways to infuse them deeply and meaningfully into their museum and
visitor studies work. The purpose of this session is to identify and explore our personal values,
investigate how, when, and why they show up at work, and reflect on the importance of values within
museums, visitor studies, and audience research and evaluation. Participants will 1) Identify individual
“core values,” 2) Assess if and how their values align with or are different from the values of their
organization and/or work, 3) Learn the “value of values” within visitor studies, especially in terms of
increasing transparency, moving individuals to action, and taking steps toward institutional bravery.

Abstract

The session starts with a participatory activity to help attendees determine their top values and assess
if/how those values show up at work. Following the guided activity, session presenters will share their
top values and how they present in their work. Each presenter will reflect on the “value of values” within
museums and visitor studies, especially in terms of being transparent, bold, brave, and moving from talk
to action:

1. Cecilia Garibay will share how her values led her toward culturally responsive approaches to research
and evaluation and advocating for equity and inclusion in the museum field. Although working with
clients and colleagues toward deep change can be rewarding, raising issues of power dynamics,
privilege, and ways our own practices can exacerbate inequity, can also be draining and lonely work.

2. Ken Morris will share how his values translate into his work at the Detroit Institute of Arts. This means
sometimes being the voice in the room no one wants to hear. It also sometimes involves reminding the
organization of its stated values a pushing it into uncomfortable roles and situations.

3. Jeanine Pollard, a VSA 2017 student scholarship recipient, will share how her values have influenced
her work in interpretive planning for visitors with disabilities at the Museum of Contemporary Art,
Chicago and how they led to her new position as Research and Project Manager at Mia’s Center for
Empathy and the Visual Arts.

4. Kate Livingston will share how her values led her to start Fund the Change, supporting marginalized
activists in museums, and to pursue certification in leadership coaching. From a consultant’s
perspective, she runs the risk of upsetting clients whose values don’t align with her own. On the other
hand, being upfront with her values filters out projects that aren’t well aligned with her passions.

Following presenters’ stories, they will share the collective values in the room by showing a data
visualization of the most frequently occurring values submitted by attendees. Panelists will also share
common “core values” of museums to elicit comparison. This will lead into an open discussion and Q&A,
which may include: 1) How do you know what your workplace values? 2) Are the values our workplaces
profess the same as the values they enact? 3) How do museums’ and museum workers’ values align?
What do we do when they don’t? 4) What might be possible for museums and for visitor studies, given
the collective values that have been shared in the room? 5) How can we bring our values into our work?
6) What are the risks and rewards?

56
 Visitor Studies Association - 2018 Conference Abstracts

Importance

As many of the references shared in this abstract confirm, when personal and workplace values are not
aligned it can cause incredible discomfort, frustration, and disenchantment. Sometimes, it leads to
quitting a job or leaving a chosen field altogether. In recent years, movements around values in
museums (and values in museums as workplaces) have gained traction and attention, from “Museum
Workers Speak” to #MuseumsAreNotNeutral. It is clear that our values inform not only our individual
thoughts and actions, but can lead to collective and systemic shifts. This session recognizes the
importance of first gaining clarity on what we each value as individuals so that we can more
authentically engage in conversations related to the 2018 Visitor Studies Association conference theme,
“Fostering Transparency, Strengthening Public Trust.”

References

Inside Higher Ed (2017): Do Your Career and Work Values Align?

https://www.insidehighered.com/advice/2017/11/13/importance-aligning-your-career-your-cor
e-values-essay

Psychology Today (2012): Personal Growth: How to Align Your Values and Your Life
https://www.psychologytoday.com/us/blog/the-power-prime/201205/personal-growth-how-ali
gn-your-values-and-your-life

Fast Company (2015): How to Find a Job that Aligns with Your Values
https://www.fastcompany.com/3053241/how-to-find-a-job-that-aligns-with-your-values

Forbes (2016): If Your Values Clash With How You're Working, You'll Suffer -- Here's How To Fix That:
https://www.forbes.com/sites/kathycaprino/2016/08/04/if-your-values-clash-with-how-youre-
working-youll-suffer-heres-how-to-fix-that/2/#7aa35966b30b

Fast Company (2014): 5 Reasons You Need to Instill Values in Your Organization:
https://www.fastcompany.com/3028201/5-reasons-you-need-to-instill-values-in-your-organizat
ion

Harvard Business Review (2014): Does Your Company Make You a Better Person?
https://hbr.org/2014/01/does-your-company-make-you-a-better-person

Harvard Business Review (2018): How to Establish Values on a Small Team:

https://hbr.org/2018/04/how-to-establish-values-on-a-small-team

Harvard Business Review (2015): Manage Stress by Knowing What You Value:
https://hbr.org/2015/09/manage-stress-by-knowing-what-you-value

Additional Links
 
Museum Workers Speak:
https://museumworkersspeak.weebly.com/

57
 Visitor Studies Association - 2018 Conference Abstracts

#museumsarenotneutral:
https://artstuffmatters.wordpress.com/
https://artmuseumteaching.com/2017/08/31/museums-are-not-neutral/
 
Who's Not Here? Defining Audiences, Broadening Methods, Changing
Relationships
Leigh Ann Mesiti - Research and Evaluation Coordinator, Museum of Science, Boston
Sarah May - Research and Evaluation Associate, Museum of Science, Boston
Juli Goss - Research and Evaluation Manager, Crystal Bridges Museum of American Art
Maddie Smith - Researcher, Slover Linett Audience Research

Purpose

Museums and cultural institutions grapple with issues related to “who’s not here?” and research and
evaluation professionals are increasingly addressing this question (Baach, Giron Mathern, Baltazar, &
McManimon, 2016). Recent work in this area shows that museums have considered characteristics such
as multilingualism (Garibay et al., 2015), income (Dawson, 2014; Dilenschneider, 2016), and diversity
and inclusion more broadly (Holmes, 2017; Cole & Kashan, n.d.) to ground their thinking and strategies
in reaching audiences. In this session, evaluation professionals from institutions at different points in
this work share their own insights and emergent challenges related to “broadening audiences.” Major
points of discussion include concrete research and evaluation strategies to engage potential visitors, as
well as how audience engagement practices influence, and are influenced by, research and evaluation.
Additionally, this session reflects on ways in which relationships between evaluators and different
stakeholder groups vary by project and institutional missions.

Abstract

Panel members from the Museum of Science, Boston (MOS), Crystal Bridges Museum of American Art,
and Slover Linett Audience Research will each share unique institutional perspectives on practices and
lessons learned from audience engagement projects centered on the question “Who’s not here?”

The MOS has a long history of engaging audiences with disabilities, and are now in early stages of
further broadening its audiences by redefining approaches to community engagement that move
beyond the “transactional” (Bowen et. al., 2010). MOS panelists will discuss how the institution is
beginning to more concretely define what it means to be “underrepresented in STEM learning,” and
how new approaches to community engagement have led to using targeted audience surveys and focus
groups, as well as discussions with community leaders to better understand potential audience
perspectives on exhibitions and the institution as a whole. Organizational change is taking place, as
community engagement teams are rethinking how to integrate research and evaluation into their
approaches, just as the research and evaluation team is exploring new methods for reaching audiences
of interest.

As a very young museum, Crystal Bridges is in the process of learning about all audiences visiting and
how the museum’s audience reflects the local community. Recently, the museum identified local
Hispanic/Latino communities as an audience of interest because it is the largest minority group and
fastest growing community in the region. This session will share the work of the organization's one

58
 Visitor Studies Association - 2018 Conference Abstracts

internal evaluator, her systematic approach to audience surveys exploring demographic differences
between general attendance and specific programming, and how she engaged local racial and ethnic
communities specifically by integrating their voices into front-end evaluations across multiple
exhibitions. An interest in seeking visitor feedback has spread across the organization and others have
begun to adopt data collection methods to shape their practices and relationships with these
communities.

Slover Linett Audience Research is a social research and evaluation firm for the cultural sector, broadly
defined. Through quantitative, qualitative, and ethnographic methods, Slover Linett helps cultural
institutions of all kinds understand their participants and communities, as well as helps connect
institutions with people not currently visiting/attending. As a part of this panel, Slover Linett will share
research they have done with a performing arts organization to qualitatively and quantitatively
understand how the institution is viewed and valued among those who don’t currently attend it’s
programming. The findings from this research will inform strategic decisions within the organization
spanning marketing, outreach, and programming to better reach a wider ribbon of the organization’s
community.

Different institutions have their own goals, resources, and structures for engaging new audiences. These
three perspectives present varied examples that aim to showcase potential approaches to starting or
advancing work related to audience development, as well as identify challenges that have confronted
institutions in the process. In each case, approaches to broadening audiences and community
engagement has affected practices spanning across the organization.

Importance

Cultural institutions, such as museums, were intended to serve as educational and experiential
resources for their communities. Today they are not exclusively providers of knowledge, but also
hands-on adventure spaces, social spaces for families and friends to convene, and scribes of history. In
order to uphold the ideal that cultural institutions serve their broader communities, research is needed
to understand the perspectives, perceptions, and values of those who are not typically reached in
audience engagement initiatives. This session has implications for future directions in participants’ and
session panelists’ practices related to research, evaluation, and community engagement, specifically
related to: 1) how to get started with this work, 2) specific strategies that others have used in defining
audiences, 3) the role of evaluation and evaluation methods in audience development, and 4) how
relationships form and shift over time in both internal and external contexts.

References

Argyris, C., & Schon, D. (1995). Organizational learning II: Theory, method, and practice: Prentice Hall.

Baach, P., Giron Mathern, A., Baltazar, N., & McManimon, S. (July, 2016). Who’s not here?: Questions
about better understanding potential visitors. Visitor Studies Association Conference Abstract, Boston,
MA. July 19-23, 2016.

Bowen, F., Newenham-Kahindi, A., & Herremans. (2010). When suits meet roots: The antecedents and
consequences of community engagement strategy. Journal of Business Ethics, 95(2). Retrieved from
http://www.uapa533.com/uploads/8/4/4/9/8449980/when-suits-meet-roots.pdf

59
 Visitor Studies Association - 2018 Conference Abstracts

Cole, J.B. & Kashan, N. (not dated). Telling all our stories: 10 steps to greater diversity, equity,
accessibility, and inclusion in museums. Retrieved from
http://cookross.com/wp-content/uploads/2018/05/TellingOurStories-DEAIMuseums.pdf

Dawson, E. (2014). “Not designed for us”: How science museums and science centers socially exclude
low-income minority ethnic groups. Science Education, 98(6). Retrieved from
http://onlinelibrary.wiley.com/doi/10.1002/sce.21133/epdf

Dilenschneider, C. (2016). Why cultural organizations are not reaching low-income visitors. Retrieved
from
https://www.colleendilen.com/2016/05/18/why-cultural-organizations-are-not-reaching-low-in
come-visitors-data/

Garibay, C., Yalowitz, S., & Guest Editors. (2015). Redefining multilingualism in museums: A case for
broadening our thinking. Museums & Social Issues, 10(1). 2-7. Retrieved from
https://doi.org/10.1179/1559689314Z.00000000028

Harde & Company Community Outreach. (2016). Investing in Cultural Participation and Financial
Sustainability: Cross-Cohort Analysis of the Arts Regional Initiative, 2009-2014. The James Irvine
Foundation.

Holmes, C. (2017, December 21). When an Elite Museum Prioritizes Inclusion. Retrieved June 1, 2018,
from ​https://nextcity.org/daily/entry/when-an-elite-museum-prioritizes-inclusion

LaPlaca Cohen. (2017). Culture Track. Retrieved from ​https://culturetrack.com/research/reports/

MASS Action. (2017). Museum as Site for Social Action Toolkit. Retrieved from
https://www.museumaction.org/resources/

McCarthy, K., Jinnett, K.(2005). A New Framework for Building Participation in the Arts. RAND.

Simon, N. (2016). The Art of Relevance. Santa Cruz, CA: Museum 2.0.

Additional Links 
 
https://crystalbridges.org/reports-and-research/
 
Parents as Partners in Supporting Early Childhood STEM Learning
Scott Pattison, Institute for Learning Innovation
Sasha Palmquist, Institute for Learning Innovation
Maureen Callanan, University of California Santa Cruz
Graciela Solis, Loyola University Chicago

Purpose

Although museums and other informal learning contexts are critical for supporting early childhood STEM
learning, programs often undervalue the important role of parents and caregivers. In this session,

60
 Visitor Studies Association - 2018 Conference Abstracts

presenters will share findings and lessons learned from three recent research-practice partnership
projects that attempted to authentically engage parents as collaborators in supporting children’s
learning. Through the presentations and discussions, participants will:

1) Learn about recent approaches to working with parents as partners in order to support early
childhood STEM interest development and learning;
2) Consider examples of research and evaluation methods to study family learning and support
authentic partnerships with parents; and
3) Discuss lessons learned, challenges, and future directions for the field.

The session will be relevant to researchers, evaluators, and other visitor studies and museum
professionals interested in family engagement and early childhood STEM learning in out-of-school
contexts.

Abstract

Presenters will describe research findings and lessons learned from three recent projects that aspired to
authentically engage parents and primary caregivers as partners in supporting and studying family
engagement. The session will highlight challenges and successes related to working with parents, as well
as a variety of research methodologies aligned with this approach to family learning.

My Sky Tonight
(Sasha Palmquist, Maureen Callanan, Jennifer Jipson)
Young children’s ideas about science often emerge in everyday conversations (Callanan et al., 2013;
Callanan & Valle, 2008). My Sky Tonight is an NSF-funded research-practice partnership (RPP) focused
on preschool-aged children’s interest in and understanding of astronomy, using parent-child
conversations as a setting for early astronomy-related thinking. Analysis of parent diary reports of
conversations about nature revealed both causal explanatory talk and talk that highlighted children’s
personal connections with the topic. The team will discuss patterns in this talk across three groups of
families from different backgrounds and reflect on how this research informed the development of a
toolkit of activities that support children’s early engagement in science (Sobel & Jipson., 2016).

Head Start on Engineering

(Scott Pattison, Pam Corrie, Gina Svarovsky, Marcie Benne, Shannon Weiss, Verónika Nuñez)
Although not often a focus of early childhood education, engineering skills are increasingly recognized as
essential for individuals to thrive in today’s society. The NSF-funded Head Start on Engineering (HSE)
project is designed to introducing low-income families to the engineering design process through a
multi-component family engineering engagement program integrated within the Head Start model
(Pattison et al., 2017; Svarovsky et al., 2017). HSE fosters connections across preschool and family
learning contexts to catalyze long-term, enduring interest pathways. The program engages Head Start
staff and families as collaborators and co-learners and uses a family system perspective (Pattison et al.,
2018) to understand and support family engineering-related interest development. In this session, the
team will share findings and lessons learned from the project, including insights about how parents and
children jointly developed engineering-related interests extending beyond program materials and
activities.

Expert Narratives and Parent-Child STEM Talk During Inquiry-Based Programs

61
 Visitor Studies Association - 2018 Conference Abstracts

(Graciela Solis, Catherine Haden, David Uttal, Diana Acosta, Lauren Pagano, Meriem Sadoun, Autumn
Crowe)
Oral narratives can support family science learning in informal learning environments (e.g., Avraamidou
& Osborne, 2009; Dahlstrom, 2014). Using a design-based research approach, this project focused on
when and how oral narratives told by engineering experts help parents to engage their children in
discussions of STEM in museums and libraries. More than 100 children ages 6 to 10 were presented with
engineering design challenges with their parents. Analysis showed that parents talked more about
engineering when the experts explicitly emphasized the engineering design process in their oral
narratives (e.g., “Alright, so first… name the problem, brainstorm ideas”). This project is yielding
important information about aspects of engineering expert narratives that advance parent-child
STEM-related conversational interactions.

Importance

To build trust and authentically engage families with young children, parents and caregivers must be
treated as collaborators for both supporting learning and researching the processes and outcomes of
these efforts. Early childhood is increasingly recognized as a critical time period for STEM learning and
interest development (Alexander et al., 2015; McClure et al., 2017; Morgan et al., 2016; NSTA, 2014).
Museums and other informal learning contexts have a central role to play in these experiences,
supporting family learning, open-ended exploration and play, and equitable access to quality education
experiences (McClure et al., 2017; NRC, 2009). Although there are many early STEM learning programs,
institutions often underestimate the important role of parents—or worse, see parents as barriers to
STEM engagement. The visitor studies field can take a leading role in supporting and empowering
parents, both in partnering with families through research and promoting authentic parent engagement
program models.

References

Alexander, J. M., Johnson, K. E., & Leibham, M. E. (2015). Emerging individual interests related to science
in young children. In K. A. Renninger, M. Nieswandt, & S. Hidi (Eds.), Interest in mathematics
and
science learning (pp. 261–280). Washington, DC: American Educational Research Association.

Avraamidou, L., & Osborne, J. (2009). The role of narrative in communicating science. International
Journal of Science Education, 31(12), 1683–1707.

Callanan, M. A., Luce, M. R., Triona, L., Rigney, J. C., Siegel, D. R., & Jipson, J. L. (2013). What counts as
science in everyday and family interactions. In B. Bevan, P. Bell, R. Stevens, & A. Razfar (Eds.),
LOST opportunities: Learning in out-of-school time (pp. 29–49). New York, NY: Springer.

Callanan, M. A., & Valle, A. (2008). Co-constructing conceptual domains through family conversations
and activities. In Ross, Brian H. (Ed.), Psychology of Learning and Motivation (Vol. 49, pp.
147–165). London, UK: Elsevier.

Dahlstrom, M. F. (2014). Using narratives and storytelling to communicate science with nonexpert
audiences. Proceedings of the National Academy of Sciences, 111(Supplement 4), 13614–13620.

McClure, E. R., Guernsey, L., Clements, D. H., Bales, S. N., Nichols, J., Kendall-Taylor, N., & Levine, M. H.

62
 Visitor Studies Association - 2018 Conference Abstracts

(2017). STEM starts early: Grounding science, technology, engineering, and math education in
early childhood. New York, NY: The Joan Ganz Cooney Center at Sesame Workshop. Retrieved
from ​http://www.joanganzcooneycenter.org/publication/stem-starts-early/

Morgan, P. L., Farkas, G., Hillemeier, M. M., & Maczuga, S. (2016). Science achievement gaps begin very
early, persist, and are largely explained by modifiable factors. Educational Researcher, 45(1),
18–35.

National Research Council. (2009). Learning science in informal environments: People, places, and
pursuits. Washington, DC: National Academies Press.

National Science Teachers Association. (2014). NSTA position statement: Early childhood science
education. National Science Teachers Association. Retrieved from
http://www.nsta.org/docs/PositionStatement_EarlyChildhood.pdf

Pattison, S. A., Svarovsky, G. N., Gontan, I., Corrie, P., Benne, M., Weiss, S., … Ramos-Montañez, S.
(2017).
Teachers, informal STEM educators, and learning researchers collaborating to engage
low-income families with engineering. Connected Science Learning, 4. Retrieved from
http://csl.nsta.org/2017/10/head-start-engineering/

Pattison, S. A., Weiss, S., Ramos-Montañez, S., Gontan, I., Svarovsky, G., Corrie, P. G., … Smith, C. (2018).
Engineering in early childhood: Describing family-level interest development systems. Presented
at the NARST 91st Annual International Conference, Atlanta, GA. Retrieved from
http://informalscience.org/engineering-early-childhood-describing-family-level-interest-develop
ment-systems

Sobel, D. M., & Jipson., J. L. (Eds.). (2016). Cognitive development in museum settings: Relating research
and practice. New York, NY: Psychology Press.

Svarovsky, G. N., Pattison, S. A., Verbeke, M., Benne, M., & Corrie, P. (2017). Head Start on Engineering:
Early findings (work in progress). Presented at the ASEE Annual Conference & Exposition,
Columbus, OH: American Society for Engineering Education. Retrieved from
https://www.asee.org/public/conferences/78/papers/20296/view
 
Understanding and Assessing Visitor Attention
Ross J. Loomis Ph.D. , Psychology Professor Emeritus, Colorado State University
Stephen Bitgood, Psychology Professor Emeritus, Jacksonville State University
Scott Pattison, Institute for Learning Evaluation

Purpose

This panel session is designed to review how visitor attention transpires and integrates with an
understanding of how visitors value their experiences. A second purpose is to present a model for
assessing visitor attention using well established approaches to evaluation in visitor studies.

Abstract

63
 Visitor Studies Association - 2018 Conference Abstracts

Understanding how visitors attend to exhibits and programs presented in museums is one of the most
basic needs for insuring a quality visit. Exhibit spaces require exploration. Helping visitors have a
satisfactory visit includes guiding their attention while also creating a satisfying informal learning and
leisure time experience. Success at these tasks insures the public will view museums as special places to
experience. This panel will use two presentations and facilitated discussion to understand and assess
visitor attention. Presentation One (Bitgood) will provide a model to help understand visitor attention
and how attention relates to visitors valuing their experiences. Following it, a discussant (Pattison) will
encourage questions and discussion. Presentation Two (Loomis) will cover a strategy for both assessing
visitor attention and guiding evaluation efforts. Following this presentation, Pattison and Loomis will
provide some exercises for using the evaluation plan covered. A bibliography and handout aides will be
provided to participants.

Presentation One
Stephen Bitgood: Understanding Visitor Attention

The Attention-Value Model (e.g., Bitgood, 2010; 2011; 2013; 2014), adopting principles from many
sources over the 100+ years of visitor studies, argues that both the psychological processes of attention
and the motivational role of economic value are fundamental to understanding how and when visitors
pay attention within the museum environment. This presentation describes the stages of visitor
attention (capture, focus, engage), how economic value serves as a primary motivator, and the
characteristics of a museum environment that support paying attention to interpretive content. In
addition, some guiding principles that encourage engaged visitor attention are offered.

Presentation Two
Ross J. Loomis: Assessing Attention by combining Screven’s Process Evaluation with a Developed Logic
Model

An integrated model (Loomis, 2017) including both Logic Analysis (Kellogg, 2001) and Screven’s Process
Evaluation (Screven, 1990) is designed to help evaluate visitor attention in two ways. First, it can be used
as a general assessment of visitor attention. Included are measuring how well existing exhibit design and
layout attract and hold visitor attention to objects on display. This work can include evidence based
Critical Appraisal. It can also be used to determine if interpretation, such as labels or media
presentation, attract and hold attention. Remedial evaluation is built into the logic analysis and can be
done as part of the general assessment. Second, the model can help evaluators and exhibit creators
evaluate a developing exhibit. Using the model to guide development of new exhibits has the advantage
of integrating with logic analyses the practical and popular Process Evaluation Model with stages of
Front-End, Formative, Remedial, and Summative assessments.

Importance

Understanding how visitors attend to exhibits and other interpretive offerings is very important for high
quality visitor experiences. This panel session will help attendees understand what draws visitor
attention and how to assess attention. Planning and assessment work of this kind aids in the important
goal of museums being quality institutions sensitive to the needs of their publics.

References

64
 Visitor Studies Association - 2018 Conference Abstracts

Bitgood, S. (2000). The role of attention in designing effective interpretive labels. Journal of
Interpretation Research, 5(2) 31-45.

Bitgood, S. (2002). Environmental psychology in museums, zoos, and other exhibition

centers. In Handbook of environmental psychology, R. Bechtel & A. Churchman
(eds.). New York, NY: John Wiley & Sons, pp. 461–480.

Bitgood, S. (2010). An attention-value model of museum visitors. The Center for the
Advancement of Informal Science Education. Available at http://caise.insci.
org/uploads/docs/VSA_Bitgood.pdf

Bitgood, S. (2011). Social design in Museums: The Psychology of visitor studies, Volumes 1 & 2.
Edinburgh, UK: MuseumsEtc.

Bitgood, S. (2013). Attention and Value: Keys to understanding museum visitors. Walnut Creek, CA: Left
Coast Press.

Bitgood, S. (2014). Engaging the visitor: Designing exhibits that work. Edinburgh, UK: MuseumsEtc.

Kellogg Foundation. (2001). Logic model development guide: Logic models to bring together planning,
evaluation & action. Battle Creek, MI: W.K. Kellogg Foundation.

Loomis, R. J. (2017). An integrated model of visitor evaluation in museum settings. National Social
Science Association Fall Conference Proceedings, San Antonio, TX, (66), 28-35.

Screven, C G. (1990). Uses of evaluation before, during and after exhibit design. ILVS Review: A Journal
of
Visitor Behavior, 1(2), 36-67.

Additional Links
 
University of Wisconsin Extension Education presentation of logic models in program development and
evaluation: ​https://fyi.uwex.edu/programdevelopment/logic-models
 
Saturday, July 21
9:45-11:00 AM - Concurrent Sessions
Gathering Data to Inform Equity-Focused Organizational Change:
Methodological Implications
Gretchen Haupt, Evaluation and Research Associate, Science Museum of Minnesota
Choua Her, M.Ed., Evaluation and Research Associate, Science Museum of Minnesota
Evelyn Ronning, Ph.D., Evaluation and Research Senior Associate, Science Museum of Minnesota

Purpose

65
 Visitor Studies Association - 2018 Conference Abstracts

Measuring, understanding and furthering equity work among museum practitioners requires evaluators
to call on innovative, responsive, and collaborative approaches. This session conveys practical and
theoretical insights on collecting practitioner data in equity-based projects, as well as provides examples
of specific methods and instruments from three differently-scaled evaluation and research projects
undertaken at the Science Museum of Minnesota -- the STEM Justice, iPAGE, and RACE Forward
projects.Using these three differently-scaled examples, panelists and participants will explore the ethical
and practical implications of working with practitioners to gather and use organizational data to
transform their institutional structures, program models, and cross-departmental relationships in
support of equity initiatives.

Abstract

Often organizational equity work primarily focuses on diversifying visitors without adequate attention to
the internal equity work among staff necessary to authentically collaborate with underrepresented
communities, and for community members to trust that our institutions are responsive of their needs.
Organizational equity work should include practitioner/colleague data collection and analysis to inform
equity initiatives and ensure their success. However, this work can be intense and transformational,
provoking methodological challenges and unforeseen ethical considerations.

The NSF-funded RACE Forward project engages museum practitioners to participate in project groups
focused on cross-departmental action research on organizational transformation of racial equity
practices and policies. Methodological approaches include: practitioner-collected qualitative data from
group meetings such as memos, observations, and project narratives, and an organization-wide equity
survey developed with practitioners, proctored by researchers, and analyzed by both project
researchers and practitioners. Specific methodological challenges to be discussed include the
communication about the purpose of and about data from the survey, and about the processes through
which the survey was developed and analyzed, in order to gain staff trust in the project and the
practitioner-researchers. and to adequately represent diverse perspectives across the organization.

The NSF-funded project iPAGE - Developing a Model for Broadening Participation in Informal STEM
studies how the iPAGE model of professional development prepares cross-institutional teams to work
collaboratively to make their institutions more inclusive learning environments. The project model
incorporates face-to-face workshops, site visits, and institution-specific activities geared to build
knowledge around, awareness of, and capacity for equity initiatives at informal STEM institutions. The
project has relied on participant surveys, interviews, observations and will implement case studies to
understand the program model design, which necessitated: awareness and sensitivity to status and
hierarchical dynamics within institutional teams (e.g. ensuring confidentiality in data collection and
reporting); unintended outcomes related to data collection on potentially sensitive topics; and fostering
effective collaborative inter-departmental relationships.

The NSF-funded project STEM Justice: Investigating a Model for Building Youth Science Capital examines,
documents, and shares a program model where high school-age youth apply STEM content to advance
social justice while also building confidence and workforce skills. Utilizing a participatory approach to
integrate theory, research, and practice in studying informal STEM education, an interwoven team of
youth and adults work together to analyze and discuss learning practices and research findings.
Throughout the research process, consideration has been given to methodologies that intentionally
center and uplift voices or perspectives that have historically been untapped in STEM. Tensions usually

66
 Visitor Studies Association - 2018 Conference Abstracts

found in the relationship between research and practice have informed how data is collected and used
to increase collaboration across museum staff that are in the same organization but have different
approaches and connections to equity work.

These project examples illuminate the benefits and pitfalls of different approaches to evaluating and
informing organizational change around equity work at different levels: organizational,
cross-departmental, and within a single department. The session utilizes the examples of these projects
as a springboard for communication across session participants about other examples of fruitful
approaches to evaluating equity.

Importance

Museums and visitor-centered organizations, particularly those with a charge to serve the public, have a
responsibility and a role to play in challenging social inequities that pervade society (Sandell 2003).
Addressing organizational equity is a key component of building trust with sectors of the public that
have been disenfranchised and marginalized by dominant, white systems and structures. Because
addressing internal issues with equitable practices and policies has structural and political implications,
the role of the evaluator in this work is often fraught. Thus, this session is both crucial for fostering trust
among visitors through transparency in addressing equity in practice, as well as pointing other
evaluators to potential methods and processes for doing this work within organizations, eventually
building capacity and a toolkit of best practices for evaluating and communicating about organizational
equity within the field.

References

Center for the Future of Museums. “Demographic Transformation and the Future of Museums.”
http://www.aam-us.org/docs/center-for-the-future-of-museums/demotransaam2010.pdf?sfvrs
n=0​ Accessed June 1, 2018.

Dawson, Emily. “Why Think About Equity and Museums.”

https://incluseum.com/2015/03/23/why-think-about-equity-and-museums/
The Incluseum. Accessed June 1, 2018.

The Equitable Evaluation Project. ​https://www.equitableeval.org/​ Accessed June 1, 2018.

GovLoop. “Tips For Advancing Equity Through Evaluation.”

https://www.govloop.com/community/blog/tips-advancing-equity-evaluation/
Accessed June 1, 2018.

NonProfit Quarterly. “Research and Evaluation in the Nonprofit Sector: Implications for Equity, Diversity,
and Inclusion.”
https://nonprofitquarterly.org/2016/10/19/research-evaluation-nonprofit-sector-implications-e
quity-diversity-inclusion/​ Accessed June 1, 2018.

The Opportunity Agenda. “Ten Tips for Putting Intersectionality into Practice.”
https://opportunityagenda.org/explore/resources-publications/ten-tips-putting-intersectionality
-practic​e Accessed June 1, 2018.

67
 Visitor Studies Association - 2018 Conference Abstracts

Sandell, R. (2003). Museums and the combating of social inequality: roles, responsibilities, resistance. In
Museums, society, inequality. New York: Routledge.
 
Researching with a Field-Wide Perspective
Nicole R. Rivera, Assistant Professor of Psychology, North Central College
Kari Ross Nelson, Research and Evaluation Associate, Thanksgiving Point Institute
Michelle L. Nguyen, Evaluation & Grants Associate, Boston Children’s Museum
Sarah Brenkert, Senior Director of Education and Evaluation, The Children’s Museum of Denver at
Marsico Campus

Purpose

In the November/December 2016 issue of Museum magazine, John Jacobsen wrote about the need for
“shared data definitions and collections methods” to improve individual museums, as well as the
museum field as a whole. The Association of Children’s Museums Research Network is an example of
researchers and practitioners from various institutions, and in various locations, sharing data definitions
and collection methods to benefit Children’s Museums more widely. The purpose of this VSA conference
session is to share what we’ve learned about the the logistics of collaborative research, as well as
methods and findings of the Network’s efforts. The session with explore three studies conducted by the
Network, with a particular focus on making connections between research and practice. We’ll then
consider the benefits and implications of taking a field-wide perspective in research, evaluation, and
visitor studies.

Abstract

At the 2017 VSA conference in Columbus, Ohio, members of the Children’s Museums Research Network
(CMRN) presented a session on collaborative research, using the CMRN as a case study. While the
session focused on logistics, benefits, and challenges of collaborative research, attendees showed a
strong interest in studies the group had conducted and their findings. This session is in response to that
interest.

In 2012, the Association of Children’s Museums (ACM) initiated the Learning Value of Children’s
Museum Research Agenda Project with funding from IMLS. ACM worked in partnership with the
University of Washington (UW) Museology program to complete a comprehensive review of available
research and evaluation in the children’s museum field. The landscape review was shared with over 100
museum professionals at the Research Agenda Symposium in September 2013. The symposium resulted
in the development of a field-wide research agenda, articulating research topics that could demonstrate
the learning impact of children’s museums. With support from a second IMLS grant, ACM and UW again
teamed up to develop the Children’s Museum Research Network in 2014 to dig into some of those
research topics. To date, the network has completed three studies, and is in data collection stage of a
fourth study.

In the Network’s first study, we examined how children’s museums define learning through the lens of
learning frameworks. A case study approach was used to explore the learning frameworks of five
network museums to look at the major vocabularies used by the museums, priorities for constructs, and
implicit or explicit theories represented by the museums. Analysis revealed key findings related to
learning approaches, learning outcomes, and the role of play in the learning frameworks.

68
 Visitor Studies Association - 2018 Conference Abstracts

Questions about play in the first study led to a second study that looked at the role of play in children’s
museums. This time we conducted interviews with representatives from a stratified random sample of
children’s museums across the United States. The study aimed to describe how children’s museums
conceptualize play, and how they position their work relative to play. Findings from this study showed
that while play is important to the field, there is not consensus regarding the role of play at children’s
museums. These findings stimulate significant exploration of arguments and counterarguments for a
shared definition of play.

The third study moved to visitors as the source of information for understanding learning in children’s
museums. Through in-depth questionnaires with over 200 parents and caregivers bringing children to
the museums, we sought to reveal what parents/caregivers learn about their children from their visit,
and what it is about the children's museum experience that they feel contributes to that learning. The
majority of caregivers indicated that they learned something about their children’s learning during a
visit. Further, they attributed that learning to the built space, opportunities to observe their children,
and socialization with other visitors.

Examination of these studies lends itself to further discussion of implications of findings, application of
research in practice, and benefits and challenges of collaborative research with a field-wide perspective.

Importance

Discussion of the the Research Agenda and the Network’s collaborative studies will provide models of
research methods and dissemination. Additionally, as participants engage in a conversation about the
wider lens of a process that focuses on larger field-wide perspectives, conference theme, New Pathways
in Visitor Studies, will take on relevance as we explore questions such as:
How research and evaluation studies can be conducted, and findings disseminated, in a way that informs
and impacts a broader field?
Are there similarities and differences across various types of information such that work in one may
benefit another?
What are the larger questions that are facing museums and cultural institutions?
These topics speak to the importance of shared efforts to strengthen the case not just for individual
institutions, but for the museum field as a whole.

Additional Links

https://www.childrensmuseums.org/childrens-museum-research-network

http://www.informalscience.org/news-views/learning-value-childrens-museums-research-agenda

https://www.imls.gov/blog/2017/02/exploring-learning-value-childrens-museums-through-research-net
work
 
What's the Benefit of Single-Visit Art Museum Programs to Students?
Stephanie Downey, RK&A
Melissa Higgins-Linder, Cleveland Museum of Art
Amanda Krantz, RK&A

69
 Visitor Studies Association - 2018 Conference Abstracts

Purpose

Through this session, panelists will disseminate results of a national study into the effects of single-visit
art museum programs on students in grades 4-6. The study was spearheaded by the Museum Education
Division of the National Art Education Association (NAEA) and the Association of Art Museum Directors.
It was conducted by researchers at RK&A with funding from the Institute of Museum and Library
Services. In sharing the results of this study:

● Participants will become familiar with a quasi-experimental research design and rationale for
the design (related to VSA Competency C).
● Participants will understand the benefit of single-visit art museum program revealed through
this study through the lens of prevalent practices and goals for single-visit art museum programs
(related to VSA Competency B).
● Participants will consider the benefits and challenges of conducting evaluation that explores the
public value of museum work (related to conference theme).

Abstract

Single-visit field trip programs are the primary ways many museums, and particularly art museums,
serve K-12 students. While museum annual reports regularly boast about number of students served, a
question lingers to demonstrate public value: What are the benefits of single-visit art museum programs
for students? Simultaneous discussions arose within the Museum Education Division of NAEA and AAMD
between 2010 and 2011 about the need for more rigorous research responding to this question. For,
while several studies examined multi-visit art museum programs (Adams et al., 2007; Curva and
Associates, 2005; RK&A, 2007; 2010; 2014; Tishman, 2003), there is a dearth of information about the
effects of single-visit programs on students. Since planning for this study began, only studies of
single-visit programs at the Crystal Bridges Museum of American Art have begun to fill this gap (Bowen
et al., 2014; Greene et al., 2014; Kisida et al., 2016).

In addition to concentrating on single-visits, this study focused on inquiry-based programs and five
interrelated student capacities museum educators believed such pedagogy supports: 1) critical thinking,
2) creative thinking, 3) sensorial experience, 4) human connections, and 5) academic connections. A
literature review by NAEA describes inquiry-based pedagogies and these five capacities (Terrassa et al.,
2016).

RK&A conducted a quasi-experimental study that included three study groups: Treatment A that
received a museum program, Treatment B that received a classroom program, and a Control that did not
receive any program. The intent of the Treatment B study group was to pinpoint differences between
inquiry-based art museum programs presented in the museum with original works of art, versus in a
school classroom with reproductions of works of art. Multiple methods, to be described during the
presentation, were used to understand the results from multiple perspectives.

70
 Visitor Studies Association - 2018 Conference Abstracts

Through the study, we found that single-visit programs had positive effects on students in a few areas.

1. Emotive recall – Students who participated in a single-visit art museum program recalled their
experience more emotionally than students who participated in a similar program in their
classroom—suggesting the influence of the art museum setting. This has implications for the capacity of
sensorial experience.

2. Question and wonder – Students who participated in a single-visit museum program were better
able to question and wonder about an unfamiliar work of art than students who did not receive an art
museum program. This has implications for the capacities of creative thinking and critical thinking.

3. Multiple interpretation – Students who participated in a single-visit art museum program were
more likely to disagree with the statement, “All people should understand art in the same way,” than
students who did not receive an art museum program (positive finding). This has implications for the
capacities critical thinking, creative thinking, and human connections.

4. Materials & mediums – Students who participated in a single-visit art museum program were
more likely to refer to materials & mediums when asked what comes to mind when they think about art.
This has implications for the capacity of creative thinking and sensorial experience.

Importance

As noted previously, this is the first national study of single-visit art museum programs. While we will
continue consider the results with the art museum educators, leaders, and stakeholders, we believe
these findings have numerous implications, which will be unpacked at a symposium at the Detroit
Institute of Arts in October 2018. In particular, art museums must look closely at how they promote and
market the benefits of single-visit art museum programs on students when speaking with teachers,
school districts, and policymakers.

71
 Visitor Studies Association - 2018 Conference Abstracts

References

Adams, M., Foutz, S., Luke, J., Stein, J. (2007). Thinking about art: Isabella Stewart Gardner museum
school partnership program year 3 research results. Annapolis, MD: Institute for Learning
Innovation.

Bowen, D.H., Greene, J.P., & Kisida, B. (2014). Learning to think critically: A visual experiment.
Educational Researcher, 43(1), 37-44.

Curva and Associates (2005). Artful citizen project: Three-year project report. Miami Beach, FL.: The
Wolfsonian.

Greene, J.P., Kisida, B., & Bowen, D.H. (2014). The educational value of field trips. Education Next, 14(1),
78-86.

Kisida, B., Bowen, D.H,, & Greene, J.P. (2016). Measuring critical thinking: Results from an art museum
field trip experiment. Journal of Research and Educational Effectiveness.

RK&A (2007). Teaching literacy through art. Unpublished report. New York, NY: The Solomon R.
Guggenheim Museum.

RK&A (2010). The art of problem solving. Unpublished report. New York, NY: The Solomon R.
Guggenheim Museum.

RK&A (2014) Program evaluation: Pages program. Unpublished report. Columbus, OH: Wexner Center
for
the Arts.

RK&A (2015). Summary of results: Survey of single-visit K-12 art museum programs. Unpublished report.
Reston, VA: National Art Education Association.

Terrassa, J., Hubard, O., Holtrop, E., & Higgins-Linder, M. (2016). Impact of art museum programs on
students: Literature review. Alexandria, VA: National Art Education Association.

Tishman, S. (2003). Investigating the educational impact and potential of the Museum of Modern Art’s
visual thinking curriculum. Cambridge, MA: Harvard University.

Additional Links
 
Study website, with links to all publications to date:
https://www.arteducators.org/research/articles/377-naea-aamd-research-study-impact-of-art-museum-
programs-on-k-12-students

Study symposium information:

https://drive.google.com/file/d/1GcaLW-UPIkMYOf6hSnP2SblcUIQqf-KC/view
 

72
 Visitor Studies Association - 2018 Conference Abstracts

Outside In: Learning from Community Voices in Visitor Studies

Cecilia Garibay, Principal, Garibay Group

Veronica Garcia Luis, Project Director, Visitor Research & Evaluation and Director of Diversity Initiative,
Organizational Development, Exploratorium

Leticia Perez Castellanos, Professor, Escuela Nacional de Conservación, Restauración y Museografía

PhD Candidate, Universidad Autónoma Metropolitana-Iztapalapa

Rebecca Teasdale, Senior Evaluation and Research Associate, Garibay Group, PhD Candidate, University
of Illinois at Urbana-Champaign

Purpose

In efforts to expand reach to better serve their communities, museums have experimented with a range
of initiatives that take place beyond the museum walls. By expanding our conceptualization of program
and exhibition planning, where we physically locate experiences and exhibitions, and our practice of
evaluation, we can begin to bring community perspectives and values into our work in new ways. This
panel discusses approaches to honoring and incorporating community voice in planning,
implementation, and evaluation of informal learning projects. Issues explored include ways we might
define “success” in ways that consider community perspectives; approaches that helps us learn about
the community’s impact; and possibilities for tracing long-term impact. After panel presentations,
breakouts discussions will allow for deeper dialogue of issues raised.

Abstract

Rebecca Teasdale will discuss her research on methods to reflect community perspectives in the
evaluative criteria (or indicators of success) used in evaluation of informal science education (ISE)
programs and exhibitions. Often, we think about achievement of desired outcomes as the key indicator
of success. However, through analysis of a sample of reports posted on informalscience.org and a survey
with the report authors, she has identified a range of criteria used in ISE evaluations including a project’s
relevance, significance, and the extent to which it fosters equity. Further, some evaluators are taking
steps to define those criteria from the perspective of the community. Questions raised include: How do
we define “success” for a program/exhibition? How might community members define “success” for the
same program/exhibition? How can we choose criteria to reflect community values and perspectives?

Cecilia Garibay will discuss approaches to integrating community voice into summative evaluation. Using
the Ciencia Publica (CP) project—a parklet space developed by the Exploratorium and located in the
Mission (a predominantly Latino neighborhood)—as an example, she will describe the collaborative
evaluation process used to consider the success of the project from the community perspective.
Questions raised include: How do we balance the need for assessing desired outcomes with the desire
to understand value as defined by the community? What are the implications for scope? What are
promising approaches and methods?

Veronica Garcia-Luis will share and reflect on the Exploratorium’s outdoor offsite exhibit
co-development experiences with underrepresented communities. A series of public space projects will
be highlighted by examining a) the role of community as contributor; b) reflecting on the

73
 Visitor Studies Association - 2018 Conference Abstracts

semi-permanence of these spaces over time; c) showing the ways in which public space is used beyond
the original project goals; and d) the reality of content accessibility in public community spaces.
Questions raised include: How can we learn more about the community’s impact on these public spaces
beyond the project timeline? How do we continue to engage and learn from community participants as
they evolve the use of the public space over time?

Leticia Perez Castellanos will discuss her current research, which considers the question of ways to
define and measure community impact of museum outreach efforts. La Casa del Museo, developed by
the National Museum of Anthropology, consisted of semi-permanent museum and programming spaces
installed in three of the most marginalized communities in Mexico City. The project ran from 1972 –
1980. Using a mix of archival research and interviews with individuals who lived in these communities
and visited exhibits in these museum sites, she examines the notion of long-term impact from
community members’ perspectives. Questions raised include: How do we trace long-term impact? How
can we identify qualitative traces or resonances of our museum work? How can we establish museum
exhibitions and programs’ outcomes considering a prospective way and future impact?

Importance

Strengthening public trust and continuing to enhance service to our communities require informal
learning organizations to honor and incorporate community voice in the planning, implementation, and
evaluation of projects. This panel shares four examples that aim to spark thinking within the field about
how this can be accomplished. Specifically, we seek to foster reflection on how we can broaden our
approaches to defining impact, consider impact beyond the museum walls, and develop strategies to
incorporate community voice into all aspects of our programs and exhibitions.

The panelists hope this session will spark several ideas and implications for future research, evaluation,
and community engagement practices. Areas of research to consider are conducting longitudinal
analysis of the visitor experience and further exploring the impact on communities beyond our museum
walls.

References

Cameron, D. F. (1993). Marble floors are cold for small, bare feet. Museum Management and
Curatorship, 12(2), 159-170.

Garcia-Luis, V., Hawkins, I., Blanco, L., & Oates, A. (Nov/Dec 2017). GENIAL Summit: Executive Summary
and Call to Action. Informal Learning Review, No. 147, 14-19.
Retrieved from ​https://www.exploratorium.edu/sites/default/files/GENIAL_IRL_Journal.pdf

Garibay, C. & Teasdale, R.M. (in press). Equity and evaluation in informal STEM institutions. In A. C. Fu,
Kannan, A., and R. J. Shavelson (Eds.), New Directions for Evaluation.

Henry, G. T. (2002). Choosing criteria to judge program success: A values inquiry. Evaluation, 8(2),
182-204.

Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally responsive evaluation: Theory, practice, and
future implications. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical
program evaluation (4th ed., pp. 281-317). Hoboken, NJ: Jossey-Bass.

74
 Visitor Studies Association - 2018 Conference Abstracts

Julnes, G. (2012). Managing valuation. In G. Julnes (Ed.), New Directions for Evaluation, 133, 3-15.

Ordoñez García, C. (1975). The Casa del Museo, Mexico City: an experiment in bringing the museum to
the people. Museum, XXVII(2), 71-77. Available at:
http://unesdoc.unesco.org/images/0012/001273/127359eo.pdf#nameddest=16103

Schwandt, T. A. (2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Palo
Alto, CA: Stanford University Press.

Additional Links

Generating Engagement and New Initiatives for All Latinos/Generando Entusiasmo y Nuevas Iniciativas
para Audiencias Latinas: GENIAL Webpage ​https://www.exploratorium.edu/education/genial

Garibay Group. (2015). Ciencia Pública summative report. Retrieved from

http://informalscience.org/ciencia-publica-summative-report

Leticia Pérez personal blog (available in English with google translator)

https://eldivanmuseologico.wordpress.com/

Exploratorium Studio for Public Spaces ​https://www.exploratorium.edu/publicspaces

 
Critical Appraisal: What's the Value?
Cathy Sigmond, Research Associate, RK&A 

Purpose

This session will discuss the value of using critical appraisal to improve museum programs. For instance,
one of the many benefits of this methodology is that it provides data expediently and at low cost.
However, the trustworthiness of the data is largely based on the expertise of the evaluator to make
judgments as to how to spend time to best understand how a program is functioning. In this session,
RK&A will break down the issues surrounding this methodology, welcoming and inviting comments from
VSA attendees about the value of critical appraisal as an evaluative tool. The session will draw heavily
from RK&A’s experience conducting critical appraisal for a series of Late Night programs offered
alongside the ICEBERGS installation at the National Building Museum (NBM) in 2016.

Abstract

In 2016, the National Building Museum approached RK&A to consider how to make data-driven
decisions about a series of after-hours Late Night Programs developed for the summer installation,
ICEBERGS. Instead of a traditional evaluation, RK&A conducted critical appraisal, an economic evaluative
strategy that provides rapid results and enables conversation and relationship-building between
educators and evaluators.

As defined by the Visitor Studies Association, critical appraisal is “the overall observations and expert
judgment of an exhibition, program or interpretive product by a professional evaluator (or panel of

75
 Visitor Studies Association - 2018 Conference Abstracts

professional evaluators) to identify obvious or suspected problems which can be immediately corrected
or studied later with visitor input.” While by definition, critical appraisal is conducted by an evaluator, it
is not generally considered “evaluation” because it is grounded in professional versus visitor input. In
other words, rather than questioning visitors directly, a professional evaluator observes, participates in,
and makes recommendations based on his/her expertise gained from evaluating similar programs,
exhibitions, etc. at various institutions. In the larger evaluation world, critical appraisal bears semblance
to rapid evaluation and assessment methods (REAM). REAM can take many forms but generally includes
audience input (e.g., interviews with program attendees), unlike critical appraisal. Generally speaking,
both critical appraisal and REAM are beneficial because they provide data expediently and at low cost.
At NBM, the museum’s education department aimed to learn from evaluation and remediate
programming throughout the summer. Critical appraisal was thus an economic choice since it would
allow RK&A to appraise three distinct programs only weeks apart.

Another distinguishing characteristic of critical appraisal is that data collection and analysis happen
simultaneously. At the same time, much of the data is not formally captured or recorded. Instead, it sits
within the evaluator, and thus requires extremely speedy reporting for reliability. The immediate
reporting following each program at NBM made it possible for education department staff to learn from
the critical appraisals “in real time,” thereby supporting the development of evaluative thinking skills
across the department and allowing educators to apply what they learned from the critical appraisals to
later summer Late Night programs. And, the fluidity and more casual nature of the approach, along with
the prompt application of results enabled educators to become more comfortable with integrating
evaluation into their work.

The major challenge or limitation of critical appraisal is that the trustworthiness of the data is largely
based on the expertise of the evaluator in both methodology of evaluation as well as in the environment
(i.e., museums). To successfully conduct critical appraisal, it is essential for the evaluator to fully
understand the study questions. While this is true of all evaluation studies, it is especially important with
rapid methods because, unlike with methods that take place over a longer timeframe, there is little to
no time to shift the direction of the study. And, having to report findings rapidly can result in a less
thorough report that does not lend itself to being widely shared or referenced a long time after the
appraised programs conclude.

Importance

It is always important for museum educators to consider what type of evaluation is required, as every
type has a distinct purpose. One of the greatest values of critical appraisal is that it can be thought of as
in the middle of a continuum of evaluative strategies, with formal evaluation conducted by evaluators
on one end and self-assessment or reflective practice conducted by educators on the other end.
Attendees will leave this session armed with an understanding of the benefits and limitations of using
critical appraisal to assess museum programs and feel inspired to try this evaluative strategy at their
institutions. To quote professor and visitor studies expert Stephen Bitgood on critical appraisal: “This
approach, if it proves to be a reliable and valid predictor of audience reaction, could save considerable
resources.”

References

Theresa Esterlund, Amanda Krantz, and Catherine Sigmond, “Using Critical Appraisal to Inform Program
Improvement,” Journal of Museum Education, Volume 42, Issue 4, 2017.

76
 Visitor Studies Association - 2018 Conference Abstracts

https://rka-learnwithus.com/wp-content/uploads/2017/12/Esterlund-Krantz-Sigmond-JME-Artic
le-for-website_PDF.pdf

The definition published in visitor studies glossary of terms was defined in 2006 after a convening of
visitor study professionals: “Glossary of Visitor Studies Terms,” Visitor Studies Association,
accessed December 7, 2017:
http://www.visitorstudies.org/index.php?option=com_content&view=article&id=77:glossary-of-
terms&catid=20:site-content&Itemid=157​ See also Stephen Bitgood’s similar description of
critical appraisal reprinted from a 1994 issue of Visitor Behavior in his book. Stephen Bitgood,
Social Design in Museums: The Psychology of Visitor Studies, Volume 1, (Edinburgh, UK:
MuseumsEtc, 2011): 145-147.

Harris Shettel, “A Return to the Future,” Visitor Behavior, (Fall 1993): 11-13.

Miles McNall and Penni G. Foster-Fishman, “Methods of Rapid Evaluation, Assessment, and Appraisal,”
American Journal of Evaluation 28, no. 2 (June 2007): 151.

Miles McNall and Penni G. Foster-Fishman, “Methods of Rapid Evaluation, Assessment, and Appraisal,”
American Journal of Evaluation 28, no. 2 (June 2007): 166.

Miles McNall and Penni G. Foster-Fishman, “Methods of Rapid Evaluation, Assessment, and Appraisal,”
American Journal of Evaluation 28, no. 2 (June 2007): 164-166.

Stephen Bitgood, Social Design in Museums: The Psychology of Visitor Studies, Volume 1, (Edinburgh,
UK: MuseumsEtc, 2011): 145.

Additional Links

RK&A’s website: ​https://rka-learnwithus.com/

ICEBERGS installation at the National Building Museum: ​https://www.nbm.org/exhibition/icebergs/

For more on the benefits and challenges of critical appraisal, see RK&A’s recently-published article in the
Journal of Museum Education, co-authored with NBM:
https://rka-learnwithus.com/wp-content/uploads/2017/12/Esterlund-Krantz-Sigmond-JME-Article-for-w
ebsite_PDF.pdf

The "Unsafe" Museum: Negotiating Difficult Heritage

Dr. Theopisti Stylianou-Lambert, Associate Professor, Cyprus University of Technology
Prof. Alexandra Bounia, Professor, University of the Aegean, UCL Qatar
Dr. Karen Exell, Honorary Senior Research Associate, UCL Qatar
Kate Flinner, Communications Manager & Researcher, New Knowledge Organization Ltd

Purpose

Museums are increasingly encouraged to become audience-focused institutions that include multiple
viewpoints, facilitate knowledge and strive to be relevant and forward-looking. However, most

77
 Visitor Studies Association - 2018 Conference Abstracts

museums dealing with difficult heritage still choose to adopt a more traditional approach because it is
considered more appropriate as it provides a “safe” authoritative environment for communicating
information in a seemingly objective manner. Often, it seems that the more contested or difficult the
heritage a museum is called to present, the more traditional the approach it adopts.

This session, that includes three paper presentations, looks at how museums or cultural heritage sites
that deal with difficult or contested heritage can provide transformative experiences. How can a “safe”,
traditional, ethnocentric museum become inclusive, open to multiple voices, a facilitator of knowledge
without reinforcing existing attitudes and beliefs? How can a museum that deals with difficult heritage
become an “unsafe”, unsettling place and provide transformative experiences?

Abstract

The first paper is titled “The ‘Unsafe’ museum: imagining the Ledra Palace Museum” (Stylianou-Lambert
& Bounia). This presentation offers an introduction to the concept of the “unsafe” museum and provides
ways to thinking about the future of museums that deal with difficult heritage. As a case study, we use
one of the most controversial sites in Cyprus: the Ledra Palace Hotel. Once praised as the jewel of
Cypriot modernity in the heart of its capital, now it is a crumbling dwelling accommodating the United
Nations Peacekeeping Force next to the division line between the southern and the northern parts of
Cyprus. This specific site is not a museum and there are no current plans to establish a museum. But it is
a perfect example of a symbolic, “loaded” building that can be used to negotiate the recent history of
conflict in Cyprus. The Ledra Palace Museum will become an imaginary space where - without the
practical and political restrictions of an actual museum - we can re-imagine the future of museums that
deal with difficult heritage.

The second paper is titled “Contemporary politics, audience expectations and new national museums:
the blockade of Qatar” (Exell). Drawing on data from a three-year audience research project which aims
to develop an understanding of the social and cultural perception, expectations and impact of the new
National Museum of Qatar across key demographic groups in Qatar, this paper explores how Qatari
nationals and residents – a community of great diversity and widely ranging opinions – imagine and
expect their new National Museum will respond to the blockade by Qatar’s neighbors, Saudi Arabia,
Bahrain and the United Arab Emirates and their allies, an event of immense political significance and
regional sensitivity. The respondents consider not only whether their National Museum is an
appropriate forum for this narrative, but if so, how should and could it be narrated? At the level of
national politics or at the level of the personal experience? The paper also contextualizes the Qatari
experience within broader approaches to politically sensitive subjects by national museums in the region
and elsewhere.

The third paper is titled “Human Rights Education in Museums” (Flinner) and has two sections. The first
section generally describes HRE, a perspective on human rights advocacy that can be uniquely helpful to
museums and other informal learning spaces. The second part of the presentation will present a brief
case study with a progressive, New York City-based historical society that did social science research to
inform the design of a new exhibition. A team of researchers from NewKnowledge worked with this
museum to understand how the public responds to difficult, charged topics like gentrification, historical
erasure, and environmental injustice. The research results advocated for a transparent approach to
presenting information in the exhibit, explicitly addressing difficult topics, and embracing multiple
voices. This presentation also aims to highlight that using HRE principles in museum work is accessible

78
 Visitor Studies Association - 2018 Conference Abstracts

for any museum, as this case study highlights a small/mid-sized institution with limited space and
resources.

Importance

Negotiating difficult heritage is a timely subject matter that many museum professionals are called to
deal with. This session can be useful to experienced evaluators and researchers, novice evaluators and
researchers, and emerging museum professionals.

Learning outcomes:
#1. Familiarize museum professionals and evaluators with current debates in museums and other
informal learning institutions
#2. Encourage the critical assessment of theories, studies, activities, and approaches relevant to
museums and other informal learning institutions

11:15-12:30 PM - Concurrent Sessions 

How Informal Learning and Science Communication Define and Assess Identity
Jamie Bell, Center for the Advancement of Informal Science Education
John Besley, Michigan State University
Kelly Riedinger, Oregon State University
Kevin Crowley, University of Pittsburgh
 
Purpose

The concept of “identity” has become increasingly relevant in STEM education and science
communication. Identity is an individually and socially constructed sense of self. One might think of it as
the way that people, on a daily basis, answer questions such as: “who do I think I am or who can I be,
where do I belong, how do I think other people see me?”

In an effort to create tools to support the evaluation of identity in informal learning and science
communication, the Center for Advancement of Informal Science Education (CAISE) conducted
interviews with leading identity researchers. Our session reports out a synthesis of what we learned,
presents prototype tools, and explores future directions.

Abstract

Individuals can have many different identities, but some are more basic or foundational to their sense of
self. Other identities might be more situated and only become prominent when a person is in a
particular context, such as in a learning setting, leisure environment, at home or at work. Various
identities can be reinforcing or in competition. Identity shapes engagement and learning, but it also
depends on factors such as ethnicity, gender, class, power, or political ideology.

People who develop identities related to science engage with these topics more often and more deeply.
Science identities increase the likelihood that students will, over the long term, continue to develop
science literacy or even follow an educational pathway towards a science career. One way that identity

79
 Visitor Studies Association - 2018 Conference Abstracts

shapes learning is through the evolving choices and expectations of a learner. First, when people engage
with science, having a positive science identity changes their expectations for how interesting and
successful the experience will be. Second, if they find the experience to be, in fact, interesting and
successful, the experience will then strengthen their science identity, leading to a positive feedback loop
that can reinforce ongoing science engagement and learning. But it can also become a negative
feedback loop for learners who are not having enjoyable and successful learning experiences.

In science communication, the approach is somewhat different: A partisan or cultural identity can
partially determine the degree to which people are attentive to information, whether they trust
scientific information sources, and how they process scientific information. Being liberal or conservative,
or leaning towards individualistic or collective preferences, all can influence how people perceive and
process scientific information. Science communication work tends to focus on adults making civic or
personal choices in light of their existing identity, as opposed to educational questions about
experiences that develop identity over the long term.

Science identity can be measured through both quantitative and qualitative approaches, and there are
considerable existing resources that can be drawn upon by evaluators. One resource would be the many
examples of open-ended interviews in which researchers ask people to talk about their identities, the
impact of identity on their lives, the roots of their identity in the learning history, and their expectations
of their goals and expectations of the future. Sometimes researchers observe people in science learning
contexts, noting how identity is expressed, performed, and changed. When researchers and evaluators
have questions that involve larger numbers of research participants, they often rely on self-report
measures consisting of a group of individual items that often ask about how one sees themselves with
respect to science, and how they perceive others see them with respect to science. There is no standard
measure of identity, and researchers and evaluators often tailor their particular survey questions to the
specifics of their audience or content. In addition to surveys, other techniques, such as implicit
association tests can provide behavioral measures of identity.

Importance

Those who attend this session will learn about how to conceptualize, measure, and support identity in
informal STEM education and science communication. They will learn about tools to help evaluators and
practitioners. They will also be able to share their own work and connect with each other around issues
of integrating the concept of identity into their ongoing evaluation and research.
 
Design-Based Research in Informal Education: Examples, Promise, and Pitfalls
Elizabeth Kunz Kollmann, Museum of Science, Boston
Marcie Benne, Oregon Museum of Science and Industry
Marjorie Bequette, Science Museum of Minnesota
Joyce Ma, Exploratorium

Purpose

This session will strengthen VSA Competency C: Knowledge of and Practices with Social Science Research
and Evaluation Methods and Analysis. Through this session, participants will gain:

80
 Visitor Studies Association - 2018 Conference Abstracts

- Some background on design-based research -- its lineage, related approaches, historical applications,
use in informal education, and opinions on future applications
- How design-based research differs from other methods such as formative evaluation
- Different ways that design-based research methodologies can be implemented in museums
- Benefits and drawbacks of conducting design-based research in informal education and specifically in
museums
- Ideas and input on design-based research questions from fellow participants

Abstract

The National Science Foundation’s 2015 Advancing Informal Science Learning program solicitation
suggested that projects include “iterative design-based data collection” or “design-based research
processes” as appropriate (NSF, 2015). As a result, a number of informal education projects have been
funded that integrate design-based research (DBR) methodologies. DBR is characterized by the creation
and iterative improvement of both an educational product and the theory that it is based upon (Collins,
Joseph, & Bielaczyc, 2004). Therefore, it involves processes similar to formative evaluation. However, it
adds an aspect of generating understandings for the broader field through the definition, refinement,
and testing of a theoretical framework upon which the educational product is based. Sandoval (2014)
laments that there is literature about what design-based research is and is not, but not about how to
implement it. Additionally, as DBR was created for the formal education field, there were certain
expectations about the kinds of educational product being tested and developed. This raises questions
for museum researchers: if we adapt DBR approaches to shorter educational experiences and products
that need to work for a broader range of people, what are the aspects of the DBR approach that hold
promise for informal education? Researchers will discuss how they implemented DBR, and the benefits
and drawbacks of using these methods.

A description of four DBR museum projects

During this session, participants will hear about four projects that have used DBR methodologies:

ChemAttitudes developed hands-on activities that promote positive changes in attitude about
chemistry. Educators and researchers worked together to optimize the abilities of the activities to
increase feelings of interest, relevance, and self-efficacy about chemistry and generate a framework to
describe the content, format, and facilitation strategies that promote these outcomes.

Living Liquid developed and studied how interactive visualizations could engage visitors in inquiry with
large scientific datasets. Using the DBR framework, it sought to investigate design factors affecting
visitors’ data inquiry practices by iteratively developing visualizations that embodied prior findings and
theories in close collaboration with museum practitioners and scientists.

Science on the Move (SOTM) advanced strategies for engaging adults in public settings with interactive
STEM exhibits. Through DBR, the team developed context-specific conjectures about public engagement
with STEM experiences at transit stations. After multiple cycles, the team created a single conjecture
that mapped to theoretical constructs drawn from the fields of environmental psychology and visitor
studies (Cardiel et al., 2016).

81
 Visitor Studies Association - 2018 Conference Abstracts

Making Connections explored how Making activities could better engage with communities of color, and
how the museum overall could engage differently with those communities. The project used DBR over
two rounds of activity iteration, and longitudinal tracking of relationships.

Benefits and Drawbacks of DBR

Researchers will discuss the benefits and drawbacks of conducting DBR in a museum-setting, including
the following:

- Benefits included generating practitioner and researcher understandings about not only if, but how,
hands-on activities achieve their goals.
- Challenges included modifying a framework during a fast development schedule and managing
expectations among stakeholders.

Importance

Design-based research is a relatively new methodology to study educational products, especially in

informal education settings. However, because of influences like the National Science Foundation, this
methodology is becoming more common and is likely to be implemented as a part of many informal
education projects. This session gives professionals a chance to think about whether and how they
should implement DBR based upon the experiences of those who have already tried this methodology as
well as the promises and pitfalls it holds for our field.

References

Brown, A. (1992). Design experiments: Theoretical and methodological challenges in creating complex
interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178.

Cardiel, C., Pattison, S., Benne, M., & Johnson, M. (2016). Science on the Move: A design-based research
study of informal STEM learning in public spaces. Visitor Studies, 19(1), pp. 1 – 22.

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational
research. Educational Researcher, 32(1), 9–13.

Cobb, P., & Gravemeijer, K. (2008). Experimenting to support and understand learning processes. In A. E.
Kelly, R. A. Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education:
Innovations in science, technology, engineering, and mathematics learning and teaching (pp.
68–95). New York, NY: Routledge.

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design Research: Theoretical and Methodological Issues.
The Journal of the Learning Sciences, 13(1), 15-42.

Kelly, A. E., Baek, J. Y., Lesh, R. A., & Bannan-Ritland, B. (2008). Enabling innovations in education and
systematizing their impact. In A. E. Kelly, R. A. Lesh, & J. Y. Baek (Eds.), Handbook of design
research methods in education: Innovations in science, technology, engineering, and
mathematics learning and teaching (pp. 3–18). New York, NY: Routledge.

National Science Foundation. (2015). Advancing Informal STEM Learning (AISL) Program Solicitation.

82
 Visitor Studies Association - 2018 Conference Abstracts

Alexandria, VA: National Science Foundation.

Sandoval, W. (2014). Conjecture Mapping: An Approach to Systematic Educational Design Research. The
Journal of the Learning Sciences, 23, 18-36.

Additional Links
 
ChemAttitudes:
http://www.informalscience.org/chemattitudes-using-design-based-research-develop-and-disseminate-
strategies-and-materials-support
http://www.nisenet.org/chemistry-kit

Living Liquid:
https://www.exploratorium.edu/cellstoself/projects/living-liquid
https://www.exploratorium.edu/sites/default/files/pdfs/visweek2012.pdf

Making Connections
http://www.informalscience.org/poster-making-connections-exploring-culturally-relevant-maker-experi
ences-through-iterative-cross
https://www.smm.org/make

Science on the Move:

http://www.informalscience.org/sites/default/files/16_04_21_SOTM_2016VisitorStudiesArticle_v19_i1.
pdf
https://programs.omsi.edu/sites/default/files/Science_on_the_Move.pdf

 
Visitor Perceptions on the Well-Being of Animals in Captivity
Mary Jackson, Woodland Zoo
Jerry Luebke, Brookfield Zoo
Manda Smith, Lincoln Park Zoo
Nick Visscher, Denver Zoo

Purpose

It is important to understand not only how visitors are interacting with exhibits, but also how emotional
responses scaffold or mediate visitors’ experiences—primarily with living collections (e.g., Luebke and
Grajal, 2011). Unlike previous research at zoos and aquariums where the focus was on guest
engagement and satisfaction with exhibits, these studies have collected information on the
predispositions of guests and how that factors into emotional interpretations of animal behavior. The
aim of this session is to focus on understanding visitor behavior and perceptions more broadly. In
addition to the foundation of information gathered by the different studies, this session will also factor
in visitors’ pre-existing knowledge about animal welfare, as well as the information on visitors’
responses in situ. By determining how this pre-knowledge of animal welfare influences their emotional
response, this may affect the methods of communication done by zoos and aquariums to achieve better
outreach and trust.

83
 Visitor Studies Association - 2018 Conference Abstracts

Abstract

Accredited institutions provide the animals in their collections with the highest standard of care by
relying on scientific indicators of animal welfare that assess environmental, behavioral, and physical
health. At the same time, however, zoo and aquarium visitors are making personal judgements on the
welfare of these animals based on their own observations and preexisting values and knowledge. This
session will incorporate case studies of visitor perceptions of animal welfare from different zoos and
provide highlights of these efforts. The aim of this session is to focus on understanding the visitor’s
perspective of animal welfare. In addition to the foundation of information gathered by these studies,
this session will also consider visitors’ pre-existing knowledge about animal welfare, emotions visitors
ascribe to animals, and understanding the underlying rationale for these perceptions. By determining
how these variables influence emotional responses, this may affect the methods of communication
done by zoos and aquariums to achieve better outreach and trust. At the end of the session, the
panelists will rotate through discussion points to maximize diversity of perspectives and keep attendees
engaged.

Brookfield Zoo
Brookfield Zoo began an ongoing audience research program in 2014 investigating visitors’ perception of
animal welfare. This presentation will share key highlights of our findings over the last four years and
discuss various strategies that address visitors’ concerns about the welfare of zoo animals.

Lincoln Park Zoo

Since Lincoln Park Zoo’s Audience Research team creation in 2015, staff members have been working on
better understanding how the zoo’s visitors perceive different types of animal behaviors, specifically
from its megafauna. Understanding guests’ prior knowledge to animal behaviors, animal care and
animal welfare will be discussed, as well as how that influenced the visitors experience throughout the
zoo.

Woodland Zoo
Woodland Zoo will present on an embedded, formative measurement strategy that fosters
understanding about visitor perceptions of animal behavior and the emotional states they subsequently
ascribe to the animal, such as excitement or boredom. Utilizing perspective taking, this strategy is
unique in that it was designed to be embedded into program delivery to gather quick, qualitative
information from a youth audience, enabling program presenters to rapidly respond to visitor
perceptions or misperceptions of behavior.

Denver Zoo
Denver Zoo has taken one standard measure of guest satisfaction, Overall Experience Rating (OER), and
modified it for use in monitoring how guests perceive the Zoo’s quality of animal care. This Perceived
Animal Care Rating (PACR) has been used to track perceptions across exhibits, programs, and general
visitation.

Importance

There has recently been an increasing occurrence of articles in the press and social media questioning
the relevancy and welfare of wildlife living in zoos and aquariums. Given this trend, it is imperative that
zoos and aquariums not only need to understand visitors’ perceptions of animal welfare, but also how to
frame their educational and interpretive strategies to facilitate visitors’ positive perceptions of the

84
 Visitor Studies Association - 2018 Conference Abstracts

managed care and welfare of zoo and aquarium animals. This session will share findings from different
zoos concerning studies in visitors’ perceptions of animal welfare and describe how these findings can
inform improvements to visitor engagement and future implications relevant to the practice of
evaluation. Understanding all aspects of the visitor perspective is essential when thinking of new ways to
engage visitors that will foster transparency and strengthen public trust in zoos and aquariums.

References

Luebke, J. F., and A. Grajal. 2011. Assessing mission-related learning outcomes at zoos and aquaria:
prevalence, barriers, and needs. Visitor Studies 14:195–208.
 
Visitor Insights Driving Change and Practice

Keith Snode
Erica Maganti
Marina Guiomar
Caitlin Ballingall

Purpose

Featuring speakers from the Marketing, Group Sales and Evaluation departments, the panel examines
practical models of social listening, as means for growing cross-institutional buy-in at an internal
evaluation capacity. Actively listening to the dynamics of users largely contributes to an ongoing
dialogue with the visitor at large and empowers museums with the adequate tools to become civic,
social and educational mainstays.

Questions driving this inter-departmental study include:

How does data about visitors provide insights into practices of inclusivity, accessibility, and social
responsiveness?
How can research studies be implemented by cultural institutions with varied environments including
staffing models, budgets, and infrastructure?
Learning outcomes for panel attendees are inclusive of:
Concrete applications and outcomes of user experience-driven data.
How prototyping and user-testing protocols can generate taking action at their own cultural institutions.
Service blueprint to visualize all the departments involved in producing a successful visitor experience.

Abstract

This session will highlight internal studies at the Intrepid Sea, Air & Space Museum and presents
perspectives from multiple departments including the research and evaluation team, marketing, and
group sales. Presenters and attendees will briefly explore these recent studies to reveal how collected
data led to change in internal practices and institution-wide.

Additionally, attendees will generate an action plan finding and collecting data needed for their work,
and transitioning existing data into action. Featuring presenters representing multiple departments and
interdepartmental projects, this session will investigate how to use data to inform change, design new
practices and protocols, and transform institutional thinking and planning.

85
 Visitor Studies Association - 2018 Conference Abstracts

In a moderated and conversational format, presenters will examine the varied projects and studies
conducted internally at the Intrepid Museum, will discuss the applications and outcomes of using
collected data, and then invite attendees to participate in smaller group discussions to more deeply
explore the successes, challenges, and opportunities for institutional by-in at their own institutions.

Topics that will be featured through the Intrepid examples:

· Visitor experience data collection
· Concept user-testing for program and outreach development including studies of visitor expectations,
interests, attitudes, and preconceptions
· Engaging in dialogue to respond more authentically, responsibly, and timely with communities served

Prompts for attendee conversation will include:

· What insights can be gained from better understandings demographics of the institutions general
visitors?
· How does data about visitors provide insights into practices of inclusivity, accessibility, and
responsiveness?
· How can understandings about visitor interests, preferences, attitudes, learning patterns, prior
knowledge, and expectations be incorporated into the programming development process?
· How can studies be conducted by cultural institutions with varied environments including staffing
models, budgets, and infrastructure?

Importance

The session directly responds to the conference theme in presenting ways that informal learning
environments can utilize and leverage community and audience feedback to drive change and be more
responsive to expectations, needs, interests, and interpretation. Additionally, utilizing social and on-site
listening, insights may be garnered from the visitor perspective to understand needs and change
institution-wide, allowing for the institution to grow and change with the visitor.

86