Museum Visitor Studies, Evaluation and Audience Research

Randi Korn & Associates, Inc.

118 East Del Ray Avenue
Alexandria, VA
22301

Educational Research:
The Art of Problem Solving

Prepared for the

Solomon R. Guggenheim Museum

New York, NY

Randi Korn & Associates, Inc.

May 2010

EXECUTIVE SUMMARY
INTRODUCTION

The Solomon R. Guggenheim Museum contracted with Randi Korn & Associates, Inc.
(RK&A) for the 2007-2009 study The Art of Problem Solving (APS). The research study
examined the Guggenheims long-standing teaching artist in residency program Learning
Through Art (LTA); specifically, the study was designed to explore LTAs pedagogical
model of building problem-solving skills using art and to determine the effectiveness of
the LTA program in teaching problem-solving skills. This study was made possible
through a three-year Arts in Education Model Development and Dissemination
(AEMDD) grant from the U.S. Department of Education. The major findings from the
study are presented below by methodology.
The findings presented here are among the most salient. Please read the
body of the report for a more comprehensive presentation of findings.

STUDENT QUESTIONNAIRES
A total of 418 fifth-grade students from six schools completed a questionnaire at the start of the school
year (pre-test) and again at the conclusion of the school year (post-test); one-half of students received
the LTA program (treatment student n = 209), while one-half did not (control student n = 209).
Questionnaire findings describe students attitudes about school and art, schoolwork and art work
practices, and perceptions of a good artist. For treatment group students, findings also describe
students evaluations of various aspects of LTA. Findings are as follows:
BASELINE FINDINGS

At baseline, control and treatment students expressed positive attitudes about school, art,
and art museums. They also responded positively to questions about art work practices and
their ideas about artists as well as about solving math problems, puzzles, and difficulties with
art projects.

Of all the responses, there was only one significant difference between control and treatment
students; when asked about art project work practices, treatment group students were more
likely than were control group students to say they would keep working on an art project at
school even if they made mistakes.

PRE-TEST TO POST-TEST CHANGES

Control and treatment students expressed less positive attitudes about school at the end of
the school year (ratings on four of the seven statements about school decreased significantly
from pre-test to post-test).

Control and treatment students attitudes about art remained about the same over the year
(treatment students ratings for all six statements about art did not differ, while control
students ratings for one of the six statements improved significantly).

ii Randi Korn & Associates, Inc.

Control and treatment students attitudes about art museums remained the same over the
year (students ratings of the five statements about art museums did not change significantly
from pre-test to post-test).

Control and treatment students strategies for solving a math problem and puzzle remained
about the same from pre-to post-test. However, more control and treatment students
reported asking another student or family member for help when encountering difficulties
solving a math problem or a puzzle (as well as an art project) on post-test than pre-test.

Control and treatment students strategies for solving a problem with an art project stayed
about the same during the year. However, treatment students responses for two of the 12
statementsdo you feel mad? and do you try to fix it by using different materials?
improved.

Control and treatments students reported more favorable art work practices at post-test than
pre-test (students responses for five of the 12 statements improved significantly from preto post-test; additionally, more control students reported that they wouldnt use the materials
with which they are most comfortable on post-test than pre-test, while treatment students
responses to this item remained the same; finally, more treatment students reported planning
for their art project on post-test than pre-test while control students responses remained the
same).

Control and treatment students ideas about the qualities of a good artist improved over the
year (responses for three of the eight sentence completions about the qualities of a good
artist improved significantly from pre- to post-test while their other responses stayed the
same).

DESIGN-A-CHAIR STUDENT INTERVIEWS AND OBSERVATIONS

A total of 447 fifth-grade students from six schools completed the Design-a-Chair activity; one-half of
students received the LTA program (treatment student n = 218), while one-half did not (control student
n = 229). As part of the activity, students were presented with a bag of art materials and asked to design
a chair using at least three different materials in 15 minutes. Specially-trained data collectors observed
students as they did the activity and interviewed them immediately after. The Design-a-Chair activity
was administered at the end of each school year in May (May 2008 and May 2009). Findings are as
follows:

Treatment students scored higher than control students on Connection of Ends and Aims I,
a measure that explored the extent to which students make intentional decisions and choices
when working on art projects.

Treatment students scored higher than control students on Flexibility, a measure that
explored the extent to which students approach accidents, difficulties, and frustration with
focus, patience, and further exploration.

Treatment students scored higher than control students for Resource Recognition III; that
is, when asked what other materials they would have liked to use, treatment students named
materials other than those already available.

Treatment students scored lower than control students on Experimentation II, a measure
that explored the extent to which students tested the properties of the materials when
working on art projects.

iii Randi Korn & Associates, Inc.

In all other areasImagining, Experimentation I, Resource Recognition I & II, Connection

of Ends and Aims II & IIIthere was no statistically significant difference between
treatment and control students.

STUDENT CASE STUDIES

RK&A conducted 25 case studies over the 2007-2008 and 2008-2009 school years. Through case
studies, RK&A looked at how students responded to LTAparticularly in relation to five problemsolving criteriaas well as investigated students performance outside LTA. As part of each case study,
RK&A observed the student during two or three LTA lessons between December and April,
interviewed the student about an artwork s/he created during LTA, and interviewed the students
classroom teacher. Findings are as follows:

Some case study students struggled in class but excelled in LTA; the converse is also true.

About one-half of case study students achieved low levels of Experimentation, which
explores the extent to which students try a number of materials, tools, techniques, or ideas.

More than one-half of case study students achieved high levels of Connection of Ends and
Aims, which explores the extent to which students make intentional decisions and choices in
when working on their art project.

Case study students achievement of Imagining, Flexibility, and Resource Recognition ranged
approximately evenly along a continuum from high to low achievement.

TEACHING ARTIST OBSERVATIONS

RK&A observed each of the three teaching artists during the 2007-2008 school year and each of the
three teaching artiststwo of whom also taught during the 2007-2008 school yearduring the 20082009 school year. Each teaching artist was observed nine times between December and April in each
school year (e.g., each teaching artist was observed three times while teaching to three classrooms).
Observations explore the extent to which teaching artists employ general LTA teaching strategies
considered best practice by the Guggenheimas well as strategies for cultivating students problemsolving skills. Findings are as follows:

Of the four general LTA teaching strategies, teaching artists least frequently referred to
students as artists; this occurred in less than two-thirds of observations

Of the four lessons that cultivate students problem-solving abilities, teaching artists most
frequently taught students to think intentionally and make deliberate choices; they did so in
all observations

Of the four lessons that cultivate students problem-solving abilities, teaching artists least
frequently taught students to see problems/mistakes/challenges as opportunities; this
occurred in slightly more than one-half of observations.

TEACHING ARTIST INTERVIEWS

In June following the 2007-2008 school year and the 2008-2009 school year, RK&A conducted
telephone interviews with the three participating teaching artists from each school year. Two of the
teaching artists who participated in the study during the 2007-2008 school year also participated in the

iv Randi Korn & Associates, Inc.

study during the 2008-2009 school year; thus, six interviews were conducted with four teaching artists.
Findings are as follows:

Teaching artists described their relationship with classroom teachers as one of the greatest
challenges of LTA. Teaching artists had trouble communicating with teachers and gaining
their support.

Teaching artists described positive experiences with the students. They said that students
thrived in the program and felt that they enjoyed it and were even empowered by it.

Teaching artists valued the professional development the Guggenheim provided, specifically
the teaching strategies they acquired.

Teaching artists said that their experiences with LTA made them more reflective teachers.

v Randi Korn & Associates, Inc.

DISCUSSION
INTRODUCTION

Over the years, the Solomon R. Guggenheim Museums LTA has used evaluation for
institutional learning and program improvement. Evaluation and self assessment are
embedded in the program, and used successfully among the teaching artists. As a result
of this iterative cycle of continual improvement, LTA has developed into a highly
effective program. In a previous study, titled Teaching Literacy Through Art (TLTA), funded
by the U.S. Department of Educations Arts in Education Model Development and
Dissemination (AEMDD) grant, participation in LTA was shown to have a positive
impact on students literacy and critical thinking skills (RK&A, 2007). This follow-up
study, also funded by the AEMDD grant, sought to examine another area important to
LTAproblem solving. Findings of this important study, titled The Art of Problem
Solving, are discussed below.
ATTITUDES TOWARD SCHOOL, ART, AND ART MUSEUMS
The research study hypothesized that students who participate in LTA would have more positive
attitudes toward school, art, and art museums than those who do not participate in LTA. Similar to the
2007 TLTA study, findings demonstrate that LTA did not have a significant impact on students
attitudes toward school, art or art museumsthere were no statistically significant differences between
the treatment and control students. Generally, students in both groups had positive attitudes in these
three areas at the beginning of the school year, while students (in both groups) attitudes toward school
were significantly less positive at the end of the school year than at the beginning (attitudes toward art
and art museums remained unchanged). Such a finding is not surprising, considering that other studies
have shown that negative attitudes toward school begin to develop around middle school (Anderman &
Midgley, 1998; Eccles & Midgley, 1989; Hogsten & Peregoy, 1999). The students in this study were fifth
graders, and thus, on the cusp of middle school by the end of the school year.
PROBLEM-SOLVING BELIEFS AND ABILITIES
While this study examined attitudes as described above, the primary focus of the research was the
hypothesis that students who participate in LTA will develop greater problem-solving skills than those
who do not participate in LTA. Problem solving has long been an important objective for LTA, but its
relevance to todays world has been heightened in the current global economy, which demands that our
workforce develop higher-order thinking skills like critical thinking and problem solving. These kinds of
skills have been coined 21st Century Skills and embraced by educators, policy makers, and business
leaders as essential to success in todays society. The premise is that thinking skills are far more valuable
than encyclopedic knowledge of content. 21st Century Skills have also taken hold in the museum
field. A 2009 publication by the Institute of Museum and Library Services (IMLS) emphasizes that
museums have an important role in helping citizens build thinking skills, listing problem solving among
the top of its learning and innovation skills, along with critical thinking and creativity (IMLS, 2009).
This notion of 21st Century Skills is not new to art museums. For decades, many art museums have
sought to translate the power of art (looking at, thinking about, and making art) into programs meant to
vi Randi Korn & Associates, Inc.

develop participants thinking skills through such means as inquiry, experimentation, and observation.
Small-scale studies, evaluations, and arts education advocates have argued that the methods of
instruction typically delivered in arts-based programming has a high probability of cultivating higherorder habits of mind and thinking skills (Constantino, 2002; Perkins & Jay 1995; Pitri, 2003; Tishman,
2002; Kowalchuk, 1999). However, few art programs have had the resources to conduct rigorous
studies to attempt to demonstrate the positive effects of arts programming on student learning
(McCarthy, Ondaatje, Zakaras, & Brooks, 2004). A couple of large-scale, quantitative studies have
examined the impact of multi-arts programming on student learning, and findings were promising. For
example, Burton et al. (1999) showed that students attending arts-rich schools outscored students in
arts-poor schools in measures of creative thinking. Catterall and Waldorf (1999) found that students
who were highly involved in the arts outperformed students who had low arts involvement on a variety
of academic measures. But these examples are few and far between.
It is within this context that LTA sought to show a link between its programming and problem solving
skills. LTA hypothesized that the program would give students the skills necessary to persistently and
adaptively work through problems. However, before moving forward on this research, LTA had to first
clearly define what it meant by problem solving, given that there is little available consensus on the
meaning of the term. While problem solving may be simply defined as the analysis and transformation
of information toward a goal (Lovett, 2002), a review of literature quickly reveals that no standard set
of behaviors is associated with problem solving. Rather, many skills and behaviors are associated with
problem solvingincluding exploration, analysis, synthesis, curiosity, persistence, planning, risk taking,
and elaboration (Costa & Kallick, 2000; IMLS, 2009; Weisburg, 2006). Moreover, there is much overlap
among definitions of problem solving, creativity, and critical thinking. Thus, the first step in the
research study was to identify the exact characteristics of problem solving as manifested in the LTA
program. In the projects first year, LTA staff assembled the Art of Problem Solving (APS) advisory team
(see Appendix U) to develop a rubric to define the behaviors associated with problem solving in LTA.
This process was long, arduous, and exhilarating. In the end, the APS advisory team created a list of six
skills to most accurately define problem solving in LTA. They are listed and described, in brief, below
(for the complete rubric, see Appendix L).
1. Imagining Students place themselves within the task to the extent that he/she can envision the
problem beyond the assignment given, including opportunities and constraints.
2. Experimentation Students try a number of materials and/or tools as they create their art project.
3. Flexibility Students approach accidents, difficulties, and frustration with focus, patience, and
further exploration.
4. Resource Recognition Students pay attention to the resources provided and seek out resources
appropriate for the task.
5. Connection of Ends and Aims Students describe intentional and deliberate decisions and choices
they made in creating art.
6. Self-reflection Students express explicit and thoughtful opinions or critiques of their art project
and/or identify problems/difficulties.
In the end, the study provides strong evidence that LTA enhanced students abilities in three of the six
areas of the problem-solving rubric: 1) Flexibility, 2) Connections of Ends to Aims, and 3) Resource
vii Randi Korn & Associates, Inc.

Recognition1. LTA students scored significantly higher than non-LTA students in these three areas on
the problem-solving activity (see page 8 for a full description of the Design-a-Chair activity).
Furthermore, questionnaire findings also show that participation in LTA is correlated with more
positive attitudes in the areas of Flexibility (i.e., not giving up when encountering problems) and
Connections of Ends and Aims (i.e., planning). Moreover, case study findings show that students
participating in LTA exhibited Connections of Ends and Aims more often than the other problemsolving skills during LTA sessions, and deliberate decision-making was a strategy that the teaching artists
used consistently and frequently. On the other hand, evidence indicates that LTA did not affect
students abilities in the other areas of the rubric, including: Imagining, Experimentation, and Selfreflection. In each of these areas, treatment students did not score significantly higher than control
students on the problem-solving activity. In fact, in one areaExperimentationcontrol students
scored higher than treatment students. Notably, case study findings provide support for this last
finding, in that LTA case study students infrequently experimented during LTA sessions.
CONCLUSION
What do these findings tell us? In plain language, the findings indicate that students who participate in
LTA are more likely to plan, persist, be deliberate and thoughtful, approach difficulties with focus, and
have greater knowledge of art materials. On the other hand, students who participate in LTA are no
more likely to imagine beyond the task at hand or self critique, and they are less likely to try a number of
materials. Though these findings are not entirely positive, they are encouraging and have positive
implications, especially when considered in the context of 21st Century Skills. For instance, though
the study did not test for transfer, one could hypothesize that Connections of Ends to Aims and
Flexibility, in particular, are skills with wide application across students academic careers and highly
relevant to 21st Century Skills described earlier. Planning, intentionality, adaptability, and persistence
are all skills that will benefit students in school and as they move into the workforce.
Ultimately, the study raises important questions about what problem solving is, especially in the context
of arts programming. This study took preliminary steps in defining and measuring problem solvinga
complex thinking skilland raised interesting questions for further research. These questions include:
How can teachers cultivate students abilities to experiment, imagine, and self-reflect? Is the ability to
experiment, imagine, and self-reflect linked to developmental stages, and if so, at what age is it
appropriate to expect children to experiment, imagine, and self-reflect? How does achievement of
Flexibility and Connection of Ends and Aims transfer to other subjects or real world experiences? Are
there stages to problem solving in the way there are stages to ones aesthetic development? How does
Resource Recognition relate to Experimentation (e.g., does achievement of resource recognition help or
hinder students achievement of Experimentation)?
Ultimately, these findings beg the question: If LTA affects half of the six problem-solving skills defined
by the APS advisory team, does that mean LTA does not affect students problem-solving skills? As
discussed previously, problem solving is a somewhat nebulous term without a standard list of behaviors
to use as a barometer of its accomplishment. Its definition is at least somewhat dependent on the
context in which it is used, and it is often referred to interchangeably with other higher-order skills, like
creativity and critical thinking. While the APS advisory team set out to capture problem solving in
terms accepted in the field and most consistent with what one would expect from LTA, the definition of
problem solving must be further refined as researchers and practitioners continue to explore what
problem solving is, particularly given its prominence in literature about 21st Century Skills.
1 Resource Recognition was broken into three categories and treatment students scored significantly higher on Resource
Recognition III, which assessed whether students asked for materials not provided.

viii Randi Korn & Associates, Inc.

REFERENCES
Anderman, L. H., & Midgley, C. (1998). Motivation and middle school students. Champaign, IL: ERIC
Clearinghouse on Elementary and Early Childhood Education.
Burton, J., Horowitz, R., & Abeles, H. (1999). Learning in and through the arts: Curriculum
implications. In E. B. Fisk (Ed.), Champions of change: The impact of the arts on learning (pp. 35-46).
Washington, DC: The Arts Education Partnership and The Presidents Committee on the Arts and
the Humanities.
Catterall, J. & Waldorf, L. (1999). Chicago Arts Partnerships in Education: Summary evaluation. In E. B.
Fiske (Ed.), Champions of change: The impact of the arts on learning (pp. 47-62). Washington, DC: The Arts
Education Partnership.
Constantino, T. E. (2002). Problem-based learning: A concrete approach to teaching aesthetics. Studies
in Art Education, 43(3), 219-231.
Costa, A. L, & Kallick, B. (2000). Habits of mind: A developmental series (Book I: Discovering and exploring
habits of mind; Book II: Activating and engaging habits of mind; Book III: Assessing and reporting growth in
habits of mind; Book IV: Integrating and sustaining habits of mind). Alexandria, VA: Association for
Supervision and Curriculum Development.
Eccles, J. S. and Midgely, C. (1989). Stage-environment fit: Developmentally appropriate classrooms for
early adolescents. In R. E. Ames & C. Ames (Eds.), Research on motivation in education. New York, NY:
Academic Press.
Hogsten, J. F., & Peregoy, P. A. (1999). An investigation of reading attitudes and self-perceptions of students reading
on or below grade level: Research report. Charlottesville, VA: University of Virginia.
Institute of Museum and Library Services. (2009). Museums, libraries, and 21st century skills. Washington,
DC: Institute of Museum and Library Services.
Kowalchuk, E. (1999). Recognizing and using higher-order approaches to teaching art, Art Education,
52(6), 13-18.
Lovett, M. C., (2002). Problem solving. In D. Medin (Ed.), Stevens' handbook of experimental psychology:
Volume 2, Memory and cognitive processes (pp. 317362). New York: Wiley.
McCarthy, K. F., Ondaatje, E. H., Zakaras, L. & Brooks, A. (2004). Gifts of the muse: Reframing the debate
about the benefits of art. Santa Monica, CA: RAND Corporation.
Pitri, E. (2003). Conceptual problem solving during artistic representation. Art Education, 56(4), 19-23.
Randi Korn & Associates, Inc. (2007). Teaching literacy through art: Synthesis of 2004-05 and 2005-06 studies.
New York, NY: Solomon R. Guggenheim Museum.
Tishman, S., Jay, E., & Perkins, D. (1993). Teaching thinking dispositions: From transmission to
enculturation. Theory Into Practice. 32(3), 147-153.
Weisburg, R. W. (2006). Creativity: Understanding and Innovation in Problem Solving, Science, Invention, and the
Arts. Hoboken, New Jersey.
ix Randi Korn & Associates, Inc.