Perception and Emotion: How We Recognize Facial Expressions

Author(s): Ralph Adolphs

Source: Current Directions in Psychological Science, Vol. 15, No. 5 (Oct., 2006), pp. 222-226
Published by: Sage Publications, Inc. on behalf of Association for Psychological Science
Stable URL: http://www.jstor.org/stable/20183119
Accessed: 12-01-2016 13:52 UTC

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/
info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact [email protected].

Sage Publications, Inc. and Association for Psychological Science are collaborating with JSTOR to digitize, preserve and
extend access to Current Directions in Psychological Science.

http://www.jstor.org

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions
 CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE

and Emotion
Perception
How We Recognize Facial Expressions
Ralph Adolphs

Division of Humanities and Social Sciences and Division of Biology, California Institute of Technology

and emotion interact, as is borne rarely, directly from the brain in surgical patients). In those
ABSTRACT?Perception
out by studies of how people recognize emotion from facial experiments, the responses to many presentations of emotional

and neurological research has stimuli are across stimuli, and the changes in electrical
expressions. Psychological averaged
elucidated the processes, and the brain structures, that can be timed in relation to stimulus onset.
potential accurately

participate in facial emotion recognition. Studies have Evidence has also come from studies in which viewers were

shown that emotional reactions to viewing can be shown facial salient as

faces expressions subliminally. Especially
and that these reactions in turn, be used to of faces are most in driving nonconscious responses:
very rapid may, pects potent

judge the emotion shown in the face. Recent experiments For instance, subliminal presentation of only the whites of the
have argued that people actively explore facial expressions eyes of fearful faces results in measurable brain activation

in order to recognize the emotion, a mechanism that em

(Whalen et al., 2004). One specific structure involved in such
the instrumental nature social and automatic neural responses is the a struc
phasizes of cognition. rapid amygdala,
ture in the medial temporal lobe that is known to be involved in
KEYWORDS?face perception; faces; emotion; amygdala
many aspects of emotion processing.

Findings such as these have been of interest to psychologists,

An influential psychological model of face processing as a mechanism consistent with two-factor
argued they provide potential

that early of a geometric theories of human behavior. For instance, the theory that affect
perception (construction representation
of the face based on its features) led to subsequently separate and are processes, and that affect
cognitive judgment separate
can some
processing of the identity of the face and of the emotional ex precede cognitive judgment, receives support from the

pression of the face (Bruce & Young, 1986). That model has neuroscience findings (Zajonc, 1980). The data also add detail to
received considerable from neuroscience studies theories of visual consciousness. In one set of studies, emotional
support sug
that the are based on facial were shown to neurological who,
gesting separate processes separable expressions patients
In neuroimaging different because of their brain were unable to report the
neuroanatomical systems: studies, damage, seeing

of the brain are activated in response to emotional ex stimuli. An individual with damage to the visual cortex had
parts

pressions
or
identity changes; and brain damage
can result in "blindsight" for emotional faces: He could discriminate the
in but not in emotion shown in faces even he no
impairments recognizing identity recognizing by guessing, though reported

emotional or the reverse. visual experience of seeing the faces. A neuroimaging experi
expressions,
ment in this same individual revealed that the was
amygdala

SOMEPROCESSING ISRAPID AND AUTOMATICAND

activated differentially by different emotional faces, despite the
absence of conscious visual deGelder,
CANOCCUR INTHE ABSENCEOF AWARENESSOF THE experience (Morris,

STIMULI Weiskrantz, & Dolan, 2001).

These and other studies have suggested a distinction between
subcortical of emotional visual stimuli (e.g., involv
Some in the brain to emotional facial are processing
responses expressions
the and brainstem nuclei such as the
so rapid (less than 100 ms) that they could not plausibly be based ing amygdala superior

on conscious awareness of the stimulus, the responses, colliculus),which may be independent of conscious vision
although
in turn, may contribute to conscious awareness. Evidence comes (Johnson, 2005), and cortical processing, which is usually ac
It is interesting to note
from studies event-related measures of the companied by conscious experience. that
using potentials,
more amphibians and reptiles have only subcortical visual process
brain's electrical recorded at the scalp (or, much
activity
since lack a neocortex. One broad of
ing, they interpretation
these observations is thus that the subcortical route for pro
Address correspondence to Ralph Adolphs, HSS 228-77, Caltech,
CA 91125; e-mail: cessing emotional stimuli is the more ancient one, and that in
Pasadena, [email protected].

222 Copyright ? 2006 Association for Psychological Science Volume 15?Number 5

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions
 Ralph Adolphs

mammals an additional, cortical route has evolved that probably abstract, and perhaps dimensional rather than categorical
as

allows more flexible behaviors based on learning and conscious pect of emotions, for which fear is but one instance?an issue

deliberation. I take up again below.

A final wrinkle has come from psychological studies of the

relationship between attention and emotion. It has been known MANYBRAIN STRUCTURESARE INVOLVED
for some time that salient stimuli can at
emotionally capture

tention, an interaction that makes much adaptive

sense
(Ohman, Just as there is good evidence that the amygdala does more than
Flykt, & Esteves, 2001). But more recent studies have also solely detect fear, there are of course brain structures other than

shown the reverse: that volitionally allocating attention to the amygdala that participate in perceiving emotion from faces.
stimuli can influence their emotional evaluation. Inhibiting at Brain-imaging studies, in particular, have found evidence for a
tention to visual stimuli that are distractors in a search task, for large number of other brain structures that may come into play,

example, results in devaluation of those stimuli when subjects depending on the emotion shown in the face and on the demands
are asked to rate them on affective dimensions of the experimental task.
(Raymond,

Fenske, & Tavassoli, 2003). Emotions thus represent the value of One framework that summarizes these data runs as follows.

stimuli?what or avoid, or behav Visual areas in the cortex encode the face's features
people approach cognitively temporal

iorally, volitionally
or
automatically. and bind them into a global perceptual representation of the
face. Subcortical visual areas as the colliculus,
(such superior
FEARAND DISGUSTMAYBE PROCESSED SPECIALLY prominent in animals such as
frogs) carry out coarser but faster

processing of the face. Both routes, the cortical and the sub
Several studies have found evidence that the amygdala, the cortical, feed visual information about the face into the amyg
subcortical structure discussed earlier, is disproportionately dala. The amygdala then associates the visual representation of

important for processing facial expressions of fear, although by the face with an emotional response in the body, as well as with
no means so. The other emotion for which a in the operation of other brain structures. For instance,
exclusively only changes
similar neuroanatomical specificity has been reported is disgust. it likely triggers autonomie responses (such
as skin-conductance

A of the cortex called the insula motor and response, the autonomie response of on
region represents sympathetic sweating

sensory aspects ofthat emotional response (Calder, Lawrence, & the palms of the hands) to the face, and it also modulates visual
Young, 2001). In lesion studies, damage to the amygdala is attention to the face.

shown to result in impairment in the ability to recognize fear Tracing the path from initial perception of the face to recog
from facial expressions. Similarly, damage to the insula can nizing the emotion it expresses is complicated by feedback loops
result in impairment in the ability to recognize disgust from fa and by multiple pathways that can be engaged. One class of
cial However, remain as influential as are controversial, argues that the
expressions. questions regarding exactly proposals, they
which emotion category or dimension is being encoded in emotional response elicited by the amygdala can in fact be used
these brain since other emotions are often also the viewer to reconstruct about the emotion shown
regions, variably by knowledge

impaired.
in the face. Roughly: if I experience a pang of fear within myself

Especially informative have been studies in a rare patient, upon seeing

a fearful face, I can use the knowledge of my own

SM, who has total damage to the amygdala on both sides of the emotion to infer what the emotional state of the person whose

brain. The subject

can
accurately judge age and gender from face is shown in the stimulus might be. Broader theories in a

faces, and she has no difficulty recognizing familiar individuals similar vein do not restrict themselves to the
amygdala
or to fear,

from their faces. She also has little difficulty recognizing most but more generally propose that we make inferences about other

facial with the notable of fear. When emotional states within ourselves aspects
expressions, exception people's by simulating
asked to judge the fearfulness of faces, SM is selectively and of those states (Goldman & Sripada, 2005). Emotional contagion
severely impaired (Adolphs, Tranel, Damasio, & Damasio, and imitation may be the earliest aspects of such a mechanism

However, other with similar are im that can be seen in infants, and and of
1994). patients damage already empathy theory

paired
on a wider array of emotions; in functional neuroimaging mind may be more complex elaborations that develop later in
studies, activation of the amygdala is seen in response to several life. In support of these ideas, a
provocative study found
recently
emotions in addition to fear; and even SM's highly specific im that constricted pupils make sad facial expressions look more
pairment depends
on the questions she is asked. For example, sad, and that this effect is correlated with an empathie pupillary
when asked to faces into of basic emotions, response in the viewer. In other words, at a sad face with
classify categories looking
SM is rather selectively impaired on fear; but when asked to rate more constricted pupils
causes one's own
pupils
to constrict, an

how the emotion shown in the face is, she is impaired on effect activation of brain known to be
arousing accompanied by regions
all emotions of negative valence (Adolphs, Russell, & Tranel, important for emotion and empathy (Harrison, Singer, Rothstein,
1999). These data suggest that, while the amygdala is critical for Dolan, & Critchley, 2006). That finding is in line with prior re
fear in faces, its role encompasses a broader, more ports facial reactions when viewed facial
recognizing documenting subjects

Volume 15?Number 5 223

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions
 Perception and Emotion

Controls Subject SM a^^^^^^^^^^M processing whereby

we go from the appearance of a face
to its social Yet there is one
meaning. ingredient missing:
It is assumed that all the sensory information on the basis of

which the subsequent processing is triggered is already given

in the stimuli that are shown. The account treats the
subjects
initial acquisition of sensory information from the stimulus as

a process. In fact, it is known from both

passive experiment
and everyday experience that our perception of the social world
is much more instrumental, and involves actively querying
other as we seek out relevant social information in the
people
first place.

A recent study in the subject with total amygdala loss, SM,

0.6 evidence for such a mechanism. As noted
y provided striking
earlier, SM is impaired in recognizing fear from facial expres
< ^H M
sions due to her
amygdala damage, but it has remained unclear

whether this means the amygdala is specialized for fear as such,

0.2- lH^
c BB 2
or whether it means the amygdala is involved in a broader
function that happens to be for fear rec
particularly important

1. Impaired recognition of fear in facial expressions, ognition. To obtain a detailed inventory of how different features
Fig. resulting from
within a fearful face are we conducted an
impaired eye gaze onto faces. The figure shows data from a study in subject processed, experiment
SM, who has hilateral damage to the amygdala. Panel a shows images de in which were shown a number of faces in which
subjects large
picting the regions in faces that viewers use to discriminate fear. Normal
most of the face was occluded and only very small portions of the
subjects, on the left, make notable use of the eyes, as well as some use of the
nose and mouth. face were revealed. these called
By contrast, SM, on the right, fails to make use of the eyes. randomly Viewing stimuli,
Panel b shows eye movements (white Unes) and locations of fixation (white was similar to facial a
"bubbles," viewing expressions through
circles) when participants view facial expressions of fear. Whereas normal
of cardboard
piece into which random little holes had been
subjects explore the faces in a particular pattern, concentrating on the eyes
on some viewers see a part of an ear, or
and the mouth (left), SM either does not explore the face at all or does so in a poked. Thus, trials, might
haphazard fashion that fails to make the normal number of fixations onto
part of the mouth or the eyes. A discrimination task asked
the eye region. The brain damage in subject SM is shown in panel c: A
subjects to discriminate whether these "bubbles" faces looked
horizontal magnetic resonance scan through SM's brain shows the front of
her brain at the top of the image, and the back of her brain at the bottom of happy or fearful. After thousands of trials, we then related the
the image. The two symmetrical black circles near the top middle part of of the on a trial to the re
performance accuracy subject given
the brain image are her lesions of the left and right amygdala. The graph in
in matching gions of the face stimulus that had been revealed on that trial.
panel d plots the accuracy expressions of fear with the word
in a task in which 6 different This method a that shows the strength
'Tear," emotions and words were given to generates resulting image
subjects. While SM is impaired (black bars) compared to healthy controls and reliability of the association between features of the face and
(white bars) when viewing the faces without instruction (in which case she discrimination of the emotion. For certain of the
does not fixate on the eyes of the faces normally), the explicit instruction to example, parts

fixate on
the eyes brought face such as the mouth and eyes would be associated with a
her performance to normal levels (gray bar,
arrow). From
"A Mechanism for Impaired Fear Recognition After better discrimination than other such as the
performance parts
by R. Adolphs, F. Gosselin, T.W. Buchanan, D.
Amygdala Damage," chin or ears (which do not provide discriminative information for
Tranel, P. Schyns, & A.R. Damasio, 2005, Nature, 433, pp. 69, 70, 71.
telling emotions
Copyright 2005, MacMillan Press. Reproduced with permission. apart).
Not surprisingly, we found that the eyes and themouth of faces
are most useful to viewers in discriminating the emotion; that is,

facial shown so that showed on the emotion-dis

expressions?even expressions briefly they they high performance accuracy
are not crimination task when these features were revealed in the face
only subliminally, consciously, perceived (Dimberg,

Thunberg, & Elmehed, 2000). stimuli. Remarkably, subject SM failed to make normal use of
the eyes in this task: Unlike healthy individuals, she did not
PERCEPTIONOF EMOTIONISACTIVE benefit in her recognition of emotion when shown the eye region
of facial expressions (Fig. la). Importantly, this impairment was
The studies discussed above have provided models of how present for all emotional expressions, not just fear: It just hap
several different brain structures interact, at various
points in pens to be the case that the eyes are the feature that most dis
time and often as a function of context and individual differ fear from those of the other basic
tinguishes expressions
ences, to infer another emotional state from observation emotions. The deficit was even more basic, however. It turned
person's
of their overt behavior. This account
emphasizes the construct out that SM failed to direct her gaze to the eye region of faces
ive role that emotion can
play in generating knowledge about the (Fig. lb). These data suggested that at least part of the reason SM
world, and the inferential, creative nature of the fails to recognize fear is that she fails to make use of the ex
emphasizes

224 Volume 15?Number 5

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions
 Ralph Adolphs

pression of the eyes in faces; moreover her impaired use of the that probe the signaling end as well?ideally in naturalistic
eye of faces was associated with her not at the eyes situations face-to-face interactions between It
region looking involving people.
in the first place. Apparently, the amygdala is involved not only is worth stressing that essentially all studies of visual emotion
in the emotional responses to faces, once but also have used rather artificial stimuli: static black
perceived, perception (often

provides the prerequisite for such perception by guiding gaze and-white) photographs of (often posed) facial expressions?or
and attention to emotionally salient of the face at the in some cases
just drawings of facial expressions. It seems rea
regions
outset. This account was in a further in sonable to suppose that our emotional reactions and eye-gaze
supported experiment,
which SM was explicitly instructed to look at the eyes in facial responses to
photographs of faces would be different than those

expressions; when she did so, she was able to discriminate fear to real faces, and that context effects and individual differences

would offer a rich source of variance to with such nat

normally (Fig. Id). explore
This active of emotion the ability to search uralistic stimuli. Does the to clear facial ex
aspect perception, ability produce
for relevant social cues in faces, may also be one of the deficits in pressions of emotion go hand in hand with the ability to recognize
people with autism. Many people with autism also do not direct such expressions? Do these two abilities develop
at the same

their at others, an that may contribute Are we more accurate at our own facial ex
gaze normally impairment ages? recognizing
to their impaired social behavior and point to possible strategies pressions than those of others, as some studies suggest (Elfen
for intervention and rehabilitation. Dysfunction in allocating bein & Ambady, 2003)? The future of research on facial emotion
attention to emotionally relevant stimuli is also likely
to con will depend on integrating the powerful tools provided by cog
tribute to other disorders, from post nitive neuroscience with to human
psychiatric ranging ethological approaches
traumatic stress disorder to schizophrenia. behavior.

FUTUREDIRECTIONS
Recommended Reading
One very intriguing approach for the future would be to combine Adolphs, R. (2002). Recognizing emotion from facial expressions:
Psychological and neurological mechanisms. Behavioral and
the "bubbles" technique (cf. Fig. la) with other dependent
Cognitive Neuroscience Reviews, 1, 21-61.
measures. For instance, viewers could be shown randomly
Adolphs, R., Gosselin, E, Buchanan, T.W., Tranel, D., Schyns, P., &
parts of faces while neural (e.g., functional magnetic
sampled Damasio, A.R. (2005). (See References)
resonance imaging, fMRI) or psychophysiological (e.g., skin Gosselin, E, & Schyns, P.G. (2001). Bubbles: A technique to reveal the
conductance measures of emotional response use of information in recognition tasks. Vision Research,
response) 41,2261
are obtained. to what features in faces drive 2271.
Analogous asking
discrimination of we could LeDoux, J. (1996). The emotional brain. New York: Simon and Schuster.
behavioral-perceptual emotions,
then ask what features drive response, or
psychophysiological
what features drive brain activation. A state-of-the-art
regional

experiment that is now in principle possible could begin

Acknowledgments?Supported by grants from the National
with the response measured by fMRI within a region such
Institute of Mental Health, the Cure Autism Now Foundation,
as the or the insula and use this as the
amygdala independent and the Autism Speaks Foundation.
variable to see how well it predicts the trial-by-trial features

revealed in the randomly sampled face parts (essentially the

reverse of a standard which uses

neuroimaging analysis, REFERENCES
the fMRI as the dependent variable and the stimuli as
response
the Such an would a R., Gosselin, F., Buchanan, T.W., Tranel, D., Schyns, P., &
independent variable). experiment provide Adolphs,
statistical visualization of the of the face that are the Damasio, A.R. (2005). A mechanism for impaired fear recognition
parts
structure. after
amygdala damage. Nature, 433, 68-72.
most effective at driving activation within that brain
Adolphs, R., Russell, J.A., & Tranel, D. (1999). A role for the human
Studies like these could also provide insights into causal
amygdala in recognizing emotional arousal from unpleasant
relationships between psychophysiological response, regional stimuli. Psychological Science, 10, 167-171.
brain activation, and overt behavioral performance. Are
Adolphs, R., Tranel, D., Damasio, H., & Damasio, A. (1994). Impaired
the features of faces that drive activation the same as of emotion in facial bilateral
amygdala recognition expressions following
those that drive skin-conductance If so, this would damage to the human amygdala. Nature, 372, 669-672.
response?
the idea that the the skin-conduct Bruce, V., & Young, A. (1986). Understanding face recognition. British
support amygdala triggers
ance Journal of Psychology, 77, 305-327.
responses.
Calder, A.J., Lawrence, A.D., & Young, A.W. (2001). Neuropsychology
A second critical future direction concerns the social com
of fear and loathing. Nature Reviews Neuroscience, 2, 352-363.
municative role of emotional an aspect
expressions, already
Dimberg, U., Thunberg, M., & Elmehed, K. (2000). Unconscious facial
pointed out by Charles Darwin. To date, the vast bulk of research reactions to emotional facial expressions. Psychological Science,
has focused on the perceptual end. What is needed are studies
11, 86-89.

Volume 15?Number 5 225

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions
 Perception and Emotion

Elfenbein, H.A., & Ambady, N. (2003). When familiarity breeds tation of emotional faces in a cortically blind field. Brain, 124,
accuracy: Cultural exposure and facial emotion recogni 1241-1252.
tion. Journal of Personality and Social Psychology, 85, 276? Ohman, A.,Flykt, A., & Esteves, F. (2001). Emotion drives attention:
290. the snake in the grass. Journal
Detecting of Experimental Psy
Goldman, A.I., & Sripada, C.S. (2005). Simulationist models of face chology: General, 130, 466-478.
based emotion recognition. Cognition, 94, 193-213. J.E., Fenske, M.J., & Tavassoli, N. (2003). Selective attention
Raymond,
Harrison, N.A., T., Rothstein, P., Dolan, R.J., & Critchley, H.D. determines emotional responses to novel visual stimuli.
Singer, Psycho
(2006). Pupillary contagion: Central mechanisms engaged in logical Science, 14, 537-542.
sadness processing. Social Cognitive and Affective Neuroscience, 1, Whalen, P.J., Kagan, J., Cook, R.G., Davis, EC, Kim, H., Polis, S.,
5-17. McLaren, D.G., Somerville, L.H., McLean, A.A., Maxwell, J.S., &
Johnson, M.H. (2005). Subcortical face processing. Nature Reviews Johnstone, T. (2004). Human amygdala responsivity to masked

Neuroscience, 6, 766-774. fearful eye whites. Science, 306, 2061.

L., & Dolan, (2001). Dif Zajonc, R.B. and no
Morris, J.S., deGelder, B., Weiskrantz, R.J. (1980). Feeling thinking: Preferences need infer
ferential and amygdala responses to presen ences. American 35, 151-175.
extrageniculostriate Psychologist,

226 Volume 15?Number 5

This content downloaded from 131.111.5.130 on Tue, 12 Jan 2016 13:52:01 UTC
All use subject to JSTOR Terms and Conditions