Chapter 4

COGNITIVE ASPECTS
 Overview
• What is cognition?
• Why it is important to understand in HCI
• Describe how cognition has been applied to
interaction design
• Explain what are mental models and how to
stimulate them
• Cover relevant theories of cognition

www.id-book.com
 What is cognition?
• Thinking, remembering, learning,
daydreaming, decision-making, seeing,
reading, talking, writing…

• Ways of classifying them at a higher level

– Experiential vs reflective cognition (Norman,
1993)
– Fast vs slow thinking (Kahneman, 2011)

www.id-book.com
 Which involves fast vs slow
thinking?
• 2+2 =
• 21 x 29 =
• What color eyes do you have?
• How many colors are there in the rainbow?
• How many months in the year have 31 days?
• What is the name of the first school you
attended?

www.id-book.com
How can understanding cognition help?

• Provides knowledge about what users can

and cannot be expected to do

• Identifies and explains the nature and

causes of problems users encounter

• Provides theories, modelling tools,

guidance and methods that can lead to
the design of better interactive products
www.id-book.com
 Cognitive processes
• It is important to note that many of these
cognitive processes are interdependent:
1. Attention • Several may be involved for a given activity.
• It is rare for one to occur in isolation.
2. Perception
For example, when you try to learn material for
3. Memory an exam, you need to attend to the material,
perceive and recognize it, read it, think about it,
4. Learning and try to remember it.

5. Reading, speaking and listening

6. Problem-solving, planning, reasoning and
decision-making

www.id-book.com
 1. Attention
• Selecting things to concentrate on at a point in time from
the mass of stimuli around us
• Allows us to focus on information that is relevant to what
we are doing
• Involves audio and/or visual senses
• Focussed and divided attention
– enables us to be selective in terms of the mass of
competing stimuli but limits our ability to keep track of
all events
• Design recommendation
– Information at the interface should be structured to
capture users’ attention
– Use perceptual boundaries (windows), colour, reverse
video, sound and flashing lights etc.
www.id-book.com
Activity: Find the price for a double room
at the Quality Inn in Pennsylvania

www.id-book.com
Activity: Find the price of a double room at the
Holiday Inn in Columbia

www.id-book.com
 Activity
• Tullis (1987) found that the two screens produced
quite different results
– 1st screen - took an average of 5.5 seconds to search
– 2nd screen - took 3.2 seconds to search

• Why, since both displays have the same density of

information (31%)?

• Spacing
– In the 1st screen the information is bunched up together, making it
hard to search
– In the 2nd screen the characters are grouped into vertical categories
of information making it easier

www.id-book.com
 Multitasking and attention
• Is it possible to perform multiple tasks without one or
more of them being detrimentally affected?

• Multitasking can cause people to lose their train of

thought, make errors, and need to start over

• Ophir et al (2009) compared heavy vs light multi-

taskers
– heavy were more likely to being unfocussed than those who
infrequently multitask
– heavy multi-taskers are easily diverted and find it difficult to filter
irrelevant information
www.id-book.com
 Multitasking experiment
• Lotteridge et al. (2015) conducted another study
involving writing an essay under 2 conditions:
relevant or irrelevant information

– heavy multitaskers were easily diverted, but able to

put this to good use if the confusing (diverting)
sources were relevant to the task in hand

– irrelevant information was found to negatively impact

task performance

www.id-book.com
 Multitasking at work
• Increasingly common for workers to
multitask
– e.g. hospital workers have to attend to multiple
screens in an operating room that provide new
kinds of real-time information
– requires clinician’s constant attention to check
if any data is unusual or inconsistent
– need to develop new attention and scanning
strategies

www.id-book.com
Is it OK to use a phone when
driving?

www.id-book.com
 No!
– Driving is very demanding
– Drivers are prone to being distracted
– There is a significant chance of causing accidents
– Drivers’ reaction times are longer to external events
when talking on the phone in a car (Caird et al., 2018)
– Phone drivers rely more on their expectations about
what is likely to happen next as conversation taking
up their attention
– Response time is slower to unexpected events
(Briggs et al., 2018)
– Drivers often try to imagine what the other person’s
face is like – whom they are talking to
• this competes with the processing resources needed to
enable them to notice and react to what is in front of them
www.id-book.com
Are hands-free phones safer to use
when driving?
• No, as same type of cognitive processing is
happening when talking

• Same happens when talking with front seat

passenger
– but both can stop in mid-sentence if see a hazard
meaning driver can switch immediately to the road
– So less dangerous talking to a front seat passenger
than a remote person
– a remote person on the end of a phone is not aware
of to what the driver is seeing and will carry on the
conversation when there is a hazard
– makes it difficult for the driver to switch all their
attention to the road
www.id-book.com
Design implications for attention
• Context: Make information prominent when it
needs attending to at a given stage of a task
• Use techniques to achieve this
– e.g. color, ordering, spacing, underlining, sequencing
and animation
• Avoid cluttering visual interfaces with too much
information
• Consider designing different ways of supporting
effective switching and returning to an interface
www.id-book.com
 2. Perception
• How information is acquired from the world and
transformed into experiences
• Obvious (clear) implication is to design
representations that are readily perceivable, e.g.
– Text should be readable
– Icons should be easy to distinguish and read

www.id-book.com
Is color contrast good? Find Italian

www.id-book.com
Are borders and white space better?
Find french

www.id-book.com
 Summary of an Activity
• Weller (2004) found people took less time to
locate items for information that was grouped
– using a border (2nd screen) compared with using color contrast
(1st screen)

• Some argue that too much white space on web

pages is disadvantageous to search process
– Makes it hard to find information

• Do you agree?

www.id-book.com
Activity: Which is easiest to read
and why?

What is the time? What is the time?

What is the time? What is the time?

What is the time?

www.id-book.com
Design implications for Perception
• Icons should enable users to readily distinguish
their meaning
• Bordering and spacing are effective visual ways
of grouping information
• Sounds should be audible and distinguishable
• Research proper color contrast techniques
when designing an interface
– Yellow on black or blue is fine
– Yellow on green or white is a no-no
• Haptic feedback should be used judiciously

www.id-book.com
 3. Memory
• Involves recalling various kinds of knowledge that allow
people to act appropriately
– e.g. recognize someone’s face, remember someone’s name
• First encode and then retrieve knowledge
• We don’t remember everything - involves filtering and
processing what is attended to
• Context is important as to how we remember (i.e. where,
when)
• We recognize things much better than being able to recall
things
• We remember less about objects we have photographed
than when we observe them with own eyes (Henkel, 2014)

www.id-book.com
 Processing in memory
• Encoding is first stage of memory
– determines which information is attended to in the
environment and how it is interpreted
• The more attention paid to something…
• The more it is processed in terms of thinking about it
and comparing it with another knowledge…
• The more likely it is to be remembered
– e.g. when learning about HCI, it is much better to reflect upon
it, carry out exercises, have discussions with others about it,
and write notes than just passively read a book, listen to a
lecture or watch a video about it

www.id-book.com
 Context is important
• Context affects the extent to which information
can be subsequently retrieved
• Sometimes it can be difficult for people to recall
information that was encoded in a different
context:
– “You are on a train, and someone comes up to you and says
hello. You don’t recognize him for a few moments but then
realize it is one of your neighbours. You are only used to
seeing your neighbour in the hallway of your apartment block
and seeing him out of context makes him difficult to recognize
initially”

www.id-book.com
 Activity
• Try to remember the dates of your grandparents’ birthday
• Try to remember the cover of the last two books you read
• Which was easiest? Why?
• People are very good at remembering visual cues about
things
– e.g. the color of items, the location of objects and marks on an object

• They find it more difficult to learn and remember arbitrary

(uninformed) material
– e.g. birthdays and phone numbers

www.id-book.com
 Recognition versus recall
• Command-based interfaces require users to
recall from memory a name from a possible set
of 100s

• Graphical interfaces provide visually-based

options (menus, icons) that users need only
browse through until they recognize one

• Web browsers provide tabs and history lists of

visited URLs that support recognition memory

www.id-book.com
The problem with the classic ‘72’

• George Miller’s (1956) theory of how much

information people can remember
• People’s immediate memory capacity is very
limited to 7 2
• Has been applied in interaction design when
considering how many options to display
• But is it a good use of a theory in HCI?
• Is it helpful?

www.id-book.com
When creating an interface should the designer
• Present only 7 options on a menu
• Display only 7 icons on a tool bar
• Have no more than 7 bullets in a list
• Place only 7 items on a pull-down menu
• Place only 7 tabs on the top of a website page?
• Not necessarily..

www.id-book.com
 The reason is…
• People can scan lists of bullets, tabs, menu
items for the one they want
• They don’t have to recall them from memory
having only briefly heard or seen them
• So can have more than 9 at the interface (e.g.
history lists of websites visited)
• Sometimes a small number of items is good
– e.g. smart watch displays
• Depends on task and available screen estate
www.id-book.com
 Personal Information management
• Is a growing problem for many users
– they accumulate a vast numbers of documents, images,
music files, video clips, emails, attachments, bookmarks,
etc.,
– where and how to save them all, then remembering what
they were called and where to find them again

– naming most common means of encoding them

– but can be difficult to remember, especially when have

10,000s

– How might such a process be facilitated taking into

account people’s memory abilities?
www.id-book.com
 Personal Information management

• Bergman and Whittaker’s 3 interdependent processes

model (2016) to help people manage their stuff:
(i) how to decide what stuff to keep
(ii) how to organize it when storing
(iii) which strategies to use to retrieve it later
• Most common approach is to use folders and naming
• Strong preference for scanning across and within folders when
looking for something
• Search engines only helpful if know the name of the file
• Smart search engines help with listing relevant files for partial
name or when type in first letter
www.id-book.com
 Memory load
• Online/mobile and phone banking now require users to provide
multiple pieces of information to access their account
– e.g. ZIP code, birthplace, a memorable date, first school attended
– Known as multifactor authentication (MFA)

• Why?
– Increased security concerns
• Password managers such as LastPass have been developed -
only need one master password
– Reduce stress and memory load on users
• Passwords could become inexistant with the widespread use of
biometrics and computer vision algorithms
www.id-book.com
 Memory aids
• SenseCam developed by Microsoft Research
Labs (now Autographer)
– a wearable device that irregularly takes photos
without any user intervention while worn
– digital images taken are stored and revisited using
special software
– has been found to improve people’s memory,
especially those suffering from dementia
• Other aids include RemArc which triggers
long term memory using old BBC materials

www.id-book.com
SenseCam

www.id-book.com
 Design implications for Memory
• Reduce cognitive load by avoiding long and
complicated procedures for carrying out tasks

• Design interfaces that promote recognition

rather than recall

• Provide users with various ways of labelling

digital information to help them easily identify it
again
– e.g. folders, categories, color, flagging, time stamping

www.id-book.com
 4. Learning
• Involves the accumulation of skills and knowledge
involving memory
• Two main types:
– Incidental learning (e.g. recognizing people’s faces, what you did
today)
– intentional learning (e.g. studying for an exam, learning to cook)
– Intentional learning is much harder!
– Many technologies have been developed to help (e.g. multimedia,
animations, VR)

• People find it hard to learn by following

instructions in a manual
• Prefer to learn by doing

www.id-book.com
 Design implications
• Design interfaces that encourage exploration
• Design interfaces that constrain and guide
learners
• Dynamically linking concepts and
representations can facilitate the learning of
complex material

www.id-book.com
 5. Reading, speaking, and
listening
• The ease with which people can read, listen, or
speak differs
– Many prefer listening to reading
– Reading can be quicker than speaking or listening
– Listening requires less cognitive effort than reading or
speaking
– Dyslexics have difficulties understanding and
recognizing written words

www.id-book.com
 Applications
• Voice user interfaces allow users to interact with
them by asking questions
– e.g. Google Voice, Siri, Alexa

• Speech-output systems use artificially generated

speech
– e.g. written-text-to-speech systems for the visually impaired

• Natural-language systems enable users to type

in questions and give text-based responses
– e.g. chatbots

www.id-book.com
 Design implications
• Speech-based menus and instructions
should be short
• Emphasize the pitch of artificially
generated speech voices
– they are harder to understand than human
voices

• Provide opportunities for making text large

on a screen
www.id-book.com
 6. Problem-solving, planning, reasoning
and decision-making
• All these processes involve reflective cognition
– e.g. thinking about what to do, what the options
are, and the consequences

• Often involves conscious processes, discussion

with others (or oneself), and the use of artifacts
– e.g. maps, books, pen and paper

• May involve working through different scenarios

and deciding which is best option

• Weighing up alternatives
www.id-book.com
 Design implications
• Provide information and help pages that are
easy to access for people who wish to
understand more about how to carry out an
activity more effectively (e.g. web searching)

• Use simple and memorable functions to support

rapid decision-making and planning

www.id-book.com
 Dilemma
• The app mentality is making it worse for people to
make their own decisions because they are
becoming risk unfavorable (Gardner and Davis,
2013)
– Instead they now rely on the assembly of apps
– This makes them increasingly worried
– Unable to make decisions by themselves
– Need to resort to looking up info, getting other’s
opinions on social media and comparing notes
• Do you agree?
• Did it happen to you when deciding which
university/school to go to?
www.id-book.com
 Cognitive frameworks
• These are used to explain and predict user
behavior at the interface
– based on theories of behavior
– focus is on mental processes that take place
– Also use of artifacts and representations
• Most well known are:
a. Mental models
b. Gulfs of execution and evaluation
c. Distributed cognition
d. External and embodied cognition
www.id-book.com
 a. Mental models
• Users develop an understanding of a system
through learning about and using it
• Knowledge is sometimes described as a mental
model:
– how to use the system (what to do next)
– what to do with unfamiliar systems or unexpected
situations (how the system works)
• People make interpretations using mental models
of how to carry out tasks

www.id-book.com
 Mental models
• Craik (1943) described mental models as:
– internal constructions of some aspect of the external
world enabling predictions to be made
• Involves unconscious and conscious processes
– imagery and analogies are activated
• Deep versus shallow (low) models
– e.g. how to drive a car and how it works

www.id-book.com
 Everyday reasoning and mental models

Remember these examples:

(a) You arrive home on a cold winter’s night to a cold house.

How do you get the house to warm up as quickly as
possible? Set the thermostat to be at its highest or to the
desired temperature?

(b) You arrive home starving hungry. You look in the fridge
and find all that is left is an uncooked pizza. You have an
electric oven. Do you warm it up to 375 degrees first and
then put it in (as specified by the instructions) or turn the
oven up higher to try to warm it up quicker?

www.id-book.com
 Heating up a room or oven that is
thermostat-controlled
• Many people when asked (a) choose the first option
• Why?
– Think It will heat the room up quicker
– General valve theory, where ‘more is more’ principle is
generalised to different settings (e.g. gas pedal, gas cooker, tap,
radio volume)
– Burt it is a wrong mental model as thermostats based on model
of on-off switch model
• Many people when asked (b) choose the first option
– Electric ovens work on the same principle as thermostats
• Most of us have erroneous mental models (Kempton,
1996)
www.id-book.com
 Erroneous mental models
• Lots of people hit the button for elevators
and pedestrian crossings at least twice
– Why? Think it will make the lights change faster or
ensure the elevator arrives!

• What kinds of mental models do users have for

understanding how interactive devices work?
– poor, often incomplete, easily confusable, based on
inappropriate analogies and misconception (Norman,
1983)

www.id-book.com
How can UX be designed to help
people build better mental models?
• Clear and easy to use instructions
• Appropriate tutorials and contextual sensitive
guidance
• Provide online videos and chatbot windows
when needing help
• Transparency – to make interfaces natural to
use
• Affordances of what actions an interface allows
– e.g. swiping, clicking, selecting

www.id-book.com
b. Gulfs of execution and evaluation
• The ‘gulfs’ explicate the gaps that exist between the
user and the interface
• The gulf of execution
– the distance from the user to the physical system
• The gulf of evaluation
– the distance from the physical system to the user
• Bridging the gulfs can reduce cognitive effort required
to perform tasks
• Can expose whether interface increases or decreases
cognitive load and whether it is understandable what
to do next
Norman, 1986; Hutchins et al, 1986
www.id-book.com
Bridging the gulfs

www.id-book.com
Bridging the gulfs

www.id-book.com
 Information processing
• Conceptualizes human performance in metaphorical
terms of information processing stages

A number of comparisons have been made, including conceptualizing the mind

as a pool, a telephone network, and a digital computer.
One established metaphor from cognitive psychology is the idea that the mind
is an information processor.
Information is thought to enter and exit the mind through a series of ordered
processing stages. Within these stages, various processes are assumed to act
upon mental representations.
Processes include comparing and matching. Mental representations are
assumed to comprise images, mental models, rules, and other forms of
knowledge.
www.id-book.com
 Limitations
• Based on modelling mental activities that
happen completely inside the head

• Do not adequately account for how people

interact with computers and other devices
in real world

www.id-book.com
 c. Distributed cognition
• Concerned with the nature of cognitive phenomena
across individuals, artifacts, and internal and
external representations (Hutchins, 1995)
• Describes these in terms of dissemination across
representational state

• Information is transformed through different media

(computers, displays, paper, heads)

www.id-book.com
A cognitive system for ATC

www.id-book.com
 What’s involved
• The distributed problem-solving that takes place
• The role of verbal and non-verbal behaviour
• The various coordinating mechanisms that are
used (e.g. rules, procedures)
• The communication that takes place as the
collaborative activity progresses
• How knowledge is shared and accessed

www.id-book.com
 e. External cognition
• Concerned with explaining how we interact with
external representations (e.g. maps, notes, diagrams)

• What are the cognitive benefits and what processes

involved

• How they extend cognition

• What technologies can we develop to help people

carry out complex tasks (e.g. learning, problem
solving, decision-making)?

www.id-book.com
Externalizing to reduce memory load
• Examples include the use of diaries, reminders,
calendars, notes, shopping lists, to-do lists
– written to remind us of what to do
• Post-its (sticky notes), shopping lists, marked
emails are used to:
– where placed indicates priority of what to do
• External representations:
– Remind us that we need to do something (e.g. to buy
something for mother’s day) (mark on calendar)
– Remind us of what to do (e.g. buy a card)
– Remind us when to do something (e.g. send a card by
a certain date)
www.id-book.com
 Computational offloading
• When a tool is used in combination with an
external representation to carry out a
computation (e.g. pen and paper)
• Try doing the two sums below (a) in your
head, (b) on a piece of paper and c) with a
calculator.
234 x 456 =??
CCXXXIIII x CCCCXXXXXVI = ???
• Which is easiest and why? Both are identical
sums

www.id-book.com
 Annotation and cognitive tracing
• Annotation involves modifying existing
representations through making marks
– e.g. crossing off, ticking, underlining

• Cognitive tracing involves externally manipulating

items into different orders or structures
– e.g. playing Scrabble, playing cards

www.id-book.com
 Design implication
• Provide external representations at the interface
that can reduce memory load and facilitate
computational offloading
– e.g. Information visualizations have been designed to
allow people to make sense and rapid decisions
about masses of data

www.id-book.com
 Embodied Interaction
• The practical engagement with the social and
physical environment (Dourish, 2001)
• Creating, manipulating and making meaning through
our interaction with things
• How our bodies and active experiences shape how
we perceive, feel, and think (Hornecker et al., 2017)
• They enable us to develop a sense of the world at
both a concrete and abstract level
• Can provide new ideas about interaction and better
design principles
– e.g. we think with our bodies not through them (Kirsh,
2013)
www.id-book.com
 Summary
• Cognition involves many processes including
attention, memory, perception and learning
• The way an interface is designed can greatly
affect how well users can perceive, attend, learn
and remember how to do their tasks
• Theoretical frameworks, such as mental models
and external cognition, provide ways of
understanding how and why people interact with
products
• This can lead to thinking about how to design
better products

www.id-book.com
 In-depth activity
Write down how you think a contactless card or
smartphone app like Apple Pay works

• What information is sent between the card/smartphone and

the card reader when it is placed in front of it?
• What is the maximum amount you can pay for something
using a contactless card, Apple Pay or Google Pay?
• Why is there an upper limit?
• How many times can you use a contactless card or
Apple/Google Pay in a day?
• What happens if you have two contactless cards in the same
wallet/purse?
• What happens when your contactless card is stolen and you
report it to the bank? What does the bank do?

www.id-book.com