Cognitive Walkthrough

3-1
 Definition

• Cognitive walkthrough involve simulating a user’s problem-solving

process at each step in the human-computer dialog, checking to see if
the user’s goals and memory for actions can be assumed to lead to the
next correct action. –Nielsen and Mack, 1994
“Cognitive walkthrough helps you understand the user’s perspective
of your product“
 Cognitive Walkthrough: Another
Inspection Technique
• Cognitive walkthrough = expert inspection focused on learnability
• Inputs:
• prototype
• task
• sequence of actions to do the task in the prototype
• user analysis
• For each action, evaluator asks:
• will user know what sub goal they want to achieve?
• will user find the action in the interface?
• will user recognize that it accomplishes the sub goal?
• will user understand the feedback of the action?
Conti..
• Cognitive walkthrough is another kind of usability inspection technique. Unlike heuristic
evaluation, which is general, a cognitive walkthrough is particularly focused on evaluating
learnability - determining whether an interface supports learning how to do a task by exploration.
• In addition to the inputs given to a heuristic evaluation (a prototype, typical tasks, and user profile),
a cognitive walkthrough also needs an explicit sequence of actions that would perform each task.
• This establishes the path that the walkthrough process follows. The overall goal of the process is to
determine whether this is an easy path for users to discover on their own.
• Where heuristic evaluation is focusing on individual elements in the interface, a cognitive
walkthrough focuses on individual actions in the sequence, asking a number of questions about the
learnability of each action.
Conti..
• Will user try to achieve the right sub goal? For example, suppose the interface is an e-commerce web site, and the overall

goal of the task is to create a wish list. The first action is actually to sign up for an account with the site. Will users realize

that? (They might if they’re familiar with the way wish lists work on other site; or if the site displays a message telling them

to do so; or if they try to invoke the Create Wish List action and the system directs them to register first.)
• Will the user find the action in the interface? This question deals with visibility, navigation, and labeling of actions.

• Will the user recognize that the action accomplishes their sub goal? This question addresses whether action labels and

descriptions match the user’s mental model and vocabulary.

• If the correct action was done, will the user understand its feedback? This question concerns visibility of system state -

how does the user recognize that the desired sub goal was actually achieved.
• Cognitive walkthrough is a more specialized inspection technique than heuristic evaluation, but if learnability is very

important in your application, then a cognitive walkthrough can produce very detailed, useful feedback, very

cheaply.
 Why we use Cognitive
walkthrough
• Cognitive walkthrough enables a designer to evaluate an interface without users
• a designer attempts to see the interface from the perspective of a user
• Low-investment technique to identify task-related usability issues early on
• no implementation or users required
• can be performed on existing interfaces
• Identify task-related problems before implementation
• invest a little now, save a lot later
• Enables rapid iteration early in design
• can do several evaluations of trouble points
• Evaluations are only effective if your team
• has the right skill set
• wants to improve the design, not defend it
 Walkthrough Basics
• Imagine how well a user could perform tasks with your low-fidelity prototype
• Manipulate prototype as you go
• evaluate choice-points in the interface
• evaluate labels or options
• evaluate likely user navigation errors
• Revise prototype and perform again
 When to do the Walkthrough
• Have a low-fidelity prototype of the interface
• Know who the users are
• Have task descriptions
• Have scenarios designed to complete the task
• you have a “functional” paper prototype
• Viable once the scenario and paper prototype are complete
What You Need
• Task descriptions
• Low-fidelity prototype with enough “functionality” for several tasks
• Evaluation team:
• design team
• design team and users together
• design team and other skilled designers
 For Each Action in a Task
• Tell a story of why a user would perform it
• Critique the story by asking critical questions
• is the control for the desired action visible?
• will a user see that the control produces the desired effect?
• will a user select a different control instead?
• will the action have the effect that the user intends?
• will a user understand the feedback and proceed correctly?
 Pros and Cons of Walkthrough
Walkthrough Pros
• Easy to learn
• Can perform early in the design process
• Questions assumptions about what a user may be thinking
• Helps identify controls obvious to the designer but not a user
• Helps identify difficulties with labels and prompts
• Helps identify inadequate feedback
Walkthrough Cons
• Is diagnostic, not prescriptive
• Focuses mostly on novice users
• Designers must put themselves in users mind
• Focus specifically on task-related issues
• The interactions are slower and not real
• Does not provide quantitative results
• A useful tool in conjunction with others
 What HE and CW have in
•
Common
Despite their differences, the two techniques, like many inspection methods, have several things in common.
• Double Experts: Having expertise in both Human Computer Action and the specific domain (e.g., Finance, Automotive Parts, etc.) will yield the best

insights.
• Uncover Many Usability Problems: In general both methods tend to find many of the problems in user-testing. The actual percentage of problems

tends to vary from (30 to 90%) depending on the study (Hollingsed and Novick 2007). Interestingly enough, this percentage is similar to software

inspection and walkthrough methods which tend to find between 30% and 70% of the logic-design and coding errors that are eventually detected (Myers

2004 p21).
• Not User Testing: Neither method is a substitute for testing with actual users. Both offer a potentially cheaper way of identifying problems at all stages

of the development process. It can be difficult to test users on a prototype and these inspection methods provide for early feedback, especially in an

iterative design methodology.

• Users’ Point of View: Both methods require the usability experts to take the users’ point of view as they inspect the interface.
• Multiple Evaluators are Best: Each evaluator is only uncovering some of the usability problems (often around 30% for more obvious issues) so having

multiple evaluators inspecting an interface will generate both more problems are identify overlapping problems.
• Inspections Can be more Thorough Than User Testing: While usability testing is often looked at as the “gold-standard” for detecting problems, users

are generally constrained to a small number of tasks. This limits their expose in the interface and the opportunity to detect more problems. With HE/CW,

a few evaluators can inspect all the “nooks and crannies” of an interface and provide more coverage of an interface.
 How to Conduct a Cognitive
Walkthrough
• A cognitive walkthrough begins by defining the task or tasks that the user would be
expected to carry out. It is these tasks that the cognitive walkthrough will examine for
usability—any tasks that can be performed in the product but are not subject to a
cognitive walkthrough will not normally be assessed during the process.
The Four Questions to be Asked during a Cognitive Walkthrough
• In their 2002 paper, “Cognitive walkthrough for the Web” Blackmon, Polson, et al. offer
four questions to be used by an assessor during a cognitive walkthrough:
• Will the user try and achieve the right outcome?
• Will the user notice that the correct action is available to them?
• Will the user associate the correct action with the outcome they expect to achieve?
• If the correct action is performed, will the user see that progress is being made towards
their intended outcome?
 Who Should Conduct a Cognitive Walkthrough?

• Anyone can conduct a cognitive walkthrough; however, there is a risk that

someone who is already familiar with your jargon, language and system will miss
things that someone who lacks that familiarity would find.
• Given that conducting a cognitive walkthrough is not particularly challenging.
• If you’ve prepared the task lists and can copy the questions (plus a little
explanation as to how to use them) onto a piece of paper – you should be able to
find someone in your organization who can do it. You can also ask users to
conduct these walkthroughs—that will add recruitment time and expense (you’ll
need to compensate the user for their time).
• If you have to use someone who is very familiar with the product, make sure they
have user personas at hand – to try and guide them to “walk a mile in the user’s
shoes”.
What Do You Do with the Answers to the Questions?

• Each assessor (a person who evaluates the quality of a person or

thing) involved in a cognitive walkthrough should record the step in
the process where they found an issue and what that issue was. This
can be done in their own words.
• When the process is complete, it’s a good idea to round up all the
assessors’ reports into a single report and then prioritize issues for
fixing.
 Differentiate between HE and CW

Heuristic Evaluation Cognitive Walkthrough

Perspective Analyst New User
Target General usability Learnability
Scope Comprehensive Targeted activities
Evaluation of interface against Exploring potential user reactions and
Method
guidelines behaviors to the system
 Differentiate between HA and CW and UT
Heuristic analysis Cognitive walkthrough User testing
WHO System expert New user End-user
WHAT Compares usability to Performs specific user tasks Uses the digital product
predefined heuristics in line with user goals in realistic
circumstances
WHY To see if the digital product To determine if the To understand how
can be used in a way that is sequential processes to get representative users
most compatible for users from point A (user task) to will complete typical
and aligns with recognized point B (user goal) work in tasks in real-life
usability principles. the correct order they were situations
designed to
 Walkthrough Example
• I have a library book that needs to be returned today. To help me remember, I want to set a
reminder on my PDA. The reminder should display and beep at 3:00pm to remind me to return the
book.
• Let’s walkthrough this task on my PDA and identify usability issues, if any
• Will a user try to produce the effect that the action has?
• Will a user see the control for the action?
• Will a user see that the control produces the desired effect?
• Will a user select a different control instead?
• Will a user understand the feedback to proceed correctly?