Refining

Design
for Business

USING ANALYTICS, MARKETING, AND TECHNOLOGY

TO INFORM CUSTOMER-CENTRIC DESIGN

MICHAEL KRYPEL
Refining
Design
for Business

Using analytics, marketing, and technology

to inform customer-centric design

Michael Krypel
Refining Design for Business:
Using analytics, marketing, and technology to inform customer-centric design
Michael Krypel
This Adobe Press book is published by Peachpit.
For information on Adobe Press books and other products, contact:
Peachpit
www.peachpit.com
For the latest on Adobe Press books, go to www.adobepress.com
To report errors, please send a note to [email protected]
Peachpit is a division of Pearson Education.
Copyright © 2014 by Michael Krypel
Project Editor: Valerie Witte
Production Editor: Becky Winter
Developmental and Copyeditor: Anne Marie Walker
Proofreader: Liz Welch
Composition: Danielle Foster
Indexer: James Minkin
Cover and Interior Design: Mimi Heft
Cover and Interior Illustrations: Paul Mavrides

Notice of Rights
All rights reserved. No part of this book may be reproduced or transmitted in any form by any means,
electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of
the publisher. For information on getting permission for reprints and excerpts, contact permissions@
peachpit.com.

Notice of Liability
The information in this book is distributed on an “As Is” basis, without warranty. While every precaution
has been taken in the preparation of the book, neither the author nor Peachpit shall have any liability
to any person or entity with respect to any loss or damage caused or alleged to be caused directly
or indirectly by the instructions contained in this book or by the computer software and hardware
products described in it.
Any views or opinions presented in the interviews in this book are solely those of the author and
interviewee and do not necessarily represent those of the companies included in this book.

Trademarks
Many of the designations used by manufacturers and sellers to distinguish their products are claimed
as trademarks. Where those designations appear in this book, and Peachpit was aware of a trademark
claim, the designations appear as requested by the owner of the trademark. All other product names
and services identified throughout this book are used in editorial fashion only and for the benefit of
such companies with no intention of infringement of the trademark. No such use, or the use of any
trade name, is intended to convey endorsement or other affiliation with this book.
ISBN-13: 978-0-321-94088-9
ISBN-10: 0-321-94088-1
987654321
Printed and bound in the United States of America
 To Darren Johnson, Jennifer Sun, Kripa Nerlikar,
Ramona Meyer-Piagentini, and Stephen Ratpojanakul:
Thank you for your guidance and enthusiasm over the last eight years,
and for your tremendous contributions to building the optimization field.
Acknowledgments
I will be forever grateful to all the wonderful people from Adobe, Omniture, and
Offermatica for allowing me to be part of their exceptional teams, including Aaron
Graham, Adam Fayne, Adam Justis, Adam Wood, Alan Gurock, Amy Lam, Andre
Prevot, Ann Chen, Aseem Chandra, Barbara Dawson, Bianca Slade, Bill Ozinga, Bill
Peabody, Brad Kay, Brent Dykes, Brian Hawkins, Brian Ivanovick, Brittany Chandler,
Cameron Barnes, Christi Terjesen, Christine Yarrow, Christy Armstrong, Colin Lewis,
Colleen Nagle, Daniel Hopkins, Daniel Wright, Darren Johnson, Darrin Poole, David
Baker, David Hoye, David Humphrey, Debra Adams, Derek Bryce, Don Abshire, Doni
Lillis, Doug Mumford, Drew Burns, Drew Phillips, Eddie Ramirez, Ehren Hozumi,
Garrett Ilg, Gene Holcombe, Georgia Frailey, Heather Razukas, Hiro Awanohara, Jacob
Favre, James Roche, Jameson O’Guinn, Jamie Stone, Jason Haddock, Jason Hickey,
Jason Holmes, Jeff Fuhriman, Jennifer Sun, Jim Sink, John Kucera, John Mosbaugh,
Jonathan Mendez, Jonathan Weissbard, Justin Patrick, Ka Swan Teo, Kaela Cusack,
Katie Cozby, Kellie Snyder, Kendra Jenkins, Kevin Lindsay, Kevin Scally, Kevin Smith,
Kripa Nerlikar, Kyle Ellis, Kyle Johnson, Lacey Bell, Lambert Walsh, Lance Jones, Lily
Chiu, Liz Quinn, Mandeep Sidhu, Mark Boothe, Matthew Lowden, Matthew Roche,
Matthew Smedley, Matthew Thurber, Michael Curry, Michael Evensen, Mikel Chertudi,
Neha Gupta, Norman Dabney, Paige Burton, Peter Callahan, Rachel Elkington, Rameen
Taheri, Ramona Piagentini, Rand Blair, Reuben Poon, Richard Oto, Rob Cantave, Ron
Breger, Rotem Ben-Israel, Russell Lewis, Sachie Reichbach, Sarah Ferrick, Serge St. Felix,
Shoaib Alam, Stephen Frieder, Stephen Ratpojanakul, Steve O’Neil, Thejas Varier, Tiffany
Olejnik, Tom Ratcliff, Tony DiLoreto, Tracy Harvey, Vincent Cortese, Vladimir Sanchez
Olivares, Wallace Rutherford, Whitney Littlewood, and Zoltan Liu.
I am extremely thankful to everyone who agreed to be interviewed or who helped
connect me with people to interview, including Justin Ramers from Active Network;
Andrew Switzer from Ally Bank; Tom Lau from American Express®; Kevin Gallagher
and Thomas Gage from AutoTrader; Adam Crutchfield from Axcess Financial; Brandon
Proctor and Justin Bergson from Build.com; Chris Kahle from Caesars; Joanne Pugh and
Stephanie Paulson from Central Restaurant Products; Kyle Power from CHG Healthcare;
Christine Cox, John Williamson, Jonathan Stein, and Ruth Zinder from Comcast; Ed Wu,
Emily Campbell, Étienne Cox, Isabelle Mouli-Castillo, Joel Wright, Lester Saucier, Nazli
Yuzak, and Will Close from Dell; Sandy Martin from Dollar Thrifty; Zimran Ahmed
from Electronic Arts; Nate Bolt and Slater Tow from Facebook; Thomas Jankowski from
FlightNetwork; Karina van Schaardenburg and Simon Favreau-Lessard from Foursquare;
Jerome Doran, Jon Wiley, and Krisztina Radosavljevic-Szilagyi from Google; Linda Tai
from Hightail; Phil Corbett from IBM; Ajit Sivadasan, Ashish Braganza, Lewis Broadnax,
and Siping Roussin from Lenovo; Amy Parnell and Lea Ann Hutter from LinkedIn; Pete
Maher from the Luma Institute; Kenyon Rogers from Marriott; Peter Davio and Steven
Webster from Microsoft; Kyle Rush from Obama for America; Blake Brossman, Natalie
Bonacasa, and Ujjwal Dhoot from PetCareRx; Amit Gupta from Photojojo; John Pace
from RealNetworks; Chris Krohn, Phil Volini, and Sarah Nelson from Restaurant.com;
Matt Curtis and Roger Scholl from Saks Fifth Avenue; Ryan Pizzuto from T-Mobile;
Eileen Krill and Mary Bannon from The Washington Post; and Matthew Pereira and Rob
Blakeley from WebMD.
Thank you to Kelly Patterson for her fabulous editing, to Rosemary Knes for her careful
proofing, and to Jeff Patterson for introducing me to them. Thank you also to everyone
at the Adobe Press and Pearson Education for their hard work, including Anne Marie
Walker, Becky Winter, Damon Hampson, Danielle Foster, Jim LeValley, Liz Welch, Mimi
Heft, Ted Waitt, Valerie Witte, Victor Gavenda, and Vidya Subramanian Ravi.
Thank you to my mom, Merrill Janover, whose curiosity, creativity, and love of learning
are an inspiration, and to my brothers, David Krypel and Brian Krypel, for their
encouragement and comfort. Thank you to Alan Schorn, Bob Klein, Chayym Zeldis,
David Roth, Elizabeth Metz, Jarek Koniusz, Jeffrey Gilden, Jerry Denzer, John Cave, John
Zannos, Joseph Rutkowski, Nina Zeldis, Philip Sorgen, Robert Abrams, and all my other
wonderful teachers for their guidance, confidence, and inspiration. Thank you also to
Daniel Schweitzer, Bonnie Walters, Greg Lowder, and Scott Epstein for their feedback
on drafts of the book and for their amazing support.
Contents at a Glance
Part 1  Creating Engaging Customer Experiences 1

Chapter 1 The Art and Science of Business. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2 Using Data to Inform Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

part 2  The Iterative Optimization Methodology 39

Chapter 3 Methodology Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Chapter 4 Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

CHapter 5 Quantitative Research and Optimization Roadmap . . . . . . . . . . . . . . 87

Chapter 6 Optimization in Action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Chapter 7 Building an Optimization Organization . . . . . . . . . . . . . . . . . . . . . . . . . 129

PART 3  Visual Business Cases 141

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
The Age of Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
How This Book Is Structured . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Part 1  Creating Engaging Customer Experiences 1

Chapter 1 The Art and Science of Business. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

The Importance of Design to Business. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Customer-Centric Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Designed to Evolve:
The Iterative Optimization Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Visual Business Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
The Washington Post: Social Networking Links. . . . . . . . . . . . . . . . . . . . 11
2012 Presidential Election: Dinner with Barack Image Test . . . . . . . . 20
Building an Optimization Organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
The Skill Set of the Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Chapter 2 Using Data to Inform Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

The Traditional Design Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Design Decisions Based on Bad Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Design Decisions Based on Pixel Politics. . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Reexamining the Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Design Decisions Based on Good Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
The Cultural Benefits of Testing Designs. . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Online Design Is Not Fine Art. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Advice from Business Leaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
x    Refining Design for Business

part 2  The Iterative Optimization Methodology 39

Chapter 3 Methodology Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Step by Step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
The Paper Trail: Why It Matters and How to Do It . . . . . . . . . . . . . . . . . 43
The Optimization Roadmap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
The Optimization Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Wireframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Design Comps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Optimization Launch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Optimization Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Case Study: PetCareRx. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Optimization Roadmap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Optimization Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Wireframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Design Comps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Optimization Launch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Optimization Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Chapter 4 Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Benefits of Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Learning More About Customer Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . .66
Starting to Answer Qualitative Questions . . . . . . . . . . . . . . . . . . . . . . . . . 67
User Experience Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Online Customer Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Experiential Business Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Experiential Competitive Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Documenting Experiential Reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Don’t Act Too Quickly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Observational Customer Research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Other Research Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Ethnographic Studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Surveys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Customer Panels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Diary Studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Card-sorting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Feedback Forms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Acting on Qualitative Research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
 Contents    xi

Case Study: Comcast. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Take a Step Back. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

CHapter 5 Quantitative Research and Optimization Roadmap . . . . . . . . . . . . . . 87

Benefits of Quantitative Studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Starting with Quantitative Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Telling a Story Using Insights and Impact . . . . . . . . . . . . . . . . . . . . . . . . . 89
Online Success Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Analytics in Action: A Case Study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Action Plan: Roadmap Assembly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Prioritizing Where to Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Prioritizing Test Ideas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Chapter 6 Optimization in Action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Optimization Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Test Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Number of Recipes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Test Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Success Metrics and Segments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Wireframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Structure of a Wireframe. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Assembling Wireframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Creating Great Wireframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Design Comps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Optimization Launch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Quality Assurance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Optimization Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Chapter 7 Building an Optimization Organization . . . . . . . . . . . . . . . . . . . . . . . . . 129

Optimization Maturity Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Maturity Model Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Core Optimization Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
xii    Refining Design for Business

Hiring and Nurturing Talent. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Roles and Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Core Optimization Team Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Hiring for Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Up Next. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

PART 3  Visual Business Cases 141

Adobe. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Visual Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Ally Bank. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
American Express®. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Caesars Entertainment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Comcast. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Dell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Dollar Thrifty Automotive Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Electronic Arts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Facebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Foursquare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Google. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Hightail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
IBM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
LinkedIn. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Marriott International. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Obama for America. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
 Contents    xiii

PetCareRx. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
Saks Fifth Avenue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
T-Mobile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
The Washington Post. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Visual Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Introduction
The Age of Optimization
This book aims to help businesspeople apply a comprehensive and powerful new way of
building business and customer value online, using a proven methodology that works
for any size of business, in any industry. Highly interdisciplinary in nature, the Iterative
Optimization Methodology will be especially useful to:
CC Business executives. Aiming to meet online customer needs through a rigorous
research and evaluation process that also fosters creativity, the optimization method
is appealing to business leaders of all ranks, including C-Level executives.
CC Marketers, designers, and information architects. Great marketers and designers
were among the first professional groups to embrace the customer-centric design
techniques described in this book.
CC Data analysts. Data analysts are essential to the success of almost every step of the
Iterative Optimization Methodology. They provide users throughout the business
with access to crucial data and help them to act on analytics-based insights.
CC Engineers and solutions architects. Engineers and solutions architects are often
the unsung heroes of online business. They spend much of their time setting up sys-
tems to deploy designs without knowing whether what they’re working on is helping
the company and its customers. The optimization method allows them to find out
whether or not their efforts have been fruitful.
CC Students, including MBA candidates. Students can learn more about an exciting
new field—one in which there is high demand for people who excel at creating great
wireframes, running optimization programs, and other skills described in this book.
How This Book Is Structured
This book is divided into three parts:
Part 1: Creating Engaging Customer Experiences. This part discusses the level of
importance design now plays in the business world, challenges the standard design
process implemented by most companies, and introduces the Iterative Optimization
Methodology by showing how design testing can lead to more creative and
impactful designs.
Part 2: The Iterative Optimization Methodology. Using real-life examples, this part
describes how to drive business and customer value in step-by-step detail. It shows
how companies can integrate qualitative and quantitative customer research, prioritize
website sections and design ideas for testing, experiment with new designs under real
market conditions, and scale optimization techniques across their organization.
Part 3: Visual Business Cases. In this part, business leaders from 20 companies,
including Google, Facebook, Comcast, Marriott, and American Express®, share examples
of their favorite design tests and discuss practical approaches for using data to inform
customer-centric design.
This page intentionally left blank
This page intentionally left blank
 Chapter 4

Qualitative Research

Many businesspeople, even those responsible for the online

experience, don’t spend nearly enough time trying to see their
business through the eyes of the customer. Qualitative research offers
an invaluable way not only to get into the customer mind-set, but also
to come up with ideas for innovation. This chapter discusses several
types of qualitative research and shows how they can be used to
generate test ideas while providing insights that will help guide the
business in an evolving marketplace.
66    Refining Design for Business

Benefits of Qualitative Research

Qualitative research is the art of capturing the customer’s perspective on the online expe-
rience offered by a business and its competitors. Companies often neglect to do qualita-
tive research and instead take the easy route by relying solely on analytics software to
understand their customers’ online behavior. However, quantitative research can only
reveal how site users are acting; it can shed very little light on what they are thinking or
feeling. In short, it is no substitute for truly seeing a company through the eyes of its users.
This is not always a comfortable experience for business leaders, but it is key to helping
them understand the customer perspective, adapt their business to evolving conditions,
and generate ideas for new designs—tasks that metrics alone cannot fulfill.
In particular, qualitative research is critical to identifying:
CC Unmet customer needs. If a business is failing to provide customers with what
they’re looking for, no analytics report can shed light on that gap. By watching and
listening to customers, or by re-creating their online experience firsthand, research-
ers can discover users’ expectations, unmet needs, and any sources of frustration.
CC The why behind the analytics data. Sometimes metrics can generate more ques-
tions than answers. For example, an analytics report showing an increase in page
views per visitor isn’t always a good sign. Although it may indicate customers’ satis-
faction through deep engagement with the site, it may also indicate confusion about
how to add products to their cart, or reveal unsuccessful attempts to locate informa-
tion such as shipping rates or to find pages on the site that customers think should
exist but do not.
Most qualitative research methods work with small, even statistically insignificant, sam-
ple sizes. As a result, this kind of research is best used for understanding broad ques-
tions—such as whether customers understand the basic business offering and whether
it’s something they need—as opposed to informing decisions about how to design every
detail of a business. For the same reason, researchers often verify hypotheses based on
qualitative data by referencing web analytics or testing to see if the hypotheses apply to
a large number of customers.

Learning More About Customer Goals

The first step in qualitative research is to define realistic customer goals that explain why
customers are choosing to interact with a business. This is not as easy as it sounds: Cus-
tomers may access a site for very different reasons, and their goals can and do change
over time. As Rob Blakeley, Senior Product Manager for WebMD, put it, “Our customers’
goals are as different as there are blades of grass in New York City’s Central Park.”
However, most businesses can easily identify a preliminary list of the most likely cus-
tomer goals and adapt this list as researchers gather further information. For example,
 Chapter 4: Qualitative Research    67

a financial services company’s customer goals may include opening or closing a deposit
account, sending and depositing checks from a mobile device, getting a replacement
credit card while traveling abroad, and so on.
If a businessperson is struggling with this task, it may help to ask the question: If cus-
tomers could come to the business looking for help with only one thing, what would it be?
It’s much better to create a top-notch experience that can satisfy one important goal
shared by many customers than to create several mediocre experiences in an attempt
to address many different objectives.

Starting to Answer Qualitative Questions

The next step in qualitative research is to build a list of questions to explore. Like the
customer goals, this list is preliminary and will change over time. The following is a list
of potential questions that companies may use as a starting point in this process; of
course, each business will tailor the list to its own priorities.
Fundamental questions:
CC Why do customers come to the business?
CC Why do they leave?
CC Do customers understand what the business has to offer?
CC Do customers want what the business has to offer?
CC Is there anything customers want from the business that it is not providing?
CC When analytics data shows areas of concern—for example, high drop-off rates,
repeat page views, and so on—what are the reasons for the customers’ actions?
CC Which product or service is most important to customers?
CC If the business could focus on only one product or service, what should it be?
On finding products, services, and content:
CC Is the full breadth and depth of the product/services/content offering apparent?
CC Are there too many or too few products/services/content to choose from?
CC Do customers easily find products/services/content using site navigation
and/or search?
On the customer’s decision-making process:
CC Are products/services/content easy to differentiate from one another, or are they
too similar, leading to customer confusion over which to choose?
CC Are the cross-sells, up-sells, and promotions clear? Do they interfere with the
primary purchase?
CC Is there enough information to make a purchase decision?
68    Refining Design for Business

CC Is it clear how to find related or popular content? Does it interfere with the primary
content experience?
On conversion:
CC Are there barriers to easily checking out?
CC Are prices, taxes, shipping, returns, and security protections apparent? Are any of
them barriers to checking out?
CC Are there barriers to easily viewing additional content?
On the competition:
CC Which competitors do customers go to, and how do the preceding questions apply
to these competitors?
CC Do competitors offer a product or service that would complement the business’s
offerings, such as accessories for a product the business sells? If so, should the busi-
ness offer it as well?
Note that some of the preceding questions are based on quantitative research; however, it
is not essential to have performed quantitative studies before tackling qualitative research.
Most businesses already have a basic understanding of their data, and the iterative
method means new insights, from both kinds of studies, will be added to the mix through-
out the process, allowing team members to refine and add to their list of questions.
The next step in qualitative research is to try to answer these questions through first-
hand experience, observing customers, and other methods, as described in the follow-
ing sections.

User Experience Reviews

In user experience reviews, or heuristic reviews, researchers strive to experience the busi-
ness firsthand in the same way customers do; then they do the same on competitors’
websites for comparison. Before we examine user experience reviews in detail, it is help-
ful to explore some guidelines on customer behavior that can help the researcher, or
reviewer, make their customer experience as authentic as possible.

Online Customer Behavior

Customers generally display the following tendencies when interacting with an online
business:
CC Customers are goal-oriented. They’re looking for a place to click on the page or
app that will take them to the specific product, service, or content that constitutes
their objective.
 Chapter 4: Qualitative Research    69

CC Customers scan pages quickly. They skip over details until they’ve found what
they’re looking for. For example, they may click through a business’s homepage and
several category and product pages rapidly, navigating with the help of images or
keywords until they find a product they’re interested in; only then are they likely to
read details such as the price, feature list, reviews, and shipping policy. They are very
good at ignoring anything that looks like advertising, a tendency that is sometimes
called banner blindness.
CC Customers don’t act like professionals. They don’t think about whether the navi-
gation is consistent, which page they’re on within the site, how individual page ele-
ments are working, what the business is trying to get them to do next, and so forth.

Experiential Business Review

Researchers first choose a realistic customer goal from the prepared list, and then follow
a “customer scenario”—the actions a customer would take to pursue that objective. It is
important here to step out of the professional mind-set and just experience the website
as a customer would: If the reviewer can relate personally to the goal, all the better. For
example, for a retail business the reviewers could go through the “browse and purchase”
process for an item they’re interested in buying or would like to get as a birthday gift for
a friend; for a travel business, the reviewers could look into booking their next trip.
Taking notes along the way, reviewers should strive to do anything customers would
do—call customer support, abandon the site when unable to attain their goal easily, and
so on. If many customers visit the site from mobile devices, reviewers should try going
through the scenarios using a phone or tablet.
After experiencing one scenario, the reviewers retrace each step of the experience to
critique it as a professional, noting any opportunities for improvement. Was it easy
to accomplish the goal? Were some elements especially difficult to navigate? Were
there steps that confused the reviewers? The reviewers then repeat this process for the
remaining customer goals.

Experiential Competitive Review

Because customers are also visiting competitors’ websites, reviewers should do so, too.
They follow the same procedure as before, exploring the same scenarios as customers,
and then return to each step to critique the experience as professionals. To make the
experience especially organic, reviewers may even start on their own business’s site and
then abandon it for a competing site based on their natural browsing behavior or on
analytics data that shows where customers tend to abandon.
70    Refining Design for Business

Documenting Experiential Reviews

It’s important to document the heuristic reviews to share the findings and to prepare
for the observation of real customer interactions. The documentation process should
consist of screen shots showing the step-by-step interaction path, notes explaining
the screen shots, and an updated list of qualitative questions based on the review.
For example, researchers may ask whether customers also struggle or get confused
where the reviewers did or whether users find it easy to navigate the same paths as
the reviewers.
The following example illustrates how this process typically unfolds through a user
experience review of the Gap, Inc. conducted in 2013. Figure 4.1 encapsulates the main
steps of the experiential review.
The accompanying notes for the steps in the figure include:
1. Reviewer visits Gap.com to look for a summer dress.
2. Reviewer sees red dress in middle right that looks cute and clicks on it. Reviewer is
not taken to the dress but to a page listing many dresses that don’t look like the one
she clicked.
3. Reviewer sees a similar dress in a different color on the second row and clicks on it.
Reviewer decides to purchase it. She tries to change the color to red by clicking on
the large image of the dress in the middle of the screen, but doing so simply shows
a zoomed-in view of the dress.
4. Just before abandoning, reviewer clicks on the small image of the dress in the middle
of the screen.
5. Reviewer now sees the color palette and realizes that the dress is no longer available
in red. She decides to buy it in coral as pictured. She clicks on “add to bag” but noth-
ing happens, so she leaves to shop at a competitor’s site. (The “add to bag” link failed
because she first needed to select a size; there was no error message explaining this.)
Upon further critiquing the experience, the following questions were generated for the
observational stage of the research:
CC Do customers also struggle to find the product page for an item they liked on the
homepage?
CC Even though it was jarring to be taken to a page with hundreds of dresses after click-
ing on a single one, would it be a good customer experience to arrive on a page like
that after clicking on the “Dresses” category in the Women’s section of the site?
CC Do customers also struggle to figure out what colors an item comes in and how to
choose a different color?
CC Do customers also find it difficult to add an item to their cart?
 Chapter 4: Qualitative Research    71

1
3

Figure 4.1  It’s easy for customers to get confused or frustrated

online. In this example of a user experience review, the customer
would have abandoned the site after struggling to purchase the
red dress she liked on the homepage.
72    Refining Design for Business

CC Do customers also try to buy before selecting a size and get confused when no
message reminds them to pick a size or offers to help them do so?
Researchers seek to validate these questions by observing customers interacting with
the site and by digging into the site analytics. Team members then create hypotheses,
or proposed answers to these questions, and attempt to vet the hypotheses through test
ideas, which are added to the Optimization Roadmap.

Don’t Act Too Quickly

The purpose of the user experience review is to gain insights into the customers’ experi-
ences by stepping into their shoes for a moment. However, team members should resist
making immediate changes to the site based solely on heuristic reviews unless clear
technical errors are uncovered. Otherwise, the changes may reflect the perspective of
online professionals, not the typical customer.
Conducting realistic user experience reviews is a skill that can be learned over time,
but improvement requires an adequate feedback loop: Reviewers need to watch actual
customers interact with the business and test new designs, not simply criticize the
company’s site in a vacuum.
Researchers must also exercise caution when exploring competitors’ sites. All too often
businesses assume a competitor’s designs are effective and simply copy them without
finding out whether these designs would work for their own business. As a result, not
only do they risk adopting ineffective designs, but their sites also wind up looking very
similar to those of their competitors.
One way to avoid this pitfall is to review sites outside of the business’s circle of competi-
tors when looking for new ideas; no matter where they find new concepts, companies
must test them before pushing them live to all customers.

Observational Customer Research

Directly experiencing a site can be a great way to generate questions that lead to test
ideas, but it’s no substitute for spending time with real customers. As noted in Chapter
1, users aren’t very good at articulating exactly what they need, so rather than asking
customers for their opinions, observational research instead pays close attention to
their behavior as a way of understanding their perspective. Team members strive to
watch customers interacting naturally with the site; they provide them with opportuni-
ties to give open-ended feedback and avoid guiding them too much or prompting them
with leading questions.
 Chapter 4: Qualitative Research    73

Customers are recruited in a variety of ways; although some volunteer, they’re more
often compensated through some form of payment, such as a check or gift card. Even
though customers know they’re being observed, the goal of this type of study is to create
as natural a setting as possible, so it may take place not only in research facilities, but
also in cafés, customer homes, and other venues. Sessions are conducted one-on-one,
with additional researchers observing behind one-way glass or through a video feed.
Depending on the business, sessions can last anywhere from 10 to 60 minutes. High-
level trends usually start to emerge after watching four or five customers, but to be on
the safe side, researchers often observe eight to ten customers during each study. The
frequency of observational studies varies according to the company, but most busi-
nesses should conduct them at least quarterly. Many sizable and successful companies
conduct this type of research whenever they have pressing unanswered questions about
their business, which could arise several times per month.
The following list outlines the basic framework for this type of research, as well as some
best practices for observing customers online:
Recruitment
CC The first few times a business conducts this type of research, it may be helpful to go
through a recruitment agency and use an expert moderator to conduct the sessions.
Later, the business can recruit by placing an ad online on a site like Craigslist or
work with existing customers contacted through an internal email list.
CC It is useful to recruit customers based on an experience they recently had or an
upcoming experience that the business can help with. For example, a hospitality
business might recruit customers who have recently booked a hotel room online or
who need to book a room for an upcoming trip.
CC If possible, the customers should not know the name of the business, because this
information might influence their behavior. For example, recruiters can tell them the
study is about online user behavior related to the general category of tasks the busi-
ness is interested in observing (purchasing clothing, reading the news, searching for
general information, and so on).
Setting scenarios
CC When customers arrive, the researcher welcomes them and asks questions that
prompt them to talk about a wide range of recent online experiences, such as “What
have you recently shopped for online?” or “Have you planned any trips recently?”
CC The researcher provides customers with a computer or mobile device that allows
them to browse the web while recording their online session and facial expressions.
(Of course, customers are informed in advance that their session will be recorded.)
74    Refining Design for Business

CC The researcher invites the customers to go through a scenario they had mentioned
earlier that pertains to the business (e.g., booking a hotel room for an upcom-
ing stay). Customers are asked to speak aloud while using the computer to let the
researcher know what they’re thinking.
Observations
CC The researcher then sits back, pays close attention, and tries not to say anything. The
key is to leave customers alone while carefully watching how they try to reach their
goal—and what obstacles they face along the way—and paying close attention to
shifts in body language, facial expressions, and so on.
CC The researcher refrains from asking customers what they think about specific web-
sites or designs. For example, the researcher does not ask whether certain sites are
easy or helpful. Instead, the researcher focuses on what customers show through
their actions; the more customers are prompted, the more likely they’ll be to say or
do something based on what they think the researcher wants to hear. If customers
get confused and ask what to do, the researcher simply says something like, “Please
do whatever you would normally do.”
On prompting the customer
CC Some customers have a tendency to remain silent throughout the session. If this
is the case, the researcher will occasionally prompt them with a kind reminder to
vocalize their thoughts.
CC If, toward the end of the session, customers haven’t used the online business, the
researcher may prompt them by saying something like, “I’m interested in seeing you
do the same thing with a few specific websites.” The researcher will give the custom-
ers a list of sites that includes the business being studied as well as competitors not
yet visited.
CC The researcher should wait until the end of the session to ask any specific questions
about the customer’s behavior, and even then, they should avoid asking for any opin-
ions. For example, a researcher can ask customers to explain what they were think-
ing at a specific moment rather than asking if they thought the experience was good
or not. When listening to their answers, the researcher should be on the lookout for
a mismatch between what customers say and their observed actions. People tend to
jump around online from website to website very quickly and usually have difficulty
remembering all the actions they took online, let alone why they took them.
CC At the end of the session, the researcher should thank the customers for their time
and pay them the agreed-upon fee.
After the session, the researcher reviews the video to capture any findings that provide
insights into the list of qualitative research questions. These insights, in turn, may lead
to the formulation of new questions for further exploration in the iterative process.
 Chapter 4: Qualitative Research    75

Other Research Methods

Although heuristic reviews and observational customer research are the most common
methods, there are many other types of qualitative customer research. This section out-
lines some that are popular among online businesses.
It should be noted that two methods that remain popular today—focus groups and
usability testing—do not figure in this section because they are not recommended.
Focus groups tend to place customers in the role of product designer by explicitly
asking them to give advice on design decisions, which is difficult for people to do
when such decisions are not their area of expertise. Additionally, focus group partici-
pants tend to try to say the “right thing” to please the researcher or they succumb
to groupthink.
Traditional usability testing asks customers to complete a predetermined list of online
tasks based on actions the business would like to see completed rather than allowing
customers to show the business their natural online behavior. Even though customers
may be good at figuring out how to perform a task when asked, doing so takes them
out of their typical frame of mind when interacting with websites.
Instead, the following are several tried-and-true methods for understanding customers
through qualitative research.

Ethnographic Studies
Ethnographic studies are a special form of observational research that take place where
the customer would normally interact with the company, such as at their home or
office, or at the location of the business itself, so that they are more likely to behave
the way they normally would. Just like observational customer research, these sessions
are most effective when they are unguided and researchers focus on active watching
and listening.
Kenyon Rogers, Director of Digital Experiments for Marriott International, said that
his business regularly conducts ethnographic studies at select hotel properties. For
example, in 2013 the company piloted a program that allows guests to “use their smart-
phones to check into the property and open their room door without needing to inter-
act with a Marriott team member.” Rogers added that very soon, “guests will be able to
control their entire experience, including ordering room service, extending their stay,
ordering transportation, and booking meeting rooms through their smartphones.”
76    Refining Design for Business

Look for the Post-its:
Pete Maher on Contextual Inquiry
Pete Maher is the Co-Founder and Chief Operating great intentions—trying to protect a user’s security—
Officer of the LUMA Institute, an educational com- enact requirements that ultimately make it too diffi-
pany that equips people to accelerate innovation. He cult for people to achieve their goals.
is also the co-author of Innovating for People: Hand- So, when conducting contextual inquiry research,
book of Human-Centered Design Methods, which has we’ll advise teams to look for the Post-its or duct
been adopted by the U.S. Office of Personnel Man- tape: What are the clever workarounds users come
agement’s Innovation Lab as core training material up with to make it possible for them to use the
for federal government employees. In this interview, product? It’s only when we ask people to show us
he shares his thoughts on the importance of primary how they actually use something that we’re able to
observational research. uncover really ripe opportunities for innovation. And,
Your book presents the “contextual inquiry” most important, close observation uncovers those
method, which is an ethnographic approach. Can gaps between what people do and what people say
you describe it here? they do.
I’ve got a deep affinity for contextual inquiry research Would you tell us more about the gaps between
because it’s a great way to get to insights that can ulti- what people do and what they say?
mately uncover opportunities for innovation. Contex- Margaret Mead, the famed anthropologist, really
tual inquiry takes place where the participants would said it best: “What people say, what people do, and
normally conduct their tasks. The interviewer asks what they say they do are entirely different things.”
the participants to go about their tasks in a normal Anybody who has spent a lot of time doing research
way, observes their actions in an unobtrusive man- knows this to be the case. It’s not because people are
ner, interjects questions at opportune moments, and necessarily trying to lie or mislead. It turns out that
records the sessions for later analysis. we, as people, are simply not wired to be able to really
Would you share an example? articulate why we did that thing we just did.
When we don’t deeply understand the context of For example, if you ask somebody to describe to
the user, we sometimes assume that we’re deliver- you in detail how they completed a purchase on a
ing against their needs. For example, it turns out website, you could imagine them describing the steps
digital products are incredibly difficult to use. From that they took and all the different things that they
my days of doing research in the financial services did along the way. But if you were to actually watch
industry, when we would go into people’s homes to that scenario play out, you would see something very
understand how they engage with financial products, different. So contextual inquiry research, especially in
we would commonly find things like Post-its stuck the digital context, allows us to observe how some-
to monitors. It’s no mystery that some of those Post- body’s moving through a digital experience to really
its contain usernames and passwords, or software know what they just did and to try to understand why.
instructions, because a lot of times companies with
 Chapter 4: Qualitative Research    77

Surveys
Businesses place surveys on their actual site or app, or email them to customers. Large
surveys can have statistically significant sample sizes, but the researcher must be on
the lookout for data not representative of the larger customer base due to self-selection
bias. For example, not every customer wants to fill out a survey, and those who do may
have the strongest positive or negative opinions.
As with all forms of qualitative research, the more open-ended the survey, the better.
Surveys that ask customers about specific design decisions place the customer in the
awkward position of being asked to provide advice outside of their area of expertise.
Rather, understanding whether customers found their overall experience to be positive
or negative and providing an open-form field for customers to write about any aspect of
the experience they choose can often lead to actionable data.

Customer Panels
Customer panels are a subset of surveys: They typically consist of thousands of partici-
pants who have elected to give survey feedback on a regular basis. Panels may be run
by a company’s research team or by consulting firms on behalf of many businesses.
Like surveys, customer panels can provide statistically significant sample sizes, but
it’s important to understand the segment of participants being queried. For example,
although panels consisting entirely of self-selected users of one business might not
be representative of the entire population, they can give the business insight into the
behavior and opinions of their more loyal customers.
Eileen Krill, research manager at The Washington Post, oversees customer research for all
of the business’s print and digital brands. A panel of about 7,000 customers is included
in the many types of qualitative and quantitative customer research she oversees. Krill
will ask the panel “a wide range of closed and open-ended questions, depending on the
objectives of the survey,” including “satisfaction rating questions.” She points out that
“open-ended feedback is generally far more meaningful and actionable than the score
itself ” because it can help to “reveal the reasons behind the scores.”
One question she often asks the panel is “whether a new product or feature will
improve the customer’s impression of The Washington Post brand.” Although in most
cases participants say such additions would have no impact, Krill still asks the question
in case it provides an important insight. For example, she said, “We once tested the idea
of starting an online dating service for Post readers, and that got a lot of people saying
they would have a lower opinion of the company.” Krill noted that “I think that research
was one of the key things that may have killed the idea.”
78    Refining Design for Business

Diary Studies
Diary studies consist of a business asking customers to take notes and regularly send
them back to the company, usually over an extended period of time. These studies may
ask participants to take notes only on a specific topic area, like their regular interac-
tions with a new site or app, or they may be more general and simply ask customers
how they spent their day.
Google Search Lead Designer Jon Wiley shared an example of an ongoing diary study being
run by the company. Through a mobile app, the study regularly asks participants to reply to
the question: “What is the last bit of information you needed to know?” The information can
be related to any aspect of their life, not only the material they were looking for online. Wiley
and his team then “look at the needs that people have in their lives” and try to answer the
question, “Is there a way that we, as Google, can find a solution for them?”

Card-sorting
A technique used to gain insight into how to organize content, such as ordering and
grouping similar navigational links, card-sorting directly involves customers in the
design process. Customers are provided with a stack of cards containing information
and asked to perform a task, such as organizing them into logical categories. Although
it is typically performed with index cards or sticky notes, card-sorting can also be per-
formed online, and there is no limit to the number of participants.
Caution should be observed, because this technique is more guided than the aforemen-
tioned qualitative methods and may place customers in the position of being asked to
act like a professional designer. However, if performed with minimal prompting, card-
sorting can provide designers with a rare opportunity to gain insight into how custom-
ers think about information architecture and content hierarchies.
Nate Bolt, design research manager at Facebook, said the company used cart-sorting as
one of many qualitative research methods to inform an ongoing redesign of users’ Face-
book News Feeds. After recruiting users, the Facebook Design Research team printed
out each user’s feed, up to the minute, on paper. Then, he explained, “they would cut out
their feed stories, place them on a table, and group them into what they considered to
be like-minded categories. That helped us reprioritize the ways that stories are grouped
and organized within people’s News Feeds in a real, human way. Obviously, this went
hand in hand with all other data, including the system (analytics) data.”

Feedback Forms
Feedback forms on websites, or email addresses for user feedback, can be a good source
of information. Users who provide open-ended feedback in this way often have timely
insights into customer frustrations, as well as positive comments on great experiences
with the website or the business’s team members.
 Chapter 4: Qualitative Research    79

John Williamson, the Senior Vice President and General Manager of Comcast.com,
explained that “One of the best sources for information I have is the ‘Website Feedback’
link” that appears on the bottom of each page on the site. Williamson noted, “I was in
banking in the mid-1990s, and I remember how excited we would get if we received a
letter from a customer. We really would.” Every day Comcast receives “hundreds of let-
ters” from customers through the feedback link, and because they are a “huge benefit”
that helps Williamson and his team understand their customers and their business, he
added, “I read every one of them.”

Acting on Qualitative Research

After the heuristic and other qualitative research sessions have been conducted,
researchers summarize the insights they have garnered, as well as any remaining ques-
tions about customer behavior. For each insight, they write down possible ways of veri-
fying the observations through analytics data, as well as any ideas for testing. This list
will be used as a guide when crafting the Optimization Roadmap.

Case Study: Comcast

A leading provider of television, high-speed Internet, and digital voice services in the
United States, Comcast uses qualitative and quantitative research to inform the itera-
tive design process it applies to all aspects of its site.
Qualitative studies play a critical role in Comcast’s design process. Williamson noted
that in one redesign process during 2012, “We had over a thousand customers and pro-
spective customers involved before we tested online….We had ethnographic research,
customer panels, and online customer panels.” The Comcast team also “went into the
homes of over 20 customers and saw how they interacted with Comcast,” which pro-
vided “critical” experiences that analytics “data cannot replace.”
The following business case shows how insights gathered from qualitative studies trig-
gered a redesign of the company’s “TV Options” page, which customers use when pur-
chasing television service online. This section walks the reader through the redesign
process from beginning to end, showing how the qualitative study findings led to the
formulation of hypotheses and redesign options, and finally to the first tests and the
selection of highly effective new designs.
The goal of the “TV Options” page is to help customers choose their receiver type and
to select service upgrades, such as premium channels. However, qualitative studies sug-
gested three page elements were underperforming: receiver selection, premium channel
logos, and other add-on options further down on the page. The default version of the
page, with these three elements highlighted, appears in Figure 4.2.
80    Refining Design for Business

Default
Figure 4.2  The default
page with three elements
highlighted for testing.
 Chapter 4: Qualitative Research    81

The qualitative study findings related to these three elements and the steps researchers
took in response are summarized here.
Finding. Not all customers interacted with the first step of the process to learn more
about upgrading to an HD or HD DVR receiver.
CC Hypothesis. Customers do not notice the option to upgrade a receiver.
CC Validate with analytics. Does the link to upgrade the receiver get very few clicks?
CC Test idea. Find out whether more customers would upgrade if more information
about the options to switch to an HD or HD DVR receiver appeared on the page
without requiring additional clicks.
Finding. All customers understood how to select and deselect premium channels, so this
function seems user friendly. However, the website highlighted specific shows carried on
these channels rather than the actual channels, and several customers commented that
they hadn’t heard of these shows. Example: “I never heard of Boss. I never saw Contagion.”
CC Hypothesis. Highlighting channel names rather than specific shows will clarify the
process of adding premium channels.
CC Validate with analytics. Do premium channel selections vary based on the
featured shows?
CC Test idea. To appeal to a wider audience, feature prominent channel names
(HBO, Showtime, etc.) rather than specific shows.
Finding. Several customers commented that they didn’t know what was included in
the add-ons. Example: “What exactly is included in the Sports Package?”
CC Hypothesis. Customers don’t realize they can click on the toggle switches for more
information. Because the switches are arranged in rows, followed by messaging for
each add-on, customers may be mistaking them for bullet points.
CC Validate with analytics. Do customers rarely click on these toggle switches?
CC Test Idea. Replace individual offers with a one-line message reading “More custom-
ization options” to help customers understand that the image to the left is a toggle
switch that, when clicked, will reveal the details of the offers.
As noted earlier, researchers targeted receiver selection, premium channel logos, and add-
on content for design testing. An alternate version of each element was built and placed
in a multivariable test (MVT). A multivariable, or multivariate, test is a special type of A/B
test in which several page elements are tested across multiple recipes. Rather than simply
testing two versions of the page—one with the original elements and one with the new
ones—multiple versions of the page are tested, each including a different combination of
the elements. In addition to determining which version of the page has the highest perfor-
mance, this type of test measures the success of each element.
82    Refining Design for Business

The default and alternate versions of each element are shown in Figure 4.3. This figure
also includes which version of each element won and which out of all three elements
most influenced the performance of the page:
CC Receiver section. The default version has a blue link to upgrade. The alternate ver-
sion adds a prominent message to upgrade to an HD or HD DVR receiver.
CC Premium channel logos. The default version features a specific show for each pre-
mium channel. The alternate version features channel logos instead of shows.
CC Other add-ons. The default version places content behind four toggle switches. The
alternate version places content behind a single toggle switch, which, when clicked,
expands to show all up-sell content at once.
The business goal was to increase the purchase conversion rate as well as the overall
revenue per visitor (RPV). As Figure 4.3 shows, the winning version of the receiver
selection element was not the new design but the original; however, the new versions
emerged as winners for both the premium channel element and the add-ons further
down on the page.
When Comcast integrated all three winning designs into the page and tested this
revamped page against the original one, it found the changes drove a 4.6 percent lift
in the purchase conversion rate and a 5.6 percent lift in RPV. The default and winning
versions of the page are pictured in Figure 4.4.
Because the impact of each element had been isolated, Comcast was also able to mea-
sure how much each one influenced the performance of the final version of the page:
The premium channel element was the most influential, accounting for over 60 percent
of the increase in RPV, indicating that, of the three elements tested, it was the original
version of this feature that had presented the largest obstacle to customers completing
their goals.
The main recommendation was to push the winning version of the page to all visitors.
Additional proposals included running follow-up design tests of each variable to further
simplify the user experience, starting with the premium channel element, because it
had emerged as the most important one overall.
 Chapter 4: Qualitative Research    83

Default Elements Alternate Elements

Winner

Element #1

Default: Minimal upgrade Alternate: More prominent HD or

messaging DVR upgrade messaging

Winner + Most Influential Element

Element #2

Default: Specific show for each channel Alternate: Channel name without shows listed

Winner
Element #3

Alternate:
All content hidden
Default: Three collapsed choices
behind one link;
when clicked,
all content expands
Figure 4.3  The default and alternate versions of each of the
three elements tested. This figure includes the winning version
of each element, as well as the most influential of the three
winning elements in terms of overall impact on the page.
84    Refining Design for Business

Default Winner

+5.6% lift in RPV

+4.6% lift in purchase conversion rate

Figure 4.4  The default version of the page next to the

redesigned page, which contains all the winning elements.
 Chapter 4: Qualitative Research    85

Take a Step Back

Although qualitative research is a key part of the ongoing design process, it can also
play an invaluable role in helping to shape a company’s overall direction. Grounded in
the customer perspective, qualitative research helps businesspeople to shape long-term
strategies by keeping their finger on the pulse of an evolving marketplace, a practice
that helps them develop better ways of fulfilling their customers’ needs in the present
and in the future.
As businesspeople focus on the day-to-day activities related to delivering the same
products and services, it’s easy to lose sight of the long-term customer needs that are
driving the demand for those same products—a phenomenon renowned marketing
savant Theodore Levitt called “marketing myopia.” Levitt argued that these companies
tend to be “product-oriented instead of customer-oriented,” and view their “marketing
effort” as a “necessary consequence of the product, not vice versa, as it should be.”
He cited the example of railroad companies: Once the titans of industry, they had
fallen on hard times by the 1960s, all because they thought of themselves as being
in the business of “railroads.” Had they instead seen themselves as being in the busi-
ness of “transportation,” they would have begun to produce cars, trucks, airplanes,
and even telephones—an expansion they were well positioned to tackle, given their
extensive resources.
Levitt may have been writing in the 1960s, but his observations remain true: All too
often, businesses focus on current successes instead of prioritizing their customers’
present—and future—needs. In recent years, however, some companies have grown to
recognize the importance of building for the future; a few examples of well-developed
marketing strategies, or long-term game plans for providing customer value, appear in
Table 4.1.
Each strategy conveys a straightforward understanding of why customers come to the
business for help that goes beyond the current core product and service offerings. For
example, Comcast’s strategy to focus on entertainment and technology doesn’t keep the
company tied to current revenue sources, such as cable lines or set-top boxes, because
these will likely be short-lived due to rapidly evolving technology; indeed, Comcast
has expanded its product offerings to allow customers to watch movies and television
shows from mobile devices even when offline. Similarly, IBM’s online strategy is focused
on work-related tasks for business customers; as such, IBM has expanded its services
into cloud computing, “big data,” and social technologies.
All these examples illustrate the importance of preparing a pathway for the future—
a journey that, in many cases, begins with asking solid qualitative-research questions
and paying close attention to the answers.
86    Refining Design for Business

Table 4.1  Online Marketing Strategies

Company Online Marketing Strategy

Ally Bank “We help our customers to achieve their savings goals.” (Andrew Switzer)

Comcast “Comcast helps our customers to enrich their lives through entertainment
and technology.” (John Williamson)

Foursquare “We help our customers to make the most out of where they are.” (Simon
Favreau-Lessard)

IBM “IBM helps enable our customers to do their jobs better.” (Phil Corbett)

LinkedIn “We help our customers to connect with new opportunities.” (Amy Parnell)

PetCareRx “PetCareRx helps to bring health and happiness to pet owners and their
pets.” (Blake Brossman)

Saks Fifth “Our strategy is to inspire customer confidence and style with every Saks
Avenue shopping experience.” (Roger Scholl and Matt Curtis)

WebMD “We help our customers to find health information.” (Rob Blakeley)

Up Next
This chapter provided an overview of qualitative research techniques, which offer
insights into customers’ goals and the challenges they face when trying to accomplish
those goals. These insights lead to test ideas that will be added to the Optimization
Roadmap, as well as inform the overall direction for a company.
The next chapter outlines quantitative research methods that companies use to verify
qualitative findings, as well as to identify which site areas need improvement the most
and what test ideas to prioritize as part of assembling the Optimization Roadmap.

References
1. “Marketing Myopia,” Harvard Business Review; Theodore Levitt (1960).
This page intentionally left blank
INDEX
A promoting social links at log-off portal,
148–149
A/B testing. See also testing
American Express, 150–152
Adobe’s use of, 143–145
business and customers’ goals for, 150
Ally Bank’s use of, 147–149
designing multiple product displays,
best practices for, 114 151–152
Caesars’ booking module, 154–156 offering customer incentives, 151
Comcast’s, 160–161, 162 optimization program for, 150
defined, 29 testing impact on business strategies,
Dell’s use of, 168–169, 170–171 150–151
email design for Dollar Thrifty, 173–176 analysis paralysis, 88
Foursquare’s, 191, 193, 194–195 AOV (average order value), 92, 166, 267
Hightail’s homepage test using, 211 assembling wireframes, 119–120
LinkedIn’s, 223–225 auto-focusing feature, 246
Marriott International, 227–230 auto-optimizing tests, 109, 116–117
multivariable tests in, 81 autocomplete feature, 194–195, 209
Obama for America, 231, 232, 233–234, average order value (AOV), 92, 166, 267
236–248, 249–250 Axelrod, David, 234
PetCareRx messaging and banner, 252–257
product storage limits, 211–212
redesigned customer loyalty pages, 157 B
segmentation and, 235 banners
specifying in methodology, 109 A/B testing of size, messaging, and images
for, 253–255
T-Mobile phone payment options, 265–267
ignored by customers, 69
Washington Post’s, 271–277
best practices
address fields, 216
customer service, 121–122
Adobe, 143–145
Foursquare’s, 193
business and customers’ goals for, 143
Hightail’s, 211, 212
use of A/B testing, 143–145
Saks Fifth Avenue’s, 258
advertising banners, 69
Bezos, Jeff, 11, 268, 271
Ahmed, Zimran, 177–184
Blakeley, Rob, 66, 86
algorithm for News Feed displays, 187–188
blue-sky approach, 120
Ally Bank, 146–149
Bolt, Nate, 34, 35, 78, 185–190
balancing creativity with research, 146
borders of wireframes, 119
business and customers’ goals for, 146
bounce rate, 92
homepage testing by, 147–148
280    Refining Design for Business

Bowman, Doug, 200 Capital Business, 268

brainstorming test ideas, 103–104, 105 card-sorting, 78, 188
brand styles, 234–235 case studies. See also specific businesses
Brin, Sergey, 204 by name
Brossman, Blake, 47, 62, 86 about, 141–142
business and customers’ goals. See specific Adobe, 143–145
case studies Ally Bank, 146–149
business plans, 104 American Express, 150–152
businesses. See also case studies Caesars Entertainment Corporation, 120,
becoming optimization organizations, 153–157
22–23 Comcast, 79–84
cultural benefits of optimization, 31 criteria applied to, 10
evangelizing optimization within, 271 Dell, 166–172
existing to help customers, 8 Dollar Thrifty Automotive Group, 172–176
fear of data-driven design, 25, 31–33, 35 Electronic Arts, 177–184
hiring optimization team members, Foursquare, 191–195
137–140 Hightail, 210–212
making decisions on bad data, 26–27 IBM, 213–219
online presence for, 3–4 Lenovo, 93–101, 102–103
pairing data with design, 34 Obama for America, 20–21
realigning to customer goals, 9 PetCareRx, 47–60, 251–257
role of design in, 4–5 Saks Fifth Avenue, 258–261
shaping market strategies, 85–86 Washington Post, The, 11–20
software as a service, 210–211 chat pop-ups, 171
starting and scaling optimization programs, Chertudi, Mike, 143–145
160, 173, 223 classified job listings, 273–276
traditional design processes in, 26 click-through rate (CTR), 19–20, 92, 192–193
understanding customer goals, 8–9 Clooney, George, 243–244
using surveys, 77 Club Pogo, 177, 178
buttons, 244, 249–250 color
Donate page button, 249–250
C Google’s research on, 35, 200–201
Caesars Entertainment Corporation, 120, pairing text and background, 33
153–157 testing form field background, 246
business and customers’ goals for, 153 Comcast, 158–165
optimization programs for, 154 business and customers’ goals for, 158
redesign of customer loyalty pages, 157 global navigation testing, 161, 162
testing semi-annual booking module, mobile product page design for, 160–161
154–156 online marketing strategies, 158–159
call-to-action elements. See CTA elements optimization strategies of, 159–160
Campbell, Emily, 166–171 personalizing self-service messaging,
164–165
 Index    281

qualitative research used by, 79–84 Hightail’s removal of, 211

testing buyflow customer information, testing IBM link for, 218
161–163 CTR (click-through rate), 19–20, 92, 192–193
communicating. See also email Curtis, Matt, 86, 258–261
during design comp phase, 122–123 customer-centric design. See also customers;
Foursquare design success, 192–193 user experience
with funnel reports, 98–99 Dell’s use of, 166–167
optimization results, 47, 125–126, 214 developing, 6–7, 178
PetCareRx’s optimization results, 61–62 framework for, 7–8
sharing Caesars test results, 154 online use of, 142
ways of, 126–127 customer loyalty programs
consistency American Express testing for, 151
data, 114, 125 Caesars Entertainment Corporation’s,
design, 122 153–154, 157
contextual inquiry method, 76 Club Pogo, 177, 178
conversion Marriott Rewards program, 226, 227
daisy-chain, 235 promotional messaging for Pet Points,
number of visits before, 97 256, 257
conversion rates targeting promotions to visitors, 228–229
Caesars’ redesign increasing, 157 testing Caesars’ redesigned pages for, 157
Dell’s security verification and, 167 customer service, 121–122, 161–163
evaluating, 97 customers. See also customer loyalty
programs; users
finding products with highest, 96
audiences of Washington Post, 268–269
measuring, 91
educating about business, 161–163, 265–267
plotting in funnel reports, 98–99
helpfulness to, 8
T-Mobile phone payment options, 267
identifying goals of, 66–67
coordinated optimization, 130, 131
interacting with business designs, 4–5
Corbett, Phil, 86, 213–219
meeting expectations of, 122
corporate reliability, 167
observing, 72–74
cost effectiveness. See also RPV
offering products for subscribers and
Comcast’s ROI, 159
guests, 177–184
economy in design testing, 30–31
online behavior of, 68–69
monthly recurring revenue, 210
prompting in test sessions, 74
T-Mobile’s ROI, 263
realigning business with goals of, 9, 211–212
creativity
recruiting for research, 73
decision making using data and, 185–186
researching goals of, 8–9
design testing’s impact on, 31–33
responding to annoyances of, 171
effects of pixel politics on, 27–28
self-service programs for, 159
credit card input errors, 237–238, 240–241
setting up segments representing, 117–118
CTA (call-to-action) elements
testing new products and services with, 9–10
Comcast’s testing for, 160–161
within and outside organization, 226
Dell’s simplification of, 168
customers’ goals. See specific case studies
282    Refining Design for Business

D industry trends in, 223

LinkedIn’s approach to, 221–222
Daily Information Needs (DIN), 196–198
Marriott panel, 227–228
daisy-chain conversions, 235
measuring success of, 192
data. See also metrics
meeting customer expectations, 122
algorithm for News Feed displays, 187–188
mitigating risk of new, 30
analysis of Optimization Results, 46
redesigning classified job listings, 273–276
analysis paralysis, 88
simplicity in, 120, 166–167
collecting user’s experiential, 196, 197–198
smallest screens first, 120–121
decision making using, 159, 185–186
testing multiple designs, 28–31
ensuring consistency of, 114
traditional process of, 26
fluctuation of, 93–94
Washington Post’s calendar redesign, 276–277
helps in understanding analytic, 66
design comps
insights and impact analysis for, 89–90
defined, 45–46
learning details behind, 66
developing PetCareRx, 55–60
Lenovo’s use of analytic, 93–101
tips for creating, 122–123
making decisions on bad, 26–27
translating wireframes into, 46
pairing with design, 34
designers
targeting promotions based on visitor,
228–229 balancing measurements with talent of, 146
use in design decisions, 29–30, 32 fears about design testing, 25, 31–33, 35
using online success metrics, 90–93 Google’s use of design teams, 203–204
Dell, 166–171 response to Google’s color testing, 200–201
business and customers’ goals for, 166 diary studies, 78
online marketing strategy for, 166–167 DIN (Daily Information Needs), 196–198
optimization programs for, 167 Dinner with Barack page, 20–21, 243–244
personalizing homepage experience, documentation
170–171 developing Executive Summary, 51–52
removing chat pop-ups, 171 how to create, 43–44
segmentation models for, 166 importance of, 42
testing coupons and shopping carts, sharing online spreadsheets, 236
168–169 sharing review results, 70–72
tracking results for, 167 dog fooding, 199
design Dollar Thrifty Automotive Group, 172–176
artistry in, 33 business and customers’ goals for, 172
consistency in, 122 email opt-out designs for, 173–174
creativity vs. testing in, 31–33 optimization program for, 172–173
developing customer-centric, 6–8, 142, testing email hero offers, 174–176
166–167, 178 donations. See also fundraising
effects of pixel politics on, 27–28 saving payment information page for,
Facebook’s approach to, 185–186 241–242
flat vs. skeuomorphic, 201 testing Donate page, 238–240, 244–248,
importance to businesses, 4–5 249–250
 Index    283

E innovation by teams at, 186

redesign of News Feed, 186–190
Electronic Arts, 177–184
release of News Feed layout, 189
business goals for, 177
research methodologies used by, 186–187
games marketed by, 177–178
team role in innovation, 186
product testing by, 178
failing faster, 30
Word Whomp testing by, 179–184
Favreau-Lessard, Simon, 86, 191–195
element discovery tests, 115
feedback
email
Google’s use of customer, 199
contests collecting addresses, 20–21
using customer panels, 77, 269
fundraising strategies for, 232
website forms for, 78–79
LinkedIn’s mobile template testing, 223–224
field studies at Google, 198
opt-out designs for, 173–174
fields
reinforcing content on landing page, 244,
245 address, 216
segmentation in, 235 auto-focusing on first name, 246
sending to targeted customers, 172–173, finding entry errors for, 232–233
174–176 marking required, 238, 239
updating lists regularly, 127 reducing credit card input errors, 237–238,
embedded optimization culture, 130, 131 240–241
errors T-Mobile testing of input, 263–265
detecting common entry, 232–233 testing background color of, 246
effects on site metrics, 124 flat design, 201
marking required fields to reduce, 238, 239 flipping a switch, 27
reducing credit card input, 237–238, focus groups, 75
240–241 fold lines, 119, 121
simplifying form to reduce, 238–240 Ford, Henry, 200
Type I/Type II, 109–110 Foursquare, 191–195
ethnographic studies, 75–76 best practices for testing, 193
executive sponsor, 135, 137 business and customers’ goals for, 191
Executive Summary, 51–52, 61–62 innovation at, 191–192
experiential business review, 69 measuring design success at, 192–193
experiential competitive reviews, 69–72 testing autocomplete for search results,
experiments. See testing 194–195
Express, 268 free services
Hightail’s paid and, 210
Word Whomp testing of, 182–184
F free shipping offers, 256, 257, 261
Facebook, 34, 78, 185–190 Fuhriman, Jeff, 143–145
algorithm for News Feed displays, 187–188 full-factorial multivariate test, 115
Bolt’s role with customers at, 185 fundraising
card sorting by, 188 auto-focusing on first name field for, 246
data and creativity in design of, 185–186 daisy-chain conversions for, 235
284    Refining Design for Business

fundraising (continued )
designing donation pages, 232–233
H
headlines, 121, 215–216
form field background color tests, 246, 247
hero offers and images, 174–176, 230
reducing credit card errors, 237–238,
Hertz Global Holdings, Inc., 172
240–241
heuristic reviews. See user experience
reinforcing email content on landing page,
244, 245 Hightail, 210–212
removing dollar signs, 246 business and customers’ goals for, 210
removing donate buttons from site, 244 homepage testing by, 211
strategies for, 231–232 success metrics for, 210–211
testing video for, 242, 243 testing product storage limits, 211–212
funnel report, 98–99 hiring team members, 137–140
homepage
Ally Bank’s testing of, 147–148
G Dell’s testing of, 170–171
games, 177–184 designing new versions of, 6–7
geo-targeting, 95 Hightail’s testing of, 211
global navigation testing, 161, 162 Marriott’s testing for, 227–229
global pre-login page, 215–216 PetCareRx message testing for, 252–253
Goff, Teddy, 236 prioritized test list for, 51
Going Out Guide redesign test, 276–277 hypotheses, 108, 109
Google, 196–209
approach to innovation, 196–198
collecting experiential data, 196, 197–198 I
developing query information sets, 204–207 IBM, 213–219
Instant, 209 blog about optimization results, 214
Knowledge Graph, 202, 204, 205, 207 business and customers’ goals for, 213
qualitative research used by, 198–199, 206 MVT testing by, 215–219
research on color, 35, 200–201 optimization program of, 213–214
Search, 34, 201–202 success metrics for, 214
search box modifications, 207–209 Illustrator product page, 143–145
sharing results via Google Documents, 236 images. See also case studies
success metrics tracked by, 201–203 A/B testing of links to photo tour, 230
teams within, 203–204 A/B testing of products and, 151–152
Voice Search, 200, 202 PetCareRx A/B testing of banner, 253–255
groupthink, 75 testing alternate, 21
guests implementing test design, 123
logging in with IBM ID, 216–219 incident optimization, 130, 131
offering products for subscribers and, Innovating for People (Maher), 76
177–184 innovation
guided editing flow, 224–225 creating new user experiences, 199–200
encouraging with design testing, 30
 Index    285

finding opportunities for, 76 KPIs (key performance indicators), 153

Foursquare’s approach to, 191–192 Krill, Eileen, 11, 77, 268–277
Google’s approach to, 196–198
LinkedIn’s approach to, 221
role of Facebook teams in, 186
L
Land, Edwin, 185
separating fads from, 104
Lau, Thomas, 150–151
insights and impact method
layouts
about, 89–90
best practices for, 120–121
evaluating Lenovo’s site metrics with, 94,
95–98, 101 IBM’s MVT testing of, 215–216
Lenovo’s use of, 93–101 PetCareRx test, 56–60
interviewing job candidates, 138–139 redesign of Facebook’s News Feed, 186–190
Iterative Optimization Methodology. See also legal approvals, 123
specific methodology stages Leibrock, Jane, 187, 188
about, 7–8 Lenovo, 93–101
core principles of, 8 path reports for, 99–101
documentation of, 42, 43–44 prioritizing test areas for, 102–103
illustrated, 43 using insights and impact method, 94,
Optimization Launch, 42, 46 95–98, 101
Optimization Plan, 42, 45–46 Levitt, Theodore, 85
Optimization Results, 42, 46–47 liberal arts backgrounds, 137–138
Optimization Roadmap, 42, 44–45 lifetime revenue (LTV), 210
organizational use of, 22–23 LinkedIn, 34, 220–230
realigning business with customer goals, 9 approach to design, 221
researching customer goals in, 8–9 business and customers’ goals for, 220
stages in, 42–43 challenges for, 221
starting from helpful point of view, 8 good user experience for, 220–221
trying new products and services, 9–10 innovation at, 221
iterative tests measuring design success for, 222
confirming results with, 226–227 optimization teams for, 223
Google’s use of, 206–297 role of research at, 221–222
numbering, 106 segmentation, targeting, and
personalization for, 222–223
Obama for America’s use of, 244–245,
249–250 links, 216, 217, 218
developing QA, 125
MVTs for help, 216, 217, 218
J promoting social, 148–149
job descriptions, 138, 139 Saks Fifth Avenue’s testing of, 259–261
testing appearance of IBM, 215, 218
K testing placement of photo tour, 230
website feedback, 79
Kahle, Chris, 119–120, 153–154
log-off portal testing, 148–149
Knowledge Graph, 202, 204, 205, 207
286    Refining Design for Business

login pages methodology for using segments and,

IBM global pre-login page, 215–216 117–118
target testing for Obama for America, Obama for America’s, 235
242–243 online success, 90–93
Washington Post, 271–273 ranking pages by key, 94–95
LTV (lifetime revenue), 210 tracked by Google, 201–203
LUMA Institute, 76 using online, 90–93
minimum recommended traffic levels,
110–112
M mobile devices. See also T-Mobile
Maher, Pete, 76 Comcast product page for, 160–161
marketing. See online marketing strategies designing first for, 120–121
marketing myopia, 85 LinkedIn’s email template for, 223–224
Marriott International, 226–230 single click donations from, 241–242
business and customers’ goals for, 226 MRR (monthly recurring revenue), 210
ethnographic studies used by, 75 multivariable tests. See MVTs
homepage A/B testing, 227–229 MVTs (multivariable tests), 81
metrics for, 226–227 IBM’s use of, 215–219
testing hero image, 230 prepaid activation field testing, 263–265
Martin, Sandy, 172–176 specifying in test methodology, 109
maturity model checklist, 130–132 types of, 114–115
Mead, Margaret, 76
messaging
A/B testing of PetCareRx, 252–257 N
best practices for, 121 naming recipes, 119
Comcast’s Social Security, 161–163 navigation testing
personalizing self-service, 164–165 Comcast global, 161, 162
working with political, 234–235 LinkedIn’s, 222
methodologies. See Iterative Optimization Saks Fifth Avenue, 259–261
Methodology; test methodologies New User Experience (NUX), 187
metrics New York Times, The, 35
changing, 113 News Feed layouts, 186–190
conducting QA on important, 124 numbering iterative tests, 106
consistency across recipes, 125 NUX (New User Experience), 187
Dell’s use of, 166
developing funnel reports for, 98–99
fluctuation over time, 93–94
O
Obama, Barack, 20–21
Foursquare’s, 192–193
Obama for America, 231–250
Hightail’s testing, 210
A/B testing for, 231, 232, 233–234, 236–248,
IBM’s, 214
249–250
Marriott’s, 226–227
adding presidential quote to, 246, 248
measuring LinkedIn design success, 222
brand styles and political messaging of,
234–235
 Index    287

customer research by, 232 Caesars Entertainment as, 154

Dinner with Barack page, 20–21, 243–244 Comcast strategies as, 159–160
Donate page testing for, 244–247 creating team for, 132–133
Kyle Rush’s role with, 231 cultural benefits of, 31
metrics used for, 235 Dell as, 167
saving payment information page for, developing business culture for, 22–23
241–242 Dollar Thrifty as, 172–173
segmentation, targeting and hiring team members, 137–140
personalization for, 235–236 IBM as, 213–214
site goals for, 231–232 Lenovo’s development as, 93–101
targeting login link testing, 242–243 maturity model checklist for, 130–132
testing platform speed for, 236–237 starting and scaling programs in, 160, 173, 223
video testing fork, 242, 243 steps for applying optimization method, 130
observational customer research, 72–74 T-Mobile as, 262–263
online marketing strategies team roles and responsibilities in, 133–135
banner blindness, 69 testing design as, 6
Comcast’s, 158–159 Optimization Plan. See also design comps;
Dell’s, 166–167 wireframes
marketing myopia, 85 about, 42, 45
marketing online games, 177–178 components of, 45–46, 108
research shaping, 85–86 developing for PetCareRx, 51–60
online presence roles and responsibilities for, 134
design and, 4–5 test methodologies in, 108–117
development of Washington Post’s, 11 optimization results
importance of, 3–4 blogs for communicating, 214
testing products and services with, 9–10 Caesars Entertainment, 154
online success metrics, 90–93 communicating, 47, 125–126
Optimization Launch Dell’s tracking of, 167
about, 42, 46 IBM’s blog about, 214
implementation, 123 Obama for America’s tracking of, 236
PetCareRx, 55 PetCareRx’s, 61–62
QA process during, 124–125 roles and responsibilities for, 134
releasing News Feed layout, 189 Optimization Results stage
roles and responsibilities for, 134 about, 42, 46
optimization lead, 135–136, 137, 139 analysis of, 46
optimization methodology. See Iterative communicating results, 47, 125–126
Optimization Methodology Optimization Roadmap
optimization organization, 133–137. See also about, 42, 44–45
teams; and specific case studies
assembling, 102–106
Ally Bank as, 146
PetCareRx, 47–51
American Express as, 150–151
roles and responsibilities for, 134
balancing existing and products in, 199–200
288    Refining Design for Business

P brainstorming on areas for testing,

103–104, 105
PACO (Personal Analytics Companion), 196
homepage test list of, 51
Page, Larry, 202, 204
setting design testing, 47–48
page views, 90, 92, 94
products
Parnell, Amy, 34, 96, 220–230
calls-to-action on mobile pages for, 160–161
partial factorial multivariate testing, 115
converting innovative ideas into, 198–199
path reports, 99–101
customer-centric design of, 178
payments
designing page with display for multiple,
options for T-Mobile phone, 265–267 151–152
providing security information during, developing innovative Google, 198–200
161–163
free shipping offers for, 256, 257, 261
saving information for, 241–242
homepage display of Ally Bank, 147–148
Personal Analytics Companion (PACO), 196
marketing games, 177–178
personalization
simplifying Illustrator page design, 143–145
Dollar Thrifty’s use of, 172–173
testing Hightail storage limits, 211–212
effect on LinkedIn design, 222–223
Washington Post, 268, 270–271
Obama for America’s use of, 235–236
prompting customers in test sessions, 74
personalizing homepage experience,
170–171
self-service messaging, 164–165 Q
PetCareRx, 251–257 QA (quality assurance), 124–125
about, 47 qualitative research, 65–86
business and customers’ goals for, 251–252 about, 9
design comps for, 55–60 acting on, 79
developing Optimization Plan, 51–60 benefits of, 66–68
failed expansion ideas for, 252 card-sorting, 78
finding customers for, 252 customer panels, 77
illustrated, 48, 49, 52–54 diary studies, 78
messaging testing by, 252–257 ethnographic studies, 75–76
Optimization Launch for, 55 experiential business review, 69
Optimization Roadmap for, 47–51 experiential competitive reviews, 69–72
origins of, 252 feedback forms, 78–79
RPV lift results for, 61 Google’s use of, 198–199, 206
updated Executive Summary for, 62 identifying customer goals with, 66–67
pixel politics, 27–28 methods for, 75–79
Pizzuto, Ryan, 262–267 Obama for America’s use of, 232–233
planned optimization, 130, 131 observational customer research, 72–74
platform speed testing, 236–237 questions for, 67–68
price testing, 121–122 shaping marketing strategies with, 85–86
priorities surveys, 77
basing on value and difficulty, 104–105 testing design ideas, 32
 Index    289

used by Comcast, 79–84 revenue and orders metrics, 90

user experience reviews, 68–72 revenue per thousand impressions (RPM), 192
Washington Post’s use of, 269–270 Revenue Per Visitor. See RPV
quantitative research, 87–101 Rogers, Kenyon, 75, 138, 226–230
about, 8–9 ROI (return on investment)
benefits of, 88 Comcast’s optimization program and, 159
Google’s use of, 199 T-Mobile’s testing and, 263
insights and impact method for, 89–90 roles and responsibilities
Obama for America’s use of, 232–233 Chris Kahle’s, 153
online success metrics, 90–93 design roles at Google, 203–204
questions for, 88–89 Eileen Krill’s, 268
queries Kyle Rush’s, 231
developing sets of information for, 204–207 Nate Bolt’s, 185
Google metrics for, 202 Simon Favreau-Lessard’s, 191
recipes for refining Foursquare, 195 team, 133–135
questions room bookings, 226–230
answering with quantitative research, Root, The, 268
88–89 RPM (revenue per thousand impressions), 192
identifying priorities, 67–68 RPV (Revenue Per Visitor)
Quick Donate, 241 Adobe designs for increasing, 143–145
calculating, 90–91, 154–156
R Comcast lifts in, 82, 84
defined, 51
Rathee, Manik, 234
Dell’s conversion rates and lifts in, 168–169,
recipes. See also specific case studies
170–171
naming, 119
increases for PetCareRx, 61
number tested, 109
Washington Post’s use of, 276
using consistent metric across, 125
Rush, Kyle, 20, 21, 231–250
Washington Post design, 12–19, 20
recruiting customers for research, 73
research. See also qualitative research; S
quantitative research Saks Fifth Avenue, 258–261
conducted by Washington Post, 269–271 best practices at, 258
done by Obama for America, 232 business and customers’ goals for, 258
Facebook’s methodologies for, 186–187 free shipping threshold testing, 261
Google’s color, 35, 200–201 left-hand navigation link testing, 259–261
Google’s DIN study, 196–198 measuring test success for, 258
recruiting customers for, 73 scaling
role in LinkedIn’s design, 221–222 Martin’s advice on, 173
results. See Optimization Results stage; Parnell’s advice on, 223
optimization results; search results user and A/B testing programs, 233–234
results library, 126 Williamson’s suggestions on, 160
return on investment. See ROI
290    Refining Design for Business

Schmidt, Eric, 204 redesigning shopping carts for, 168–169

Scholl, Roger, 86, 258–261 splitting one form into four steps, 238–240
search results skeuomorphic design, 201
developing querying sets for, 204–207 Slate, 268
Google search box modifications, 207–209 Smarter Planet website (IBM), 214
measuring effectiveness of, 192–193 social networking, 11–20, 148–149
recipes for query refinement, 195 software as a service (SaaS) business,
testing autocomplete feature for, 194–195 210–211
using Google Voice Search, 200, 202 strategic alignment, 130, 131
section types, 92 success metrics. See metrics
segmentation surveys
Dell’s model for, 166 qualitative research using, 77
Dollar Thrifty Automotive Group’s use of, Washington Post’s online user, 269, 271
172–173 Switzer, Andrew, 86, 146–149
effect on LinkedIn design, 222–223
examples of, 118
Obama for America’s use of, 235–236
T
T-Mobile, 262–267
setting up, 117–118
business and customers’ goals for, 262
testing pre- and post-game pages for user,
179–184 optimization programs within, 262–263
self-service prepaid activation field testing, 263–265
Comcast’s philosophy on, 159 ROI for testing, 263
personalizing messaging for, 164–165 Tai, Linda, 210–212
senior hires, 138, 139 targeting
services. See also products A/B testing on login links, 242–243
self-service programs, 159, 164–165 Dollar Thrifty customers, 172–173
software as a service businesses, 210–211 effect on LinkedIn design, 222–223
testing free, 182–184, 210 geo-targeting, 95
shopping carts Obama for America’s use of, 235–236
Comcast’s testing for mobile, 160–161 promotions based on visitor data, 228–229
Dell’s A/B testing of, 168–169 search results for timeliness, 193
evaluating paths to, 99–101 testing, 116
PetCareRx messaging tests for, 255–257 teams
removing chat pop-ups from, 171 allocating member time in, 136–137
significance level brainstorming test ideas, 103–104, 105
defined, 109–110 creating core optimization, 132–133
getting A/B test results quickly, 234 developing wireframes, 120–122
of Type I/Type II error occurrences, executive sponsor for, 135, 137
110–112 Google, 203–204
simplicity hiring members of, 137–140
in design, 120, 166–167 interviewing candidates for, 138, 140
developing News Feed redesign, 188–190 LinkedIn optimization, 223
 Index    291

optimization lead, 135–136, 137, 139 form field background color, 246, 247
prioritizing test areas, 102–103 Google’s use of, 35
regular meetings for, 127 hero offers and images, 174–176, 230
role in Facebook innovation, 186 hypothesis development for, 11–12
roles and responsibilities for, 133–135 IBM link for CTA, 218
senior hires for, 138, 139 keeping track of Caesars’, 154
setting testing priorities, 104–105 multiple design versions, 6
starting and scaling new programs, 160, new products and services online, 9–10
173, 223 numbering tests in, 106
technical lead, 136, 137 platform speed, 236–237
technical lead, 136, 137 priorities for, 47–48, 102–103
test methodologies, 108–117. See also purpose of, 113
A/B testing; Iterative Optimization QA process during, 124–125
Methodology; MVTs
quantitative questions shaping, 88–89
components of, 108–109
roadmap for PetCareRx, 47–51
duration of testing, 109
Saks Fifth Avenue, 258, 259–261
evaluating effect of traffic levels on,
selected web browsers, 125
110–113
site’s query refinement, 195
evaluating Type I/Type II errors, 109–110
Social Security messaging, 161–163
Facebook’s research, 186–187
types of, 109
insights and impact method, 89–90
using data in, 29–30
number of recipes in, 109–113
Washington Post layouts, 11–20
purpose and effectiveness of testing, 113
Thomas, Scott, 235
used in qualitative research, 75–79
toolbox options, 11–19
test objectives, 108
TOS (time on site), 93
test types
touch devices, 120–121, 200
A/B testing, 29, 81, 109, 114
traffic
about, 114
estimating minimum visitors per design,
auto-optimizing tests, 109, 116–117
110–112
multi-variable testing, 114–115
methodologies for low volume site, 113
targeting and personalization tests, 116
trust
testing. See also test methodologies
building customer’s, 121
autocomplete feature, 194–195
testing messaging for, 252–253
benefits of, 31
two-sided Z-test, 112
brainstorming ideas for, 103–104, 105
Type I/Type II errors, 109–110
Comcast’s use of, 79–84, 159
courage required for, 262
creativity in iterative, 32 U
defined, 29 usability testing shortcomings, 75
designing user, 234 user experience
duration of, 109, 234 business responses to, 72
fears about, 25, 31–33, 35 collecting data about, 196, 197–198
292    Refining Design for Business

user experience (continued )

creating new, 199–200
W
Washington Post, The, 11–20, 268–277
defining good, 220–221
audiences of, 268–269
documenting, 70–72
customer panels by, 77, 269
experiential business and competitive
default template for, 12
reviews of, 69
design recipes for, 12–19, 20
Google engineers involved in, 204
developing online business model, 11
LinkedIn’s focus on, 223
developing test hypothesis, 11–12
reviews of, 68–72
Krill’s role at, 268
user profiles
primary products of, 268
monitoring use experience of, 222
product testing for, 270–271
testing guided editing of, 220, 224–225
research conducted by, 269–270
user test labs
results of testing, 19–20
Google, 206
sign-in and registration page testing,
Washington Post, 269, 270, 271
271–273
users. See also user experience; visitors
user test labs, 269, 270, 271
customers vs., 185
web browsers, 125
don’t focus on advanced, 122
web sites. See also homepage; and specific
Google field studies with, 198 case studies
Google’s collection of experiential data art vs. function of, 33
from, 196, 197–198
detecting common entry errors, 232–233
live interviews of, 269–270
determining country from IP address, 95
query refinement for, 195
educating customers about business, 161–
testing experience of new, 187 163, 265–267
testing pre- and post-game pages for effect of pixel politics on design, 27–28
segmented, 179–184
experiential business review of, 69
testing sites with volunteer, 234
experiential competitive reviews of, 69–72
feedback links on, 79
V flipping switch on, 27
variables, 114–115 fluctuation of key metrics, 93–94
video testing, 242, 243 full-site redesigns of, 263
visitors. See also RPV internal optimization, 214
determining entry criteria for, 109 listing page views for, 94–95
registering site, 271–273 online customer behavior on, 68–69
targeting promotions to, 228–229 online success metrics for, 90–93
tracking, 90 page types for, 92
visits prioritizing areas for testing, 102–103
calculating number before converting, 97 redesigning TV Options page, 79–84
measuring time of, 90 reviewing, 68–72
Voice Search, 200, 202 setting testing priorities for, 47–48
 Index    293

template analytics for, 92 defined, 45

testing platform speed for, 236–237 developing for PetCareRx, 52–55
tracking page views, 90, 92, 94 PetCareRx, 52–55
WebMD, 66 structure of, 119
Webster, Steven, 134–135 tips for creating, 120–122
white space, 121 translating into design comp, 46
Wiley, Jon, 34, 35, 78, 196–209 Word Whomp testing, 179–184
Williamson, John, 79, 86, 158–165
wireframes
assembling, 119–120
Y
YouSendIt, 210
candidates’ skills in, 140