ORIGINAL ARTICLE

Design Thinking and AI: A

New Frontier for Designing
Human-Centered AI Solutions
by Martin B€
ockle PhD, Iana Kouris PhD

Abstract
In recent years, design thinking (DT) has become a pervasive innovation
approach in the managerial community as a structured way of solving
Martin B€ockle challenging problems through a human-centered perspective. The approach
Lead Strategic fosters lateral thinking and has a huge impact on the organizational culture
Designer, Boston and the core elements of the innovation process. At the same time, the
Consulting Group, number of organizations trying to solve problems through the application of
BCGX, London, artificial intelligence (AI) is constantly growing, although more than 80% of
United Kingdom AI projects never reach deployment due to the wrong strategic approach,
poor data quality, or a lack of AI awareness among employees—and those
that do remain below profitability expectations. The aim of this paper is
therefore to connect the AI and DT communities by proposing a first
framework supporting the development of AI solutions in a human-centered
way. The contribution of this paper is thus twofold: First, we investigate
ways in which design thinking adds real value along the design process of
intelligent solutions, thereby enriching the present body of design
management literature. Second, we propose a first framework by linking core
elements of both worlds to support strategic designers and data engineers in
Iana Kouris designing such applications. Finally, we propose a research agenda that can
Managing Director, serve as a basis for future research directions in order to advance
Boston Consulting contributions at the intersection of the two fields.
Group, BCGX, London,
United Kingdom Key words: artificial intelligence, design thinking, human-centered design

Introduction

I n recent years, design thinking (DT) has attracted significant interest

and increasing attention from both industry and academia as a novel
problem-solving methodology. The community is constantly growing,

20 ª 2023 DESIGN MANAGEMENT INSTITUTE (DMI).

 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
recently by management scholars of technology, and the emergent approaches like agile
(Liedtka, 2014; Verganti, 2020b) requirements for business product development, in which
who are focusing on design methods success” (Brown, 2014). iteration and experimentation
and how they are applied to Generally, this definition represent key components, or the
innovation challenges, since the ways highlights the need to connect lean startup approach, which
and forms in which design can designers, principles, approaches, highlights rapid iteration and the
impact the first phase of newly methods, and tools in a problem- development of a minimum viable
emerging technologies is an solving context, while describing product (MVP) to validate and
unexplored field (Verganti, 2020b). the three spaces of innovation: test the product with end users at
Although the term design thinking First, inspiration, which refers to an early point (Micheli et al.,
has been widely discussed in the actual problem and opportunity 2018).
prominent product management that aims to motivate the search Similarly situated, the process of
journals (Di Benedetto, 2012; Seidel for solutions; second, ideation, the designing artificial intelligence (AI)
and Fixson, 2013) such as the process through which ideas are solutions highlights the need for a
Academy of Management (Dunne and developed and tested; and third, more human-centered thought
Martin, 2012) as well as in business implementation, the phase in which process, since more than 80% of AI
publications like The Economist, the level of maturity is reached and projects never reach deployment, due
Harvard Business Review the developed artifact turns its the wrong strategic approach, poor
(Liedtka 2018), The Wall Street focus towards the market (Brown data quality, or a lack of AI
Journal (e.g., Wladawsky-Berger, and Katz, 2011; Liedtka, 2014). awareness among employees—and
2015), or The New York Brown and Katz (2011) also those that do remain below
Times (e.g., Parker-Pope, 2016), a mention that for any organization, profitability expectations
generally accepted definition is still these skills need to be dispersed (Chawla, 2020). Although the
lacking. and moved up to the executive current body of literature does not
This is not surprising, since level, to support but also inform provide a widely accepted process for
even the research stream around the strategic decision-making designing human-centered artificial
product design, which consists of a process. Over time, different intelligence (HCAI) solutions, we
body of literature with a higher models of design thinking have believe that the intersection of
level of maturity, still suffers from a been proposed, for instance by design thinking and AI provides a
large number of differing definitions IDEO or the Stanford Design valuable contribution to both
introduced over time (Liedtka, School (e.g., empathize, define, communities. Consequently, we aim
2014). Nevertheless, one of the ideate, prototype, and test), which to answer the following research
most widely known definitions, represent iterative cycles of questions:
introduced by Tim Brown, the exploration, idea generation,
CEO of the innovation consulting prototyping, and experimentation. RQ1: How can design thinking be
firm IDEO, which focuses on Moreover, Kimbell (2011) enabled towards a human-centered
product development and, more introduces different ways of AI approach?
recently, also on service and strategy describing and clustering design
design, defines design thinking as a thinking, for instance through What are the main challenges for the
“human-centered approach to cognitive styles, theory of design, development of intelligent solutions
innovation that draws from the or as an organizational resource. within organizations, and how can
designer’s toolkit to integrate the These characteristics of design they be tackled through HCAI
needs of people, the possibilities thinking also relate to other design practices?

21
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Management Journal

Thus, the aim of this paper is to intersection of the two fields within shows the potential of classifying the
connect the AI and DT communities the field of design management. explanations of AI solutions for
by proposing a first framework that different types of users—such as
informs the design process of developers, AI researchers, and
intelligent solutions in a more Research background and domain experts—by considering
human-centered way. The main related work their needs, objectives, and goals in
focus is placed on challenges like data different application contexts
collection, the intersection of user Human-Centered Artificial (Ribera, 2019). Furthermore, the
needs and AI capabilities, and the Intelligence framework developed by
explanation (xAI) of results to key Many different research streams are Wang (2019) includes relevant
stakeholder groups. The contribution related to the domain of human- theory on human decision-making,
of this paper is thus twofold: First, centered AI. First, ethically for instance how end users should
we investigate ways in which design responsible AI (Xu, 2009), which reason in order to inform the xAI
thinking practices can be enabled and highlights the objective of avoiding techniques (Wang, 2019) by
shifted towards a more human- discrimination and achieving fairness highlighting a taxonomy of
centered AI approach along the and transparency of the intended questions: “What is explained?” (e.g.,
design process of intelligent solutions solution. Second, the well-known data or model), “How is it
and thereby enrich the present body research stream of explainable AI explained?” (e.g., direct/post-hoc,
of design management literature. (xAI) aims to design explainable, static/interactive), and “At what
Design methods and skills are usable, and useful AI solutions level?” (e.g., local versus global)
mentioned by Borja de Mozota and (Xu, 2009; Riedl, 2019), since these (Arya, 2019). Such works show that
Wolff (2019) as criteria of design aspects have not been considered in AI policy and governance play an
management, which leans more the past due the strong technology- important role in the future of
towards co-creation with other driven focus. The application of xAI intelligent solutions.
experts in project teams (e.g., data practices is becoming more relevant
scientists) when designing intelligent as AI is increasingly applied in
solutions. Furthermore, there is a different end user applications, while Design Thinking and HCAI
discussion around the role and most end users, specifically those The current body of literature
potential of applying design thinking with a limited technical background, building a bridge between design
in user-centered AI scenarios, a perceive intelligent systems as a black thinking and AI is limited, but the
crucial element of design box and view them with a low level few studies that have been identified
management tools (Borja de Mozota of trust. Consequently, the design of reveal the potential of a meaningful
and Wolff, 2019). Second, we AI solutions needs to consider how intersection of the two communities,
propose a first framework linking to present the output to different although the question of how AI
core elements of the two worlds to user groups in order to increase the impacts design thinking practices in
support strategic designers and data level of trust. Questions like “Why the form of tools has not been
engineers in designing such did you succeed or fail? Why did considered. While design thinking is
applications and overcoming you do that? Why is this the result? discussed within the design
challenges within organizations. When can I trust you?” (Xu, 2009; management literature as a way to
Finally, we propose a research agenda Riedl, 2019; Preece, 2018) reveal the address new problems in
that can serve as a basis for future challenge of a one-size-fits-all organizations (Cooper et al., 2009),
research directions in order to approach when interpreting the the link to AI has been discussed very
advance contributions at the output of such systems. Recent work little, although it has been considered

22
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

as a future involvement of designers • Scale and standard design solutions the algorithm directs
(Borja de Mozota and Wolff, 2019). processes produce solutions that the learning strategy towards a
Relevant work on how to look at target several users of predefined better experience, with significant
design in the age of AI has recently customer segments or average implications on innovation
been proposed by Verganti archetypes, which have been (Iansiti and Lakhani, 2020). End
et al. (2020a, 2020b), highlighting considered through “personas” users always experience the best
that design work for AI solutions along the design thinking process. solution, which evolves over time.
should not focus on the ideation of AI capabilities remove this scale Consequently, in AI factories the
products to be commercialized at limitation in design, since “design-build-test “learning loop
scale, but rather on the design of intelligent solutions embed is fully automated, which means
problem-solving loops, described as design rules that are inherently that a new version of the product
human-capital-free design systems user-centered, for instance in the is released once the end user
that replace people with computing case of Netflix, leveraging a rich accesses the service (Verganti
power, which then will develop the stream of data on each individual et al., 2020b).
specific solution and deliver it to user (Verganti et al. 2020a) to
In addition to the academic body
the end user. develop a specific user experience.
of knowledge, resources like Google
The authors also mention that The development of these
PAIR (2021) inform towards a
AI does not undermine the basic experiences becomes very
practice-driven approach by
principles of design thinking but powerful as the number of users
providing design guidelines for
rather reinforces them by overcoming and the complexity of insights
developing such intelligent solutions.
past limitations (Verganti grows.
Similarly, the contribution of
et al., 2020b). Intelligent solutions • Scope and abduction refers to the
Microsoft (Amershi et al., 2019)
are much more user-centered due the fact that, in traditional design
provides such guidelines for human-
high level of granularity and practices, products and services
AI interaction in the categories
the provision of unique user are designed for a specific
“initially, during interaction, when
experiences (UX) for every single industry, with little flexibility to
wrong, over time.” We strongly
user of the application. A recent apply the developed solution to a
believe that these practices provide a
study by Iansiti and Lakhani (2020) different application context. The
strong contribution towards our
reveals that scale, scope, and learning design of AI solutions removes
proposed framework.
are highlighted as key differentiators these limitations by reframing
in designing such solutions (Verganti design artefacts in different
et al. 2020a), described through the domains. Verganti et al. (2020b)
Research design
term AI factory, a software that, for refer to the example of reusing
instance, runs millions of daily ad developed experiences in Netflix To answer the defined research
auctions at Google or Baidu, or for AirBnB. questions and build on a solid
decides about the availability of rides • Learning and iterations represent baseline, we conducted a structured
on Didi, Grab, Lyft, and Uber a limitation in standard design literature review (SLR) with the
(Iansiti and Lakhani, 2020). These processes to a certain degree, keywords “design thinking AI” and
algorithms also set the prices of once the developed product is “design thinking artificial intelligence”
products on platforms like Amazon. launched. Input for future and analyzed the existing literature at
However, at their heart, they use developments and revised the intersection of the two research
internal and external data for versions are informed by product communities. The literature review
predictions, insights, and choices: usage, while in intelligent addresses general and specialized

23
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Management Journal

scholars within the domain of AI and semi-structured interviews were character. An important aspect to
design thinking who are interested in carried out by two strategic designers highlight is that value is added by
novel AI and design thinking for a duration of one hour. combining AI with changes in
research from various domains. In Based on the developed insights, business processes and not through
the conceptualization phase of the we created a first framework that was AI itself. Consequently,
review, an initial explorative search designed in an iterative process by organizations should aim to design a
using Google Scholar was conducted adding veins of knowledge from the deployment process in which
to conceptually identify current design thinking community to guide technology is improved continuously
approaches and trends. Furthermore, data scientists as well as business by providing the right quality of
to identify core challenges in the professionals along the process of training data to the developed
development of AI solutions within designing AI solutions. The defined algorithms, while products and
organizations, we carried out five research challenges, which inform the business processes are adapted at the
semi-structured interviews with proposed research agenda, were same time.
selected data scientists who derived from the structure of the Results were collected from two
experienced the end-to-end process, proposed framework. workstreams. First, from the
starting from data collection to the structured literature review, and
actual business application. second, from the semi-structured
Results
Demographic information reveals interview sessions. The framework
that five out of five participants were A structured design process becomes proposed in Figure 1 is organized in
male data scientists, two of them with even more relevant in the three layers, where the top one
a senior profile and three within a development of AI applications due follows the design thinking model
leadership role in the age group 26– the rapid evolution of AI proposed by the d.School (HPI).
35. All participants had a master’s possibilities and the fact that every This layer aims to connect towards a
degree and were based in France. The AI solution is highly unique in its human-centered design approach by

FIGURE 1. Design Thinking/AI Process Framework. [Color figure can be viewed at wileyonlinelibrary.com]

24
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

proposing the second layer, a strategic participant observation, interviews, Very often, challenges are linked to
HCAI design layer that highlights an and customer journey mapping in an absence of the AI mindset needed
AI-informed, end-to-end design which identified problems and pain to develop such solutions. For
process. The bottom layer highlights points are highlighted. By mapping instance, AI needs to be embedded in
challenges identified within existing workflows, AI opportunities the company’s strategy, vision, and
organizations that inform the layers can be identified, especially those that purpose in order to develop a clear
described above. have not been articulated. perspective of where AI will drive
Quantitative approaches are helpful if business outcomes. Furthermore, AI
existing data is provided to development needs to be treated as a
Empathize understand historic behaviors of the business transformation, where
In this phase of the strategic design end user group. The AI canvas redesign is collaboratively organized
layer, the framework starts with the (Maillet, 2019) provides a toolbox to with business owners, including
pillar of the AI baseline, which better understand the intersection of iteration and improvement, based on
focuses on finding the intersection of user needs and AI strengths, but also the human-AI learning process.
user needs and AI strengths. These to place the outcomes of the
opportunities can be identified abovementioned methods. Generally,
through a process of mapping the the aim of this tool is to find a Define
existing workflow (Google PAIR). consensus between data scientists, The strategic design layer proposes
Consequently, the list of designers, and key stakeholders of the the identification and clear definition
improvements needs a decision as to intended AI efforts. The proposed of the intelligent behavior and what
where and how AI might add unique canvas includes topics like the AI is expected to do. This exercise is
value (Google PAIR). Generally, collaboration between human and in line with the second design
there are tasks where AI can add real machine activities, benefits for thinking phase, where the aim is to
value, for instance by recommending humans, and considerations and define a clear problem statement.
content to different user groups or implications, but it also covers change Guidance is provided by Google
predicting future events, but tasks management activities. The best PAIR, which suggest the definition
that might require complete practice, provided by Google PAIR, of the reward function used to
transparency are still a huge also emphasizes whether AI is being determine “right” versus “wrong”
limitation of the technology and used to automate a task or to predictions. The design of these
should be considered in the design improve a person’s ability to carry out functions aims to be a collaborative
process. the task themself (Google PAIR). process across disciplines, where UX
From a design thinking and The research activities described designers, product designers, and
product development perspective, the above should conclude on this topic engineers share their perspectives. A
aim is to understand the target group within the “empathize” phase. simple template for defining true
by analyzing their needs, goals, and Organizational challenges at the positives and false negatives is helpful
objectives through close interaction beginning of an AI initiative are not for the final definition of the
with them. Depending on the limited to understanding the intelligent behavior (e.g., our AI
context, Liedtka (2014) proposes technological landscape or model will be optimized for
ethnographic research, a qualitative the availability and quality of data, {precision/recall} because {user
approach to develop an although it is important that key benefit} (Google PAIR).
understanding by observing and decision makers comprehend the The ambition for value creation
interacting with them in their daily technological shift that might be through AI needs to be clearly
environment. Methods include needed as part of the future solution. articulated from an organizational

25
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Management Journal

perspective. This also includes setting Consequently, before starting to where AI can create the greatest
the focus and identifying the desired develop models in the process, it is value. This activity is usually done
AI outcome at the department level necessary to assess the maturity of through ideation sessions led by
to support the corporate strategy data capability and understand the designers, where several solutions are
(Ransbotham et al., 2019), for organizational limitations. Design created and discussed with data
instance in marketing (e.g., cross- and thinking activities that support scientists and engineers. The
upselling, churn and retention, and organizations within this step include identified opportunities should create
next best product and action) or the the application of the data landscape clarity on how AI will be used to
supply chain (advanced forecasting, canvas in order to understand and compete long-term in the economy.
simulations, and route optimization). categorize data assets in terms of In this case, the delta model
Challenging choices within the their source and ownership (Wirth proposed by Hax and Wilde (1999)
“define” phase include the definition and Szugat, 2020). provides three different strategic
of performance metrics, which options to define the scope of the
requires an understanding of how the intelligent solution. The first is to be
performance frontier may be evolving Ideate the best product on the market,
within the domain in which the The business process comes into play either through low cost or through
intended AI solution might be in this phase. Based on the design differentiation. The second is a
developed. This is very challenging principles proposed by Google PAIR, customer solution that might satisfy
due the fast-moving capabilities the decision to use AI to automate a a wider group of people, with a focus
of AI. task or augment a person’s ability to on the customer rather than on the
Other well-known carry out the task is crucial at this product economics (Hax and
organizational challenges reveal the stage, since the different ideas might Wilde 1999). The third is having a
availability of data, which includes consider certain ways in which AI system lock-in that covers all the
the cleanliness and understanding of can solve the problem and support players that contribute to
the source data and quality. the end users in accomplishing their the creation of economic value while
Organizations may lack, for instance, goals (Google PAIR). Newly providing the widest scope of all
the setup of an internal data developed ideas might focus on three options through bonding.
economy, such as when data increasing efficiency or reducing Furthermore, in the process of
scientists hoard data with no tedious tasks, while others might creating an idea and finding the focus
organizational support for ease of consider a higher level of involvement of the developed solution, design
publishing. Data access is often by the end users, where AI augments thinking provides a meaningful
limited, or flexibility of access their existing abilities. A successful framework for choosing and ranking
management may not be guaranteed, augmentation usually increases the developed ideas. The well-known
including strong data compliance end user’s enjoyment of a task or Desirability, Viability, Feasibility
guidelines that make the increases their responsibility and (DVF) approach is a valuable tool in
development of AI solutions control. the innovation process to move to a
challenging. Finally, individuals Thus, to tackle this challenge, more structured level of assessment
within organizations may not be previous insights from the and sophistication with the planned
aware of what data is available to “empathize” phase might be helpful business model.
them or where to find it. In most of in understanding the end users’ With a clear vision through
the cases, the data quality does not motivation of their daily tasks, ideation, organizations often struggle
meet certain thresholds such as those combined with analyzing current to integrate a clear data strategy that
regarding completeness and accuracy. business processes in order to assess embodies the baseline for designing

26
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

and developing the intended AI certain limitations attached. Best when an updated version of software
solution. In the ideation phase, the practice shows that the more development tools (e.g., PyTorch)
overall data strategy needs to be metadata is available, the more requires a code reassessment. The
strong and have a high level of options there are for developing a development process thus requires a
maturity before starting the software successful model, which also impacts high level of flexibility, combined
development process. This means the quality of the user experience with an adaptive design process.
that it has to enable more clarity and (UX) (Google PAIR) in the Solutions developed by any
understanding of data, as well as data subsequent steps. For instance, for organization always deal with certain
product discoverability, for instance supervised learning, one of the initial limitations when it comes to AI. One
through a data marketplace. Research steps is creating a labeled data set, of them is the lack of generalization
shows that every AI strategy needs to to be divided into training and (e.g., face recognition) or the bias of
be complemented by a data strategy. validation (Verganti et al., 2019). data to be used to train the
Recent research shows that three out The company Netflix is using this algorithm. Explainability is an
of four organizations declare AI as a approach for recommendations ongoing issue that is being researched
core component of their based on labeled data sets (e.g., heavily under the term xAI.
transformation plans, although only watched and liked movies), where a Furthermore, unintended behaviors
11% of them have started large group of user choices can lead (e.g., output of a GPT3-based
implementing a solid data strategy, to effective recommendations. In chatbot or self-driving cars) need to
i.e., designing effective data platforms this case, design thinking could be be taken into consideration as well.
and processes that enable effective used to identify opportunities for At this stage, design thinking has
machine learning approaches (Kiron incentivizing end users or trained huge potential to intervene with more
and Schrage, 2019). Further subject matter experts to participate creative approaches such as designing
elements of a proper data strategy in labeling activities. The design of communication patterns that enable
include evaluating the data and the the incentive approach, the labeler the right type of information to be
collection method in terms of experience, diversity, as well as presented to the right end user in the
whether they are appropriate for the existing tools need to be taken into expected format, based on the user’s
intended project, including consideration. Generally, approaches needs.
the documentation of contents and like gamification have the potential
decisions made during the data to motivate the end user to carry
collection process (Google PAIR). out certain tasks (B€ockle, 2017; Test
B€ockle, 2018); such approaches are In the testing phase of the developed
connected to either intrinsic or model, the strategic design layer
Prototype extrinsic motivational theories. refers to the user experience (UX) of
This section emphasizes the The decision on the software AI. From early on, it is necessary to
software development approach development approach has huge receive qualitative feedback through a
around the intended solution. The implications and a large impact on diverse group of end users.
AI technology on the strategic layer the business outcomes. There are Dashboards and customer data
highlights clarity regarding the several reasons to pivot in the visualizations, which are also part of
selected data strategy to follow. One development of AI solutions, for human-centered design practices,
of the most critical issues is how instance when a new source of data enable the UX quality of the
data is labeled (e.g., data, time, increases accuracy or lowers the developed system to be monitored.
content description) since data that computational costs, when a new Testing and tuning is an ongoing
is not properly tagged comes with channel offers the same service, or process for adjusting the ML model

27
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Management Journal

(Google PAIR) and needs to be • Mental model: This set of and enhance the overall learning
considered carefully. guidelines concerns the end users’ process through experimentation
Trust plays a central role in understanding of how AI systems and error resolution processes.
testing the developed AI solutions work and how their interactions
with the end user group. If the affect the interface. Generally, These best practices provide a
designed user experience does not mental models aim to set solid baseline for designing and
convey trust instantly and expectations about functionalities testing a meaningful user experience
consistently over time, end users and communication limitations. while at the same time fostering a
might quickly stop using the product • Explainability and trust: These high level of trust. Organizations
or service. Thus, to increase the level guidelines address how the end need to have a clear plan for their
of trust between the end user and the user receives an appropriate level testing strategy; for instance, what
developed intelligent solutions, of explanation regarding how the does the plan for early testing of the
certain design best practices have system works and the degree of model look like? Are the users diverse
been proposed. These include how to confidence in its output. After enough? Which metrics might be
meaningfully introduce the end user developing a clear mental model useful to measure whether the tuning
to the AI system and how to set and awareness of the system’s process is successful? Since design
expectations for adaptation and overall capabilities, these thinking provides a large repository
create effective mental models, which guidelines help end users learn of methods for testing, the
need to be considered in the testing how and when to trust the application of structured problem-
scenarios. underlying system. solving approaches enables the
Furthermore, guidelines show • Feedback and control: This set of quality of the developed model to be
that feedback is crucial to developing guidelines concerns the design improved.
trust in AI-enabled user interfaces, as of feedback and control
is explaining how to review, collect, mechanisms that provide a
Conclusion
and connect implicit and explicit meaningful end user experience
feedback to inform and enhance the (UX) when suggesting The result of the present work aims
user’s product experience (Google personalized content. These to support researchers and designers
PAIR). The Google design team mechanisms can also be used to in developing human-centered AI
proposed further guidelines on how improve the underlying AI model applications by applying a structured
to define errors and failure by output. design thinking AI process
providing a path forward from • Errors and graceful failures: framework to increase the level of
failure, since AI capabilities can These guidelines help identify maturity and success of the intended
change over time. As explainability is and diagnose AI context errors AI solutions. Generally, the
considered one of the major drivers and communicate the way developed framework can be used as
for increasing trust in these systems, forward. Context errors include a tool to guide and control the design
there are best practices that explain false starts, misunderstandings, process of such solutions, since there
how AI systems work by connecting and edge cases that cannot be is currently no structured process
explanations to the end users’ foreseen within the development that applies patterns of human-
interactions with the AI system. The process. Google suggests that centered design principles. To answer
proposed guidelines follow a human- these errors should be seen as the first research question, we
centered design approach and should opportunities to correct the end strongly believe that the development
be considered in the developed user’s mental model, encourage process of AI solutions is lacking
testing scenarios: the end user to provide feedback, creative approaches that actually have

28
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

the potential to increase the needs. Regarding RQ2, the identified connection between design thinking
likelihood of success. We therefore challenges were not limited to data and AI is limited. This is also one of
suggest that these approaches have a quality but more importantly the results of our structured
strong impact at the beginning of the included a lack of collaboration literature review. Consequently, the
design process, where baseline between departments including developed framework did not receive
elements like the vision or mission of frictions in sharing data on the strong input from the theoretical
the intended solution are still unclear operational level or the overall perspective but rather presents a first
or very loosely defined. Furthermore, missing AI mindset on the approach where existing practices
design thinking evolves its potential organizational level. We believe that have been merged with a non-
in the right selection of the intelligent design thinking is very well suited for structured AI development process.
behavior through the definition of a organizational change in the long run Second, most of the best practice
clear statement, while considering the but might be limited to having a large examples were selected from Google
users’ needs at the same time. Since impact on the operational level. PAIR, which provides a rich
the development of such solutions is Therefore, we suggest the application repository of examples for developing
closely linked to the user needs and of behavioral design approaches such such solutions. Third, since we only
the overall organizational design, as gamification, which are very carried out five interviews with data
there is a need to closely connect AI powerful in enabling behavior change. scientists and AI experts, the
with design thinking due the nature A crucial element is the definition of presented results are limited to
of the structured problem-solving the data strategy, which is often one the organization types (in terms of
approach. Consequently, by of the reasons why AI projects fail in size, structure, existing IT
intertwining design thinking practices the first run. We identified several architecture, etc.) where the
with phases of the AI development meaningful points of contact between participants provided their insights.
process, we propose the first the design thinking approach and the Fourth, the developed framework has
structured approach of an AI-enabled AI development process and believe not been tested and validated in a
design process by shifting the that this first framework serves as a real-world scenario and thus provides
standard design thinking approach baseline for further revision and a first approach to showcase the
towards a more human-centered AI improvements. connection between the existing
approach. Therefore, for RQ1, we literature in design thinking and AI,
believe that design thinking adds real mixed with insights from practice,
Limitations and future research
value in the first two phases highlighted through organizational
directions
(empathize, define), but also in the challenges. While there are many
testing phase where the user The present paper is also subject to unexplored issues in this domain, we
experience (UX) of the proposed AI several limitations. First, the current strongly believe that this paper makes
solution needs to match the end user body of literature investigating the a valuable contribution to the design

TABLE 1 Proposed research challenges

Research
challenge Description

RC1 How can the proposed framework be extended and shaped towards different types of organizations?

RC2 Which design thinking practices need to be revisited in order to meet the fast-evolving AI technology?

RC3 Which additional research streams should be merged with those of design thinking and AI in order to improve current practices
(e.g., behavioral design, gamification)?

29
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Management Journal

thinking and AI community by Journal, 12(01), https://doi.org/10. Iansiti, M. and K. Lakhani. (2020).
providing a first concept of how AI 4013/sdrj.2019.121.02. Competing in the Age of Artificial
should be treated along the end-to- Brown, T. (2014). Design Thinking Intelligence. Cambridge, MA: Harvard
end process in a human-centered Defined. Retrieved 28 March, 2022, Business Press.
way. Finally, based on our initial from: https://designthinking.ideo. Kimbell, L. (2011). Rethinking Design
com/. Thinking: Part 1. Design and Culture,
structured literature review (SLR),
Brown, T. and Katz, B. (2011). Change Volume 3(3), pp. 285–306.
we identified challenges that are
by Design. Journal of Product Kiron, D. and Schrage, M. (2019).
summarized in a proposed research Innovation Management, Volume 28 Strategy for and with AI. MIT Sloan
agenda in Table 1. This is intended (3), pp. 381–383. Management Review. Summer 2019
as a research outlook within the Chawla, V. (2020). Why Majority of Issue, Vol. 60, No. 4.
emerging and fast-growing research Data Science Projects Never Make It to Liedtka. J. (2014). Perspective: Linking
stream of human-centered AI. Production. Retrieved 25 March, Design Thinking with Innovation
2022, from: https://bit.ly/36EpZt7. Outcome through Cognitive Bias
References Cooper, R.F.D., Junginger, S., and Reduction. Journal of Product
Amershi, S., Weld, D., Vorvoreanu, M., Lockwood, T. (2009). Design Innovation Management. https://doi.
Fourney, A., Nushi, B., Collisson, P., Thinking and Design Management: org/10.1111/jpim.12163.
Suh J., Iqbal, S., Bennett, P.N., A Research and Practice Perspective. Liedtka, J. (2018). Why Design
Inkpen, K., Teevan, J., Kikin-Gil, R., Design Management Review. https:// Thinking Works. Harvard Business
and Horvitz, E. (2019). Guidelines for doi.org/10.1111/j.1948-7169.2009. Review, pp. 72–79. Retrieved
Human-AI Interaction. Proceedings of 00007.x. 13 March, 2022, from: https://hbr.
the 2019 CHI Conference on Di Benedetto, C.A. (2012). The JPIM org/2018/09/why-design-thinking-
Human Factors in Computing Thought Leadership Symposium. works.
Systems. Journal of Product Innovation Maillet, A. (2019). Introducing the
Arya, V, et al. (2019). One Explanation Management 29(3), pp. 344–348. Human-Centered AI Canvas.
Does Not Fit All: A Toolkit and Dunne, D. and Martin, R. (2006). Retrieved 25 March, 2022, from:
Taxonomy of AI Explainability Design Thinking and How It Will https://medium.com/@albmllt/
Techniques. arXiv:1909.03012. Change Management Education: An introducing-the-human-centered-ai-
B€ockle, M., Micheel, I., Bick, M., and Interview and Discussion. Academy of canvas-a4c9d2fc127e.
Novak, J. (2018). A Design Management Learning & Education, Micheli, P., Wilner, S.J.S, Bhatti, S.,
Framework for Adaptive Gamification Vol. 5, No. 4, pp. 512–523. Mura, M., and Beverland, M.B.
Applications. Proceedings of the 51st Google PAIR. People + AI Guidebook. (2018). Doing Design Thinking:
Hawaii International Conference on Retrieved 25 March, 2022, from: Conceptual Review, Synthesis, and
System Sciences. pair.withgoogle.com/guidebook. Research Agenda. Journal of Product
B€ockle, M., Novak, J., and Bick, M. Hax, A. C. and Wilde, D.L. (1999). The Innovation Management. Volume 36
(2017). Towards Adaptive Delta Model: Adaptive Management (3), pp. 124–148. https://doi.org/10.
Gamification: A Synthesis of Current for a Changing World. Retrieved 25 1111/jpim.12466.
Developments. Proceedings of the March, 2022, from: https:// Parker-Pope, T (2016). “Design
25th European Conference on sloanreview.mit.edu/article/the-delta- Thinking” for a Better You. Retrieved
Information Systems (ECIS). model-adaptive-management-for-a- 13 March, 2022, from: https://well.
Borja de Mozota, B. and Wolff, F changing-world/. blogs.nytimes.com/2016/01/04/
(2019). Forty Years of Research in HPI—Hasso-Plattner-Institut. School of design-thinking-for-a-better-you/.
Design Management: A Review of Design Thinking. Retrieved 25 March, Preece, A. (2018). Asking “Why” in AI:
Literature and Directions for the 2022, from: https://hpi.de/school-of- Explainability of Intelligent Systems
Future. Strategic Design Research design-thinking.html. —Perspectives and Challenges.

30
 19487177, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/dmj.12085 by EESC ESCP Europe, Wiley Online Library on [23/11/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

Intelligent Systems in Accounting, Management, Volume 37(3), pp. stream of human-centered artificial
Finance and Management Volume 25 212–227. intelligence (HCAI). Furthermore,
(2), pp. 63–72. Wang, D., Yang, Q., Abdul, A., and Lim he focuses on other topics such as
Ransbotham, S., Khodabandeh, S., B. (2019). Designing Theory-Driven design due diligence (DDD), design
Fehling, R., Lafountain, B., and User-Centric Explainable AI. leadership and design strategy for
Kiron, D. (2019). Winning with AI: Proceedings of the 2019 CHI
Metaverse applications.
Pioneers Combine Strategy, Conference on Human Factors in
Iana Kouris is a core member of
Organizational Behavior, and Computing Systems.
Technology. MIT Sloan Management Wirth, N. and Szugat, M. (2020).
BCG X, Boston Consulting Group’s
Review. Design Thinking for Data Products. tech build and design unit. She is
Ribera, M., Lapedriza, A. (2019). Can Retrieved 25 March, 2022, from: head of design for Central and
We Do Better Explanations? A https://archive.researchworld.com/ Eastern Europe for BCG X, and
Proposal of User-Centered Explainable design-thinking-for-data-products- leads customer experience work in
AI. In Joint Proceedings of the ACM part-3-of-3/. that region. Iana combines a human-
IUI 2019 Workshops, Los Angeles, Wladawsky-Berger, I. (2015). The centered design approach with her
USA. Evolution of Design Thinking. management background and tech
Riedl, M.O. (2019). Human-Centered Retrieved 13 March, 2022, from: expertise to achieve results for her
Artificial Intelligence and Machine https://www.wsj.com/articles/BL- clients. She is also a core member of
Learning. Human Behaviors and CIOB-8086. the firm’s Marketing, Sales & Pricing
Emerging Technologies. Volume 1(1), Xu, W. (2009). Toward Human-
practice. Iana has 15 years of
pp.33–36. Centered AI: A Perspective from
experience in leadership roles
Seidel, V.P. and Fixson, S.K. (2013). Human-Computer Interaction.
Adopting Design Thinking in Novice Interactions 26(4), pp. 42–46.
spanning business, design, and tech.
Multidisciplinary Teams: The She has particularly deep expertise in
Application and Limits of Design Author biographies customer experience, innovation, and
Methods and Reflexive Practices. design management, including
Journal of Product Innovation Martin B€ockle has a strong focus on customer journey redesign, product
Management, 30(1), pp. 19–33. designing human-centered digital design, design thinking, and service
Verganti, R., Vendraminelli, L., and experiences within the practice areas design.
Iansiti, M. (2020a). Design in the Age of BCGX solving strategic design Before joining BCG, Iana
of Artificial Intelligence. Working challenges through the application of worked at Nokia and Nokia Bell
Paper: Harvard Business School. qualitative and quantitative research. Labs for six years. She spent the first
Verganti, R., Vendraminelli, L., and He is a strong advocate for behavioral seven years of her career in
Iansiti, M. (2020b). Innovation design concepts like gamification and management consulting at McKinsey
and Design in the Age of Artificial contributes to the emerging research & Company Inc.
Intelligence. Journal of Innovation

31