See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/379284314

Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in AI-

Assisted Decision Making

Article in Proceedings of the AAAI Conference on Artificial Intelligence · March 2024

DOI: 10.1609/aaai.v38i9.28872

CITATION READS
1 13

3 authors, including:

Zhuoyan Li Zhuoran Lu
Purdue University Purdue University
9 PUBLICATIONS 22 CITATIONS 14 PUBLICATIONS 119 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Zhuoyan Li on 04 April 2024.

The user has requested enhancement of the downloaded file.

 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in

AI-Assisted Decision Making
Zhuoyan Li, Zhuoran Lu, Ming Yin
Purdue University, USA
[email protected], [email protected], [email protected]

Abstract the decision maker with an AI model’s decision recommen-

dation on a task (Passi and Vorvoreanu 2022; Wang, Liang,
With the rapid development of AI-based decision aids, differ-
and Yin 2023), though it is found that such assistance some-
ent forms of AI assistance have been increasingly integrated
into the human decision making processes. To best support times foster a degree of over-reliance on the AI recommen-
humans in decision making, it is essential to quantitatively dation (Ma et al. 2023). In contrast, the delayed recommen-
understand how diverse forms of AI assistance influence hu- dation paradigm, another form of AI assistance where hu-
mans’ decision making behavior. To this end, much of the mans are first required to deliberate on a task before receiv-
current research focuses on the end-to-end prediction of hu- ing the AI model’s decision recommendation, has the poten-
man behavior using “black-box” models, often lacking inter- tial to mitigate this over-reliance, but possibly at the cost of
pretations of the nuanced ways in which AI assistance im- increased under-reliance (Buçinca, Malaya, and Gajos 2021;
pacts the human decision making process. Meanwhile, meth- Fogliato et al. 2022). Thus, to best utilize diverse forms of
ods that prioritize the interpretability of human behavior pre- AI assistance and to determine when and how to present the
dictions are often tailored for one specific form of AI assis-
most suitable type of AI assistance to humans, it is critical
tance, making adaptations to other forms of assistance dif-
ficult. In this paper, we propose a computational framework to quantitatively understand and predict how these AI assis-
that can provide an interpretable characterization of the influ- tance influences humans on different decision making tasks.
ence of different forms of AI assistance on decision makers
in AI-assisted decision making. By conceptualizing AI assis- A few existing studies have worked on modeling and pre-
tance as the “nudge” in human decision making processes, dicting human behavior in AI-assisted decision making (Ku-
our approach centers around modelling how different forms mar et al. 2021; Bansal et al. 2021), but they come with sev-
of AI assistance modify humans’ strategy in weighing differ- eral limitations. For example, some research focuses on pre-
ent information in making their decisions. Evaluations on be- dicting humans’ interaction with AI assistance in an end-to-
havior data collected from real human decision makers show end manner using black-box models (Subrahmanian and Ku-
that the proposed framework outperforms various baselines mar 2017). While these methods can be effortlessly adapted
in accurately predicting human behavior in AI-assisted deci- to various forms of AI assistance, the black-box nature of the
sion making. Based on the proposed framework, we further model makes it challenging to unpack the cognitive mech-
provide insights into how individuals with different cognitive
anisms driving humans’ decision behavior under AI influ-
styles are nudged by AI assistance differently.
ence. Meanwhile, other studies that aim for interpretability
propose computational models based on economics or psy-
Introduction chology theories. For instance, Wang, Lu, and Yin (2022)
As AI technology advances, AI models are increasingly in- employed the Cumulative Prospect Theory (CPT) to under-
tegrated into the human decision making process spanning stand how humans decide whether to adopt AI recommen-
various domains from healthcare to finance. This has created dations by analyzing the utility and cost of different deci-
a new paradigm of human-AI collaboration—Given a deci- sion options. However, these models are typically tailored
sion making task, AI provides assistance to humans while for one specific form of AI assistance, making generaliza-
humans make the final decisions. To fully unlock the po- tions to other forms of AI assistance a challenging task that
tential of AI-based decision aids in enhancing human deci- requires significant methodology adaptation.
sion making, a growing line of research has been developed
The absence of a unified computational framework to
in the human-computer interaction community to develop
quantitatively characterize how diverse forms of AI assis-
different forms of AI assistance to better support decision
tance influence human decision making processes in an in-
makers (Lai et al. 2023). Each form of AI assistance shows
terpretable way impedes the further intelligent utilization of
different impacts on human decision makers. For instance,
AI assistance. As such, decision makers often have to inter-
the most intuitive type of AI assistance is to directly provide
act with the default forms of AI assistance instead of benefit-
Copyright © 2024, Association for the Advancement of Artificial ing from personalized and intelligent AI assistance that can
Intelligence (www.aaai.org). All rights reserved. best support them. Therefore, in this study, we aim to bridge

10083
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

this gap by proposing such a computational framework1 . tion (Green and Chen 2019; Guo et al. 2019; Zhang,
Specifically, inspired by Callaway, Hardy, and Grif- Liao, and Bellamy 2020; Levy et al. 2021), These indi-
fiths (2022) that explores the designs of optimal nudges for cators may help decision makers gauge the credibility of
cognitively bounded agents, we conceptualize the AI assis- AI recommendation and calibrate their trust in AI. Since
tance as a “nudge” to the human decision making process, various indicators of the AI recommendation serve simi-
which would modify how humans weigh different informa- lar purposes, aligning with prior research (Tejeda et al.
tion in making their decisions. Therefore, in our framework, 2022; Wang, Lu, and Yin 2022), we focus on model-
we first establish an independent decision model that reflects ing how immediate assistance influences human decision
how humans form their independent decisions without any makers when the model’s prediction confidence is used
AI assistance. We then model the nudge of AI assistance to as the indicator in this study.
humans as the alterations to their decision models. To eval- 2. Delayed recommendation (Park et al. 2019; Grgic-Hlaca,
uate the performance of the proposed framework, we collect Engel, and Gummadi 2019; Lu and Yin 2021; Buçinca,
data on real human subjects’ decisions in AI-assisted dia- Malaya, and Gajos 2021; Fogliato et al. 2022; Ma et al.
betes prediction tasks with the aids of three common types 2023): Humans need to first make an initial decision on
of AI assistance through a randomized experiment. By fit- the task before the AI model’s decision recommendation
ting various computational models to the behavior dataset is revealed to them; this type of AI assistance forces hu-
collected, we find that our proposed framework consistently mans to engage more thoughtfully with the AI recom-
outperforms other baseline models in accurately predicting mendation.
the human decision behavior under different forms of AI 3. Explanation only (Lucic, Haned, and de Rijke 2019;
assistance. Furthermore, the proposed framework demon- Alqaraawi et al. 2020; Rader, Cotter, and Cho 2018;
strates robust performance in accurately predicting human Schuff et al. 2022; van Berkel et al. 2021): Only the AI
behavior in AI-assisted decision making even with limited model’s decision explanation but not its decision recom-
training data. Lastly, through a detailed analysis of the nudg- mendation is provided to decision makers. The explana-
ing effects of AI assistance identified by our framework, we tion often points out important features of the task that
offer quantitative insights into how individuals with different contribute the most to the AI model’s unrevealed decision
cognitive styles are nudged by AI assistance differently. For recommendation, aiming to highlight information that AI
instance, we observed that AI explanations appear to show a believes as highly relevant for decision making.
larger effect in redirecting the attention of intuitive decision
For more details of the literature review, please see the sup-
makers than reflective decision makers.
plementary material. In this study, we focus on building
a computational framework to characterize how different
Related Work forms of AI assistance, such as the three types identified
Empirical Studies in AI-Assisted Decision Making. The above, impact humans in AI-assisted decision making.
increased usage of decision aids driven by AI models has Modeling Human Behavior in AI-assisted Decision Mak-
inspired a line of experimental studies that identify different ing. Most recently, there has been a surge of interest among
forms of AI assistance to enhance human-AI collaboration researchers in computationally modeling human behavior
in decision making (Lai et al. 2023). By surveying the lit- in AI-assisted decision making (Bansal et al. 2021; Ku-
erature related to AI-assisted decision making in the ACM mar et al. 2021; Tejeda et al. 2022; Pynadath, Wang, and
Conference on Human Factors in Computing Systems, ACM Kamireddy 2019; Li, Lu, and Yin 2023; Lu et al. 2023).
Conference on Computer-supported Cooperative Work and Many of these studies build their models on economics
Social Computing, ACM Conference on Fairness, Account- frameworks (Wang, Lu, and Yin 2022), which explain hu-
ability, and Transparency, and ACM Conference on Intelli- man decision making behavior under uncertainty, or on psy-
gent User Interfaces from 2018 to 2021, we identify three chological frameworks that describe the relationship be-
common types of AI assistance: tween human trust and reliance on automated systems (Aje-
1. Immediate assistance: The AI model’s decision recom- naghughrure et al. 2019; Li, Lu, and Yin 2023). However,
mendation on the decision making task and other in- most existing works are either tailored to one specific form
dicators of the recommendation are provided to deci- of AI assistance or lack interpretations of how AI assistance
sion makers upfront. Typical indicators of the AI rec- influences human decision making processes. Inspired by
ommendation include the AI model’s accuracy (Lai, Liu, the recent research in computationally modeling the effects
and Tan 2020), explanations of the AI recommenda- of nudges (Callaway, Hardy, and Griffiths 2022), we take a
tion (Poursabzi-Sangdeh et al. 2018; Cheng et al. 2019; different approach in this paper and build a framework to
Smith-Renner et al. 2020; Liu, Lai, and Tan 2021; Tsai characterize diverse forms of AI assistance as nudges in the
et al. 2021; Bansal et al. 2020; Zhang, Liao, and Bel- human decision making process.
lamy 2020), and confidence levels of the recommenda-
1
Methods
In this study, we narrow down the scope of our framework to
model the influence of AI assistance on human decision makers on
Problem Formulation
each individual decision making task. That is, we do not consider We now formally describe the AI-assisted decision making
the sequential or temporal influence of AI assistance on humans in scenario in this study. Suppose a decision task can be char-
a sequence of decision making tasks. acterized by an n-dimensional feature x ∈ Rn , and y is

10084
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

the correct decision to make in this task. In this study, we given a training dataset of the DM’s independent decisions
focus on decision making tasks with binary choices of de- D = {xi , yih }N
i=1 , we adopt a Bayesian approach and set out
cisions, i.e., y ∈ {0, 1}, and each feature xi of the task x to learn from the training dataset the posterior distribution
is normalized to fall within the interval of [0, 1]. We use of model parameters for a population of diverse DMs, i.e.,
M(x; wm ) to denote the AI model’s output on the deci- P(wh |D), instead of learning a point estimate. As directly
sion task (wm are model parameters), and it is within the computing this posterior P(wh |D) is intractable, we lever-
range of [0, 1]. Given M(x; wm ), the AI model can pro- age variational inference to approximate it using the param-
eterized distribution qϕ (wh ) = N (wh ; µϕ , Σϕ ) and mini-
vide a binary decision recommendation to the human deci-
sion maker (DM), i.e., y m = 1(M(x; wm ) > 0.5). The AI mize the KL divergence between qϕ (wh ) and P(wh |D):
model’s confidence in this recommended decision is cm = Z
qϕ (wh )
max{M(x; wm ), 1 − M(x; wm )}. Following explainable KL(qϕ (wh )∥P(wh |D)) = qϕ (wh ) log dwh
wh P(wh |D)
AI methods like LIME (Ribeiro, Singh, and Guestrin 2016) Z
and SHAP (Lundberg and Lee 2017), the AI model could qϕ (wh )
= qϕ (wh )(log − log P(D|wh ) + log P(D))dwh
also provide some “explanations” of its decision recommen- wh P(wh )
dation, e = E(M(x; wm )), e ∈ Rn , by highlighting the = KL(qϕ (wh )∥P(wh )) − Eqϕ (wh ) [log P(D|wh ) − log P(D)]
“important” features that contribute the most to the decision (1)
recommendation. Here, ei ∈ {0, 1}, where ei = 1 means the
feature xi is highlighted as important, while ei = 0 means where P(wh ) is the prior distribution of wh and P(D) is a
the feature xi is not highlighted. In addition, we assume that constant2 . Given the learned qϕ (wh ), and without additional
the human DM also independently forms their own judg- knowledge of a human DM’s unique independent decision
ment of the decision task, which is characterized by the func- making model, we can only postulate that the DM follows
tion H(x; wh ) whose output is in the range of [0, 1]. Thus, an average model to make their decision:
y h = 1(H(x; wh ) > 0.5) represents the human DM’s inde-
pendent binary decision. y h,t = 1(Eqϕ (wh ) [H(xt ; wh )] > 0.5) (2)
We consider the setting where the human DM is asked to
Moreover, after we possess additional observations of the
complete a set of T decision tasks with the help of the AI
human DM’s decision making behavior (e.g., the initial de-
model. For each task t (1 ≤ t ≤ T ), the human DM is given
cision y h,t that they make), we can update our belief of the
the feature vector xt and the AI assistance. As discussed
DM’s independent decision making model from the general
previously, we focus on studying the following three forms
parameter distribution qϕ (wh ) in order to align with the ob-
of AI assistance:
served human behavior:
• Immediate assistance: The AI model’s binary decision
recommendation y m,t and its confidence cm,t are imme- q̂ϕ (wh ) ∝ qϕ (wh ) · 1(1(H(xt ; wh ) ≥ 0.5) = y h,t ) (3)
diately provided to the DM along with the task xt .
• Delayed recommendation: The DM is required to first Without loss of generality, in this study, we assumed that hu-
make an initial independent decision y h,t on the task. Af- mans’ decision making model H(xt ; wh ) follows the form
ter that, the AI model’s binary decision recommendation of logistic model:
y m,t will be revealed to the DM. H(xt ; wh ) = sigmoid(wh · xt ) (4)
• Explanation only: The DM is only provided with the AI
model’s explanation et , which highlights the important Quantify the Nudging Effects of AI Assistance
features of the task that contributes the most to the AI
model’s unrevealed decision recommendation. Inspired by a recent computational framework for under-
standing and predicting the effects of nudges (Callaway,
The DM’s independent judgement on the task is y h,t —this
Hardy, and Griffiths 2022), in this study, we introduce a
is observed as the DM’s initial decision when AI assistance
computational framework to provide an interpretable and
comes in the form of delayed recommendation, but is unob-
quantitative characterization of the influence of diverse
served (thus requires inference) when AI assistance comes
forms of AI assistance on human decision makers, which
in the other two forms. Given both their own judgement and
enables us to predict human behavior in AI-assisted decision
the AI assistance, the DM then makes a final decision ŷ t
making. The core idea of this framework is to conceptualize
on the task. The goal of our study is to quantitatively char-
the AI assistance as a “nudge” to the human decision making
acterize how the DM is “nudged” by different forms of AI
process, such that it can modify how the human DM weighs
assistance in making their final decision on each task.
different information in their decision making and alter their
Model Decision Makers’ Independent Judgement independent decision model accordingly. Depending on the
type of AI assistance used, this alternation could be oper-
To characterize how AI assistance nudges human DMs in
AI-assisted decision making, it is necessary to first under- ationalized as the human DM changing their belief in the
stand how human DMs form their independent judgement relevance of certain task feature to their decisions, or as the
without being nudged by AI. That is, we need to quan- human DM redirecting their attention to certain task feature
tify human DMs’ independent decision model H(x; wh ). when making their decisions.
Since each DM may have their own unique independent de-
2
cision making model with different model parameters wh , In this study, P(wh ) is set to be N (wh ; 0, In ).

10085
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

Figure 1: The illustration of how immediate assistance

Figure 2: The illustration of how delayed recommendation
nudges human decision makers.
nudges human decision makers.

Immediate Assistance. As shown in Figure 1, in this

scenario, human DMs are directly presented with the AI AI model’s decision recommendation is provided. The ob-
model’s decision recommendation y m,t and confidence cm,t served human DM’s initial decision y h,t can be used to up-
before they deliberate on the task trial xt . Human DMs may date our belief of the DM’s independent decision making
consciously or unconsciously incorporate the AI recommen- model. Specifically, after observing the human DM’s initial
dation into their final decisions. The specific influence of AI decision y h,t , an adjustment is made to the distribution of the
on a human DM may largely vary with the DM’s inherent at- human DM’s independent decision model qϕ (wh ) to filter
titudes towards AI. For example, DMs with a high tendency out decision models that are inconsistent with the observed
to trust AI may simply adopt AI’s recommendation while decision, yielding q̂ϕ (wh ) as given by Eq. 3. Then, as the
skeptical DMs may simply adopt the opposite decision than
what AI suggests. In less extreme cases, DMs are not simply DM compares their initial decision y h,t with the AI model’s
trust or distrust the AI recommendation, but the AI recom- decision recommendation y m,t , two scenarios may arise:
mendation may change their belief of the relevance/impor- 1. AI affirms human decision (y h,t = y m,t ): For DMs who
tance of different task features so that they can either align trust AI, this agreement can boost their confidence in
more with the AI recommendation ŷ t , or deviate more from their initial decision y h,t . Conversely, for DMs who are
the AI recommendation ŷ t . The magnitude of this adjust-
ment may be controlled by the AI model’s confidence level skeptical of AI, they may become less confident in their
cm,t . Therefore, given any human DM whose independent own judgement due to this agreement.
decision making model is decided by wh ∼ qϕ (wh ), with 2. AI contradicts human decision (y h,t ̸= y m,t ): DMs who
the information y m,t and cm,t , the DM’s final decision ŷ t tend to trust AI might reflect on their initial decision and
would be nudged by y m,t and cm,t as: could be inclined towards switching to y m,t . On the other
hand, DMs who are skeptical of AI may be more inclined
ŷ t = 1(Eqϕ (wh ) [H(xt ; wh + (2y m,t − 1)cm,t δdirect )] > 0.5)
to stand by their own judgement y h,t .
(5)
Depending on which scenario that the DM encounters, we
where δdirect ∈ Rn (δdirect,i ·δdirect,j ≥ 0, ∀i, j ∈ {1, . . . , n}) model the DM’s final decision ŷ t as follows:
represents the updates in the DM’s belief of the relevance of
different task features after receiving the immediate AI as-
sistance. Note that δdirect,i > 0 indicates that the DM has ŷ t = 1(Eq̂ϕ (wh ) [H(xt ; wh + (2y m,t − 1)(1(y m,t = y h,t )δaffirm
a disposition to trust the AI (hence they update their belief + 1(y m,t ̸= y h,t )δcontra ))] > 0.5)
of task features’ relevance to align more with the AI rec- (6)
ommendation), whereas δdirect,i < 0 suggests that the DM
has a tendency to distrust AI (hence they update their belief Here, δaffirm , δcontra ∈ Rn (δaffirm,i · δaffirm,j ≥ 0, δcontra,i ·
of task features’ relevance to deviate more from the AI rec- δcontra,j ≥ 0, ∀i, j ∈ {1, . . . , n}) represent the updates in
ommendation). cm,t moderates the magnitude of the update, the DM’s belief of the relevance of different task features
and y m,t controls the direction of the update. For example, after seeing AI confirms their judgement (y m,t = y h,t ) and
if y m,t = 1, δdirect,i > 0 (or δdirect,i < 0) increases (or de- AI contradicts their judgement (y m,t ̸= y h,t ), respectively.
creases) the chance of the final decision ŷ t being 1 compared
to that of the DM’s independent decision y h,t . Conversely, Explanation Only. As shown in Figure 3, in this scenario,
If y m,t = −1, δdirect,i > 0 (or δdirect,i < 0) increases (or de- human DMs are presented with the AI explanation et , which
creases) the chance of the final decision ŷ t being −1 com- highlights the critical features of the task trial xt . These ex-
pared to that of the DM’s independent decision y h,t . planations may nudge human DMs to redirect their attention
to these highlighted features when forming their final deci-
Delayed Recommendation. As shown in Figure 2, in sion ŷ t . Intuitively, information that is marked as important
this scenario, human DMs are required to deliberate and might be prioritized by DMs, while other information may
make their initial decision y h,t on task trial xt before the be overlooked. As such, the highlighted task features may

10086
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

the four treatments. Subjects started the HIT by completing

a tutorial that described the diabetes prediction task that they
needed to work on in the HIT and the meaning of each fea-
ture they would see in a patient’s profile. To familiarize sub-
jects with the task, we initially asked them to complete five
training tasks. During these training tasks, subjects made di-
abetes predictions without AI assistance, and we immedi-
ately provided them with the correct answers and the end of
each task. The real experiment began after the subject com-
pleted the training tasks. Specifically, subjects were asked to
complete a total of 30 tasks, which were randomly sampled
from a pool of 500 task instances. After subjects completed
Figure 3: The illustration of how explanation only nudges all 30 tasks, subjects were asked to undertake a 3-item Cog-
human decision makers. nitive Reflection Test (CRT) (Frederick 2005), intended to
assess the subject’s tendency in engaging with intuitve vs.
reflective thinking. We offered a base payment of $1.2 for
exert a more salient influence on the DM’s final decision ŷ t : the HIT. The HIT was open to US-based workers only, and
each worker can complete the HIT once. We further included
ŷ t = 1(Eqϕ (wh ) [δexp H(et ⊙ xt ; wh )+ an attention check question within the HIT, where subjects
(7)
(1 − δexp )H((1 − et ) ⊙ xt ; wh )] > 0.5) were required to select a randomly determined option. Data
collected from subjects who successfully passed the atten-
where ⊙ is the element-wise product. δexp ∈ [0, 1] quantifies tion check were considered valid for our study (see the sup-
the degree to which the human DMs redirect their attention plementary materials for more details of the human-subject
to the highlighted features after seeing the AI explanation, experiment).
with larger values indicating that the DM puts a greater em-
phasis on the highlighted information. Evaluations
After filtering the inattentive subjects, we obtained valid data
Human-Subject Experiment from 202 subjects in our experiment (Independent: 53, Im-
To evaluate the proposed framework, we conducted a mediate assistance: 50, Delayed recommendation: 53, Ex-
human-subject experiment to collect behavior data from real planation only: 46). Below, we conduct our evaluation using
human decision makers in AI-assisted decision making un- the behavior data collected from these valid subjects.
der different forms of AI assistance. Model Training and Baselines
Decision Making Tasks. The decision making task we used We first learned the general parameter distribution qϕ (wh )
in our experiment was to predict diabetes in patients based of human DMs’ independent decision making model utiliz-
on demographic information and medical history. Specifi- ing the data collected in the Independent treatment. Through
cally, in each task, the subject is presented with a patient 5-fold cross-validation, we found the average accuracy in
profile encompassing six features: gender, age, history of predicting an average human DM’s independent decision us-
heart disease, Body Mass Index (BMI), HbA1c level, and ing qϕ (wh ) and Eq. 2 is 0.673. In the following evaluations,
blood glucose level at a given time interval. The subject was we used qϕ (wh ) to reflect the human DM’s independent de-
then asked to determine whether this patient has diabetes. cision making model, and used the maximum likelihood es-
The patient profiles were randomly sampled from the dia- timation to learn how different forms of AI assistance nudge
betes prediction dataset (Mustafatz 2023). each individual human DM to modify their decision making
Experimental Treatments. We created four treatments in model.
this experiment. One of these is an independent treatment Specifically, to evaluate the performance of the proposed
where human subjects complete decision making tasks with- framework, for each human subject, we randomly split the
out any AI assistance. In the other three treatments, hu- behavior data collected from them into training (50%) and
man subjects receive one of the three forms of AI assis- test (50%) sets. We computed the average negative log-
tance as what we introduced earlier. The AI assistant used in likelihood (NLL) to measure how well different models cap-
the experiment was based on a boosting tree model trained ture the likelihood of subjects making their final decisions
on the diabetes prediction dataset. The accuracy of the AI under the influence of AI assistance, and we averaged the
model was 87%. In the Explanation only treatment, we used NLL values across all subjects in each treatment. A lower
SHAP (Lundberg and Lee 2017) to explain the predictions mean NLL indicates a better prediction performance. In ad-
of the boosting tree model. The two most influential features dition, we also employed F1-score and Accuracy scores to
are highlighted as the AI model’s explanation. evaluate the performance of different models. For both these
Experimental Procedure. We posted our experiment on metrics, higher scores denote better performance. To ensure
Amazon Mechanical Turk (MTurk) as a human intelligence the robustness of evaluations, all experiments were repeated
task (HIT) and recruited MTurk workers as our subjects. 5 times, and the average performance across these repeti-
Upon arrival, we randomly assigned each subject to one of tions was reported.

10087
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

Immediate assistance Delayed recommendation Explanation only

Treatment
NLL ↓ Accuracy ↑ F1 ↑ NLL ↓ Accuracy ↑ F1 ↑ NLL ↓ Accuracy ↑ F1 ↑
Logistic Regression 0.522 0.753 0.789 0.446 0.809 0.782 0.549 0.728 0.767
XGBoost 0.533 0.768 0.737 0.472 0.812 0.797 0.617 0.711 0.753
MLP 0.656 0.753 0.729 0.554 0.777 0.751 0.606 0.686 0.778
SVM 0.530 0.754 0.707 0.461 0.791 0.758 0.603 0.721 0.743
Utility 0.573 0.739 0.779 - - - - - -
Ours 0.435 0.800 0.818 0.413 0.825 0.812 0.563 0.715 0.791

Table 1: Comparing the performance of proposed method with baseline methods on three forms of AI assistance, in terms of
NLL, Accuracy, and F1-score. “↓” denotes the lower the better, “↑” denotes the higher the better. Best result in each column is
highlighted in bold. All results are averaged over 5 runs. “-” means the method can not be applied in this scenario.

We consider utility-based model proposed by Wang, Lu,

and Yin (2022) and a few standard supervised learning mod-
els as baselines in evaluations, including Logistic Regres-
sion, XGBoost, Multi-Layer Perceptron (MLP), and Support
Vector Machines (SVM). These supervised learning models
directly predicts human DMs’ final decisions ŷ h,t in a deci-
sion task based on various features:
1. Immediate assistance: task trial features xt , as well as
the AI model’s decision recommendation y m,t and con-
fidence cm,t in the task trial. (a) Negative Log Likelihood (b) F1-score
2. Delayed recommendation: task trial features xt , human
DMs’ initial decision y h,t , and the AI model’s decision Figure 4: Comparing the performance of our method with
recommendation y m,t in the task trial. logistic regression models when changing the size of train-
ing data under three forms of AI assistance.
3. Explanation only: task trial features xt and the AI expla-
nation et in the task trial.
as the number of training instances reduces. In contrast, lo-
Comparing Model Performance gistic regression models are highly sensitive to the size of
the training data. As the number of training samples de-
Table 1 presents the comparative performance of various
creases, their performance degrades significantly. In other
models in predicting human DMs’ decisions across three
words, unlike the standard supervised learning models like
forms of AI assistance. Overall, our proposed method con-
logistic regression—which requires retraining from scratch
sistently outperforms the baseline methods in the Immediate
for each individual human DM—our approach is endowed
assistance and Delayed recommendation scenarios across all
with the knowledge of how human DMs in general make
metrics by a significant margin. For instance, within the Im-
decisions. This knowledge makes it possible for us to only
mediate assistance scenario, the NLL for our method stands
tune the parameters of AI’s nudging effects δ on each in-
at a mere 0.435, whereas the best baseline achieves an NLL
dividual human DM with a few training instances, yet still
of 0.522. In the Explanation only scenario, the performance
achieving comparable or even higher performance compared
of our method is comparable with the best-performing base-
to the supervised learning models.
line model, logistic regression, in terms of NLL and Accu-
racy, and outperforms it on the F1-score.
To assess the robustness of our approach, we varied the
Examining the Nudging Effect of AI Assistance
proportions of training and testing data and observed how across Individuals
the performance of our method changes with the training We now examine how may the AI assistance nudges de-
data size. Given the high performance of the logistic regres- cision makers with different cognitive styles similarly or
sion model, we selected it as the baseline model in this eval- differently. To do so, we compared the size of the learned
uation. As shown in Figure 4, our approach demonstrates nudging effects on decision makers for three forms of AI
consistently superior performance compared to logistic re- assistance. Specifically, we first used all behavior data col-
gression models across three AI-assisted decision making lected from a human DM to learn the nudging effect of AI
scenarios particularly when the number of training instances assistance on them (i.e., δ). We then used sign(δ)||δ|| to
is limited. Specifically, the performance of our model re- represent the direction and magnitude of the nudging ef-
mains robust with respect to variations in the amount of fects of AI assistance on the human DM (sign(δ) = 1
training data; it shows only a slight decrease in performance when ∀i, δi > 0; otherwise sign(δ) = −1). To catego-

10088
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

(a) Immediate assistance: (b) Delayed recommendation (c) Delayed recommendation (d) Explanation only: δexp
||δdirect || (y h = y m ): ||δaffirm || (y h ̸= y m ): ||δcontra ||

Figure 5: Comparing the nudging effects of AI assistance on decision makers with different cognitive styles across three forms
of AI assistance.

rize the cognitive style of each human DM (i.e., each sub- reflective DMs are significantly more likely to align their fi-
ject), we utilized their scores in the 3-item Cognitive Re- nal decisions with the AI recommendation when AI affirms
flection Test (CRT) in the experiment—Following previous rather than contradicting their initial judgement (p < 0.05).
research (Frederick 2005), subjects scoring 3 were classi- In contrast, the intuitive and moderately reflective DMs do
fied as having a reflective thinking style, those with a score not appear to be impacted by AI significantly differently un-
of 0 were categorized as having an intuitive thinking style, der these two conditions.
and those scoring 1 or 2 were categorized as the moderate Finally, under the Explanation only scenario, we also ob-
reflective style. serve that intuitive DMs tend to place more emphasis on
Figure 5 shows the comparison results of the nudging ef- the features highlighted by the AI explanations. The nudg-
fects of AI assistance on DMs with different cognitive styles. ing effect of AI explanations on intuitive DMs is found to
To examine whether the nudging effects of AI assistance are be significantly greater than that on moderately reflective
different across DMs with different cognitive styles, we first DMs (p < 0.05). While the nudging effect also appears to
used a one-way ANOVA test 3 to determine if there are sta- be slightly larger for intuitive DMs compared to reflective
tistically significant differences in the values of sign(δ)||δ|| DMs, the difference is not statistically significant.
across different groups of DMs. If significant differences are
detected, we proceed with post-hoc pairwise comparisons Conclusion
using Tukey’s HSD test 4 . Overall, our findings suggest that In this paper, we propose a computational framework to
under the Immediate assistance and the Delayed recommen- characterize how various AI assistance nudges humans in
dation scenarios (when AI affirms human decision), AI as- AI-assisted decision making. We evaluate the proposed
sistance exerts a larger influence on DMs with a reflective model’s performance in fitting the real human behavior data
thinking style than intuitive DMs (p < 0.05) and DMs with collected from a randomized experiment. Our results show
moderate reflective styles (p < 0.05). One potential expla- that the proposed model consistently outperforms other
nation is that reflective DMs are inclined to deliberate more baselines in accurately predicting humans decisions under
extensively on tasks and the AI model’s recommendations. diverse forms of AI assistance. Additionally, further analy-
Thus, through their interactions with the AI model, reflec- ses based on the proposed framework provided insights into
tive DMs may have sensed the high performance of the AI how individuals with varying cognitive styles are impacted
model (its accuracy is 87% for the task), making them more by AI assistance differently.
willing to align their decisions with the AI recommendation, There are a few limitations of this study. For example, the
especially when the AI recommendation affirms their own behavior data is collected from laypeople on the diabetes
independent judgement. However, when the human DM’s prediction task, which contains a relatively small number
initial decision differs from the AI recommendation in the of features. It remains to be investigated whether the pro-
Delayed recommendation scenario, there isn’t a statistical posed model can perform well on tasks that involve many
difference in the AI’s nudging effects across the three types more features and thus more complex. Additionally, the AI-
of decision makers. In fact, by comparing the AI’s nudging assisted decision scenario we examined in this study lacks
effects on different groups of DMs under the two conditions sequential or temporal feedback regarding AI performance.
of the Delayed recommendation scenario—AI affirms hu- Further exploration is required to generalize the propose
man decisions or contradicts human decisions—we find that framework to the sequential settings. Lastly, we assumed
3
Analysis of Variance (ANOVA) is a statistical test for identi- that the independent human decision model follows the form
fying significant differences between group means. of logistic regression. Additional research is needed to ex-
4
Tukey’s HSD (Honestly Significant Difference) is a post-hoc plore how to adapt the current ways of altering humans’ de-
test used to determine specific differences between pairs of group cision models for reflecting the nudging effect of AI assis-
means after a one-way ANOVA test has found significance. tance on human DMs to other forms of decision models.

10089
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

Acknowledgements Alternatives in Event Sequence Predictions. Proceedings of

We thank the support of the National Science Foundation the 2019 CHI Conference on Human Factors in Computing
under grant IIS-2229876 on this work. Any opinions, find- Systems.
ings, conclusions, or recommendations expressed here are Kumar, A.; Patel, T.; Benjamin, A. S.; and Steyvers, M.
those of the authors alone. 2021. Explaining algorithm aversion with metacognitive
bandits. In Proceedings of the annual meeting of the cog-
References nitive science society, volume 43.
Ajenaghughrure, I. B.; Sousa, S. C.; Kosunen, I. J.; and Lai, V.; Chen, C.; Smith-Renner, A.; Liao, Q. V.; and Tan, C.
Lamas, D. 2019. Predictive model to assess user trust: a 2023. Towards a Science of Human-AI Decision Making:
psycho-physiological approach. In Proceedings of the 10th An Overview of Design Space in Empirical Human-Subject
Indian conference on human-computer interaction, 1–10. Studies. Proceedings of the 2023 ACM Conference on Fair-
Alqaraawi, A.; Schuessler, M.; Weiß, P.; Costanza, E.; and ness, Accountability, and Transparency.
Bianchi-Berthouze, N. 2020. Evaluating saliency map ex- Lai, V.; Liu, H.; and Tan, C. 2020. ”Why is ’Chicago’ de-
planations for convolutional neural networks: a user study. ceptive?” Towards Building Model-Driven Tutorials for Hu-
Proceedings of the 25th International Conference on Intelli- mans. Proceedings of the 2020 CHI Conference on Human
gent User Interfaces. Factors in Computing Systems.
Bansal, G.; Nushi, B.; Kamar, E.; Horvitz, E.; and Weld,
D. S. 2021. Is the Most Accurate AI the Best Teammate? Levy, A.; Agrawal, M.; Satyanarayan, A.; and Sontag, D. A.
Optimizing AI for Teamwork. In AAAI Conference on Arti- 2021. Assessing the Impact of Automated Suggestions on
ficial Intelligence. Decision Making: Domain Experts Mediate Model Errors
but Take Less Initiative. Proceedings of the 2021 CHI Con-
Bansal, G.; Wu, T. S.; Zhou, J.; Fok, R.; Nushi, B.; Kamar, ference on Human Factors in Computing Systems.
E.; Ribeiro, M. T.; and Weld, D. S. 2020. Does the Whole
Exceed its Parts? The Effect of AI Explanations on Comple- Li, Z.; Lu, Z.; and Yin, M. 2023. Modeling Human Trust
mentary Team Performance. Proceedings of the 2021 CHI and Reliance in AI-Assisted Decision Making: A Markovian
Conference on Human Factors in Computing Systems. Approach. Proceedings of the AAAI Conference on Artificial
Buçinca, Z.; Malaya, M. B.; and Gajos, K. Z. 2021. To trust Intelligence, 37(5): 6056–6064.
or to think: cognitive forcing functions can reduce overre- Liu, H.; Lai, V.; and Tan, C. 2021. Understanding the Ef-
liance on AI in AI-assisted decision-making. Proceedings fect of Out-of-distribution Examples and Interactive Expla-
of the ACM on Human-Computer Interaction, 5(CSCW1): nations on Human-AI Decision Making. Proceedings of the
1–21. ACM on Human-Computer Interaction, 5: 1 – 45.
Callaway, F.; Hardy, M.; and Griffiths, T. 2022. Optimal Lu, Z.; Li, Z.; Chiang, C.-W.; and Yin, M. 2023. Strate-
nudging for cognitively bounded agents: A framework for gic adversarial attacks in AI-assisted decision making to re-
modeling, predicting, and controlling the effects of choice duce human trust and reliance. In Proceedings of the Thirty-
architectures. Second International Joint Conference on Artificial Intelli-
Cheng, H. F.; Wang, R.; Zhang, Z.; O’Connell, F.; Gray, T.; gence, 3020–3028.
Harper, F. M.; and Zhu, H. 2019. Explaining Decision- Lu, Z.; and Yin, M. 2021. Human Reliance on Machine
Making Algorithms through UI: Strategies to Help Non- Learning Models When Performance Feedback is Limited:
Expert Stakeholders. Proceedings of the 2019 CHI Con- Heuristics and Risks. Proceedings of the 2021 CHI Confer-
ference on Human Factors in Computing Systems. ence on Human Factors in Computing Systems.
Fogliato, R.; Chappidi, S.; Lungren, M. P.; Fitzke, M.;
Parkinson, M.; Wilson, D. U.; Fisher, P.; Horvitz, E.; Inkpen, Lucic, A.; Haned, H.; and de Rijke, M. 2019. Why does my
K.; and Nushi, B. 2022. Who Goes First? Influences of model fail?: contrastive local explanations for retail forecast-
Human-AI Workflow on Decision Making in Clinical Imag- ing. Proceedings of the 2020 Conference on Fairness, Ac-
ing. Proceedings of the 2022 ACM Conference on Fairness, countability, and Transparency.
Accountability, and Transparency. Lundberg, S. M.; and Lee, S.-I. 2017. A unified approach
Frederick, S. 2005. Cognitive Reflection and Decision Mak- to interpreting model predictions. Advances in neural infor-
ing. Journal of Economic Perspectives, 19: 25–42. mation processing systems, 30.
Green, B.; and Chen, Y. 2019. The Principles and Limits Ma, S.; Lei, Y.; Wang, X.; Zheng, C.; Shi, C.; Yin, M.; and
of Algorithm-in-the-Loop Decision Making. Proceedings of Ma, X. 2023. Who Should I Trust: AI or Myself? Leveraging
the ACM on Human-Computer Interaction, 3: 1 – 24. Human and AI Correctness Likelihood to Promote Appro-
Grgic-Hlaca, N.; Engel, C.; and Gummadi, K. P. 2019. Hu- priate Trust in AI-Assisted Decision-Making. Proceedings
man Decision Making with Machine Assistance: An Exper- of the 2023 CHI Conference on Human Factors in Comput-
iment on Bailing and Jailing. DecisionSciRN: Decision- ing Systems.
Making in the Legal Field (Topic). Mustafatz. 2023. Diabetes Prediction Dataset.
Guo, S.; Du, F.; Malik, S.; Koh, E.; Kim, S.; Liu, Z.; Kim, https://www.kaggle.com/datasets/iammustafatz/diabetes-
D.; Zha, H.; and Cao, N. 2019. Visualizing Uncertainty and prediction-dataset.

10090
 The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)

Park, J. S.; Berlin, R. B.; Kirlik, A.; and Karahalios, K. 2019. Wang, X.; Lu, Z.; and Yin, M. 2022. Will you accept the ai
A Slow Algorithm Improves Users’ Assessments of the Al- recommendation? predicting human behavior in ai-assisted
gorithm’s Accuracy. Proceedings of the ACM on Human- decision making. In Proceedings of the ACM Web Confer-
Computer Interaction, 3: 1 – 15. ence 2022, 1697–1708.
Passi, S.; and Vorvoreanu, M. 2022. Overreliance on AI Lit- Zhang, Y.; Liao, Q. V.; and Bellamy, R. K. E. 2020. Effect of
erature Review. Microsoft Research. confidence and explanation on accuracy and trust calibration
Poursabzi-Sangdeh, F.; Goldstein, D. G.; Hofman, J. M.; in AI-assisted decision making. Proceedings of the 2020
Vaughan, J. W.; and Wallach, H. M. 2018. Manipulating and Conference on Fairness, Accountability, and Transparency.
Measuring Model Interpretability. Proceedings of the 2021
CHI Conference on Human Factors in Computing Systems.
Pynadath, D. V.; Wang, N.; and Kamireddy, S. 2019. A
Markovian Method for Predicting Trust Behavior in Human-
Agent Interaction. Proceedings of the 7th International Con-
ference on Human-Agent Interaction.
Rader, E. J.; Cotter, K.; and Cho, J. 2018. Explanations as
Mechanisms for Supporting Algorithmic Transparency. Pro-
ceedings of the 2018 CHI Conference on Human Factors in
Computing Systems.
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. ” Why
should i trust you?” Explaining the predictions of any clas-
sifier. In Proceedings of the 22nd ACM SIGKDD interna-
tional conference on knowledge discovery and data mining,
1135–1144.
Schuff, H.; Jacovi, A.; Adel, H.; Goldberg, Y.; and Vu, N. T.
2022. Human Interpretation of Saliency-based Explanation
Over Text. Proceedings of the 2022 ACM Conference on
Fairness, Accountability, and Transparency.
Smith-Renner, A.; Fan, R.; Birchfield, M. K.; Wu, T. S.;
Boyd-Graber, J. L.; Weld, D. S.; and Findlater, L. 2020. No
Explainability without Accountability: An Empirical Study
of Explanations and Feedback in Interactive ML. Proceed-
ings of the 2020 CHI Conference on Human Factors in Com-
puting Systems.
Subrahmanian, V.; and Kumar, S. 2017. Predicting human
behavior: The next frontiers. Science, 355(6324): 489–489.
Tejeda, H.; Kumar, A.; Smyth, P.; and Steyvers, M. 2022.
AI-Assisted Decision-making: a Cognitive Modeling Ap-
proach to Infer Latent Reliance Strategies. Computational
Brain & Behavior, 5: 491 – 508.
Tsai, C.-H.; You, Y.; Gui, X.; Kou, Y.; and Carroll, J. M.
2021. Exploring and Promoting Diagnostic Transparency
and Explainability in Online Symptom Checkers. Proceed-
ings of the 2021 CHI Conference on Human Factors in Com-
puting Systems.
van Berkel, N.; Gonçalves, J.; Russo, D.; Hosio, S. J.; and
Skov, M. B. 2021. Effect of Information Presentation on
Fairness Perceptions of Machine Learning Predictors. Pro-
ceedings of the 2021 CHI Conference on Human Factors in
Computing Systems.
Wang, X.; Liang, C.; and Yin, M. 2023. The effects of AI
biases and explanations on human decision fairness: a case
study of bidding in rental housing markets. In Proceedings
of the Thirty-Second International Joint Conference on Ar-
tificial Intelligence, IJCAI-23, Edith Elkind (Ed.). Interna-
tional Joint Conferences on Artificial Intelligence Organiza-
tion, 3076–3084.

10091

View publication stats