The Effect of AI Assistance on Task Performance and Decision Confidence

- Ananya Bhagwat (019), Ananya Nahar (020), Aryan Gupta (033), Ellion Dsilva (049),
Ipshita Roy (063), Jui Gaikwad (073)

Artificial Intelligence (AI) are computer systems that are coded to perform tasks that typically
requires human intelligence such as pattern recognition, solving problems, getting information or
even decision making. In simple words, machines are taught to “think” in ways that resembles
human cognition. It is used in several areas such as Voice assistants (Siri, Alexa, etc.), Self
Driving Cars (Tesla, BYD, etc.) and even integrated in platforms such as Netflix or shopping
applications like Amazon for personalized recommendations and profiling.

With better and better Artificial Intelligence systems becoming increasingly popular and
accessible to the public, its use is being adopted by students and professionals the most to
“enhance” their efficiency in task completion. Students use AI tools such as ChatGPT,
Grammarly, Quillbot to generate essays, check mistakes in grammar and rephrase content, that
reduces the manual effort and thinking time. Working professionals use AI to analyse data,
automating routine tasks, content generation, including strategies and reports which are produced
significantly faster compared to the manual process and effort.

Despite the productivity boost that comes with the power of AI, concerns related to dependency
of humans on it and ethical implications are frequently raised. For example, gyration of essays
by the students using AI tools may result in the bypassing of the whole learning process because
they are not spending much time with the material, affecting their development of critical
thinking skills. Similarly, proessionals using Ai for decision making without understanding the
generated data’s accuracy and nuance risks wrong and biased decisions.

Artificial intelligence’s impact on real life decision making of humans has both positive and
negative consequences. On the one hand, AI is capable of processing vast amounts of data sets
and provide insights at unimaginable speed that humans can’t match. For example, in a lot of
places medical diagnosis has been improved through the analysis of patient history and reports
by integration of AI in institutions (Topol, 2019).

However, too much reliance on Artificial Inteligence may neglect the very important human
oversight if decisons are made based solely on the machine’s recommendation withought
actually getting the underlying logic of the diagnosis.

Concern of biases embedded in AI algorithms is also very much present because of their data
generation based on how and when they were trained (what sample sizes were fed, how old and
how advanced). (Raghavan et al., 2020).

Literature Support
 The paper titled ‘Learning with AI Assistance: A Path to Better Task Performance or
Dependence?’ by Karny et al. (2024) explores the use of AI assistance in task execution and the
effect it has on one’s performance on that task, including skill transfer and cognitive engagement.
The researchers designed a behavioral experiment to test whether dependence on AI decreases or
increases learning, autonomy and intrinsic skills in a sample of 189 participants comprising US
residents, both male and female, from 18 to 73 years of age. The participants were divided into
four groups: (a) group that did not receive any AI assistance throughout the experiment; (b)
group that received AI assistance only during the first block of the experiment; (c) group that
received AI assistance only during the second block of the experiment; (d) group that received
AI assistance throughout the experiment. The experiment administered an online game to the
participants on their personal computers, designed for them to score maximum points by
intercepting objects within a circular field. AI assisted them in following and intercepting these
objects depending on factors like “player’s position, object positions, direction and velocity of
object movements” (Karny et al., 2024). The results supported neither the positive nor negative
claims about AI assistance in task execution, suggesting that it can improve performance in
certain tasks without having any positive or negative effect. The authors conclude that there are
some tasks that are not entirely automatable and are benefitted from AI assistance without
threatening performance in humans. The experiment design by Karny et al. (2024) was able to
test several hypotheses to examine the effects of different levels of AI assistance. Their sample
size was also inclusive of many age groups. However, reliance on a gamified task limits the
generalizability and applicability of the study’s results in real-world scenarios where complexity
is higher. Moreover, their findings are inconclusive, prompting future research to consider a
larger task variety to understand the impact of AI assistance on human performance.

‘The Effect of AI Advice on Human Confidence in Decision-Making’ by Taudien et al.

(2022) examines how advice by AI affects human confidence in their decision-making, and if
disclosing the level of certainty of this advice leads to any changes in the same. The researchers
chose a total of 458 subjects to participate in their laboratory experiment involving an image
classification task. The participants were randomly divided into three groups: (a) group with no
AI advice; (b) group with AI advice; (c) group with AI advice and certainty of the same. Results
from the study revealed that AI advice led to an increase in the confidence of decision-makers in
their final decisions, but it was not often the desirable kind as overconfidence even in the wrong
image classifications occurred. Consequently, the researchers suggested that revealing the
certainty of AI advice could help mitigate overconfidence in any wrong advice given by AI to
aid human decision-making. The study by Taudien et al. (2022) has a clear focus, strengthened
by a controlled environment and design to provide insight on how AI is related to human
confidence in decision making. However, reliance on the laboratory setting may have limited its
generalizability in real-world scenarios, like the medical field, where controllability is low.
Hence, future research could further inquire into the findings provided by Taudien et al. (2022)
in real-world scenarios and tasks larger than image classification for a deeper understanding of
the collaboration between humans and AI.
 ‘Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and
Task Satisfaction’ by Hemmer et al. (2023) studies the effect of AI delegation on human task
performance and task satisfaction. The researchers designed a web-based experiment for a total
of 196 participants, both male and female, to test the hypotheses that AI delegation improves
task performance and task satisfaction in humans, and that an increase in self-efficacy results in
these effects. The results of this study confirm that humans experience an increase in their task
performance and task satisfaction because of their collaboration with AI regardless of their
awareness or unawareness of the same, and that an increase in human self-efficacy is responsible
for it. Hemmer et al. (2023) make an insightful contribution to the realm of AI-human
collaboration by presenting how AI delegation can enhance human performance and satisfaction
in tasks, and their findings are relevant to real-world scenarios like workplace settings where
one's satisfaction leads to better productivity, commitment and well-being. The experimental
design used in this study was also able to reveal how self-efficacy underlies and explains the
increase in human task performance and task satisfaction. However, Hemmer et al. (2023) do not
focus on human delegation, making their study on AI-human collaboration one-sided. Moreover,
the generalizability of their results is limited because their sample consisted largely of “non-
experts drawing upon a non-specialized task”, suggesting that AI delegation may not have the
same effects in environments that require expert knowledge. Thus, future research could inquire
into human delegation with a more inclusive sample and larger task variety.

Contemporary Relevance

“In recent years, the technological landscape has witnessed a remarkable transformation
with the widespread adoption of artificial intelligence (AI) systems across various
sectors” (Karny et al., 2024). AI has become an apparent part of our daily lives,
facilitating everyday tasks like transportation and navigation by automating them.
While it has the potential to bring about significant benefits for humanity, over-reliance
on AI assistance may lead to a decline in human independence, innovation and
creativity in making decisions and executing tasks. Intrigued by the nature of AI-human
collaboration, our study titled ‘The Effect of AI Assistance on Task Performance and
Decision Confidence’ aims to examine how AI assistance affects task performance and
decision confidence in humans. This study is highly relevant in the present, given how
the world is becoming increasingly driven by AI, widely using this advancing technology
in decision-making processes across industries like education and healthcare. Our
experiment would reveal the kind of impact that AI assistance has on human
performance and the kind of confidence it leads to in decision-making processes, the
results of which would suggest how positive effects can be utilized to human benefit or
how negative effects can be mitigated.

Research Question
How does the presence of AI-generated suggestions affect accuracy, response time, and
confidence in decision-making during a cognitive task?

Hypothesis and Objectives

It is hypothesized that participants who receive AI-generated suggestions will demonstrate

higher accuracy in their task performance due to the efficiency and precision of AI assistance.
However, this may be accompanied by lower decision confidence, as reliance on external inputs
could reduce participants' trust in their own abilities.

The objectives of the study are:

1. To examine the impact of varying levels of AI support (full, partial, and none) on task
accuracy.
2. To measure the response time of participants under different AI assistance conditions.
3. To analyze changes in decision confidence across groups with differing levels of AI
intervention.
4. To provide insights into the broader implications of AI assistance on cognitive
engagement and skill development.

METHODOLOGY

Participants

The experiment will have 60 subjects where these subjects will be split into three equal groups
of 20. Based on the inclusion criterion, the sample will include participants between, 18 to 45
years old who have somewhat a basic understanding about technology are or may be novices in
terms of using artificial intelligence services. In accordance with the purpose of the study,
participants for the study will be recruited from a disaggregate population and will not be
technologically savvy, will not be from the technology industry, and or not users of AI assistance
in their professional lives; however, will be university students, teleworkers, and working
professionals who occasionally or regularly use digital technology at their jobs. This selection
guarantees that participants are apt to participate in the task though may have different levels of
familiarity with AI which is core to the research.

Variables:

Independent Variable (IV): Degree of AI support with three considerations: full, partial and no
support.

Dependent Variables (DV):

Task Accuracy: Converting the grade from the problem solving task to the number of correct
responses.

Response Time: The period that was used on average to take each decision.

Confidence Level: Stated decision-making confidence obtained using a categorical range from 1
to 10.

Experimental Design:

The experiment will follow a Between-Subjects Design, where participants are randomly
assigned to one of three groups: A full AI suggestions phase, a partial AI hints phase, and a no
AI help (control) phase.

Task: Negative incentives are there for participants to undergo a set of questions or problems in
line with cold cognition, customarily in the form of numerous types of pattern matching, tests
involving numbers, and spatial problems. These tasks will challenge ‘intelligence’ in the number
of domains: spatial, for instance, which AI is still unable to address adequately.

To assess task accuracy, response time, and confidence, the questions will be like the ones used
in the IQ tests covering one or another type of cognition. Furthermore, simple spatial recognition
tasks will be incorporated to determine how the use of AI assistance affects performance in areas
in which AI may not produce answers.

Data Collection:

Response Time: To evaluate the influence of AI assistance the time participants take to answer
each question shall be recorded.

Eye-Tracking: The participants’ experiences and engagement with the task and potential AI
support will be captured through eye-tracking apparatus to compare approaches to
problem-solving in different teams. This means that the performance on the tasks, the confidence
in the decision made and whether or not the AI support has an impact on elapsed time and
distribution of cognitive processes can be compared.

Procedure:

Pre-Test Briefing: Participants will be informed of the fate of the task and told to try to do it as
precisely and self-assuredly as possible.

AI Assistance Manipulation:
Full AI Suggestions Group: The participants will be allowed to use an ordinary AI prompt, for
example, ChatGPT or Gemini open ended prompt freely to propose their solutions for each
question before they make their decisions.

Partial AI Hints Group: Users will have restricted access to the pilot AI tool where the number
of prompts will be less than the number of the questions and the reaction will be heuristic,
pointing them in the right direction.

No AI Assistance Group: There will be no use of Artificial Intelligence by the participants in

the task.

Task Execution: Participants will solve a set of 15 problem-solving tasks. To complement this,
they will give each answer a confidence level out of 10 at the end of each answer.

Data Collection:

For each participant the following will be documented:

● Greater care taken on a task (per cent correctness).

● Response time per question.
● A confidence rating for each answer thus has to be provided.

In order to minimize this risk and to ensure that all participants are using the same kind of AI,
they will be advised not to use any AI tool different from the one assigned to their group. This
approach will assist in the regulation of the externally oriented tools and the outcome of the
imposed AI assistance level.

Controls

The study will incorporate a number of controls to ensure the validity of the results across the 3
different groups. This will also allow for the accurate assessment of the impact of AI usage of
performance and confidence. Participants will be randomly divided into three groups: the full AI
group will be given unlimited access to the AI chatbot, the partial AI group will have a
restriction of 10 prompts through the whole questionnaire, the control group will be given no
access to the AI chatbot. To standardise the testing environment all participants will be made to
do the experiment on identical monitors and computers. In addition to this all the questions will
be set to a high school graduate level across all groups to ensure consistency in the difficulty.
Participants will also be instructed to maintain their focus on the screen throughout the
experiment to minimize distractions which may impact performance. At the end of the test
participants will then be asked to self-report their confidence in their answers.
Data analysis

For the eye tracking data an Area of Interest (AOI) Analysis will be used to measure the time
spent focusing on different elements (How to Analyze Eye Tracking Data, 2024), the
questionnaire itself and the AI chatbox. In addition to this descriptive statistics will be employed
to compare the performance metrics of response times, confidence levels, and accuracy of
answers across the 3 different groups. Post this using Pearsons Coorelation Coefficient, the
researchers will be able to examine the relationship between the usage of AI and response times,
accuracy, and confidence in participants. This also allows for the measure of the strength of the
corelation as well as its direction (Turney, 2022)

Expected Results and Discussion

It is anticipated that participants in the full AI suggestions group will demonstrate the highest
accuracy, as AI-generated suggestions can provide direct, efficient solutions to problem-solving
tasks. However, this group may report lower confidence levels, reflecting a reliance on external
assistance rather than intrinsic cognitive effort. The partial AI hints group may achieve moderate
accuracy and slightly higher confidence than the full AI group, as limited guidance allows for
more individual reasoning. The no AI assistance group is expected to show lower accuracy but
higher decision-making confidence, driven by the autonomy of problem-solving. These results
would underscore the nuanced impact of AI on cognitive tasks, suggesting that while AI can
enhance performance, it may inadvertently diminish self-assurance. The findings have significant
implications for educational and professional settings, emphasizing the need for a balanced
integration of AI to augment productivity without undermining individual confidence or critical
thinking skills.
These findings would greatly impact how AI functions in cognitive activities. While AI helps
improve accuracy and efficiency, it can also erode confidence, which could impact
decision-making when confidence is crucial. This demonstrates how AI support has two sides:
while it can improve results, it can also undermine users' confidence in their own skills.

The results broadly apply to AI ethics, workplace productivity, and education. They emphasize
the importance of creating AI tools that balance assisting and encouraging autonomous
decision-making. For example, partial AI cues may be more appropriate for skill-development
tasks since they encourage cognitive engagement without being overly dependent.

Furthermore, the eye-tracking data will reveal how users engage with AI, including hints of
cognitive involvement and methods for completing problems with varying degrees of assistance.
This could help designers create AI systems suited to users' cognitive requirements and
preferences.

Ultimately, the experiment advances a more complex knowledge of how humans and AI interact,
opening the door for more morally and practically integrating AI in various cognitive scenarios.
Conclusion

The experiment designed intends to explores the impact of AI-generated suggestions on task
performance and decision confidence in a cognitive task, revealing nuanced insights into the
interplay between human cognition and AI assistance. It is expected that while AI support
enhances accuracy and efficiency, it may undermine intrinsic decision-making confidence,
especially when full reliance is permitted. Partial AI support appears to strike a balance, enabling
cognitive engagement and self-assurance while still providing guidance.

These findings emphasize the dual nature of AI assistance—offering productivity benefits while
potentially reducing self-efficacy in users. By integrating AI tools thoughtfully, with
consideration of task complexity and user confidence needs, stakeholders in education,
workplaces, and beyond can maximize the advantages of AI without diminishing human
creativity and autonomy.

It also highlights the importance of ethically designed AI systems that foster a collaborative
relationship with users, supporting decision-making and skill development without fostering
dependency. These insights pave the way for future research into optimizing AI-human
interaction in varied real-world scenarios.

References

How to analyze eye tracking data. (2024). CoolTool.

https://cooltool.com/blog/how-to-analyze-eye-tracking-data

Turney, S. (2022, May 13). Pearson Correlation Coefficient (r) | Guide & Examples. Scribbr.

https://www.scribbr.com/statistics/pearson-correlation-coefficient/#:~:text=The%20Pears

on%20correlation%20coefficient%20(r,the%20relationship%20between%20two%20vari

ables.&text=When%20one%20variable%20changes%2C%20the,changes%20in%20the%

20same%20direction.

Topol, E. J. (2019). High-performance medicine: the convergence of human and artificial

intelligence. Nature Medicine, 25(1), 44-56. https://doi.org/10.1038/s41591-018-0300-7

Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020). Explaining Explanations in AI,

469-481. https://doi.org/10.1145/3287560.3287574

Learning with AI Assistance: A Path to Better Task Performance or Dependence? (2024).

https://dl.acm.org/doi/fullHtml/10.1145/3643562.3672610

Taudien, A., Fügener, A., Gupta, A., & Ketter, W. (2022). The effect of AI advice on human

Confidence in Decision-Making. Proceedings of the . . . Annual Hawaii International

Conference on System Sciences/Proceedings of the Annual Hawaii International

Conference on System Sciences. https://doi.org/10.24251/hicss.2022.029

Hemmer, P., Westphal, M., Schemmer, M., & Satzger, G. (2023). Human-AI collaboration: The

effect of AI delegation on human task performance and task satisfaction. ResearchGate.

https://doi.org/10.48550/arXiv.2303.09224