1|Page

BUSINESS RESEARCH METHODS

MBA CORE
DIVISION J
GROUP VIII
Professor: Dr. Purnima Mehrotra
Date of Submission: 14/11/2024

Group Members:
Amol Kawade- J008
Sagnik Sinha- J018
Pranav Maru- J028
Praveen PP-J038
Pratyush Sinha-J048
Akanksha -J058
Manjiri Patil- J068

2|Page
Executive Summary
The purpose of this research was to compare how efficiently AI-based chatbots and human
agents can resolve customer queries. Data was collected through surveys from the customers by
measuring the resolution rates, levels of satisfaction, and response times. For establishing
differences in outcomes between groups, statistical tests used were Chi-Square Test and Z-Test
for Proportions. This paper assesses the AI-based customer service in the Indian e-commerce
market along with several other key metrics, such as query resolution rates, customer
satisfaction, and response efficiency. The results would indicate whether the AI chatbots are
more efficient in handling high volumes of queries and providing faster responses or not, but are
human agents demonstrate superiority when tackling complicated issues with greater customer
satisfaction. Based on these findings, the report recommends a hybrid model that marries AI with
human agents' speed and scalability. It also suggests ongoing AI training, analysis of regular
feedback, and an adaptable customer service framework in order to improve net resolution rates
and enhance customer satisfaction.

Introduction
India's transition to digital has made e-commerce explode, compelling businesses to achieve
scalable customer support. Today, the e-commerce companies have a viable way to handle large
volumes of customer inquiries with AI-powered offerings such as chatbots and virtual assistants.
This change raises a question in everyone's mind: Will AI-based support ever match human
agents in terms of speed, satisfaction, and quality of issue resolution?
This study aims to take a close, critical look at how AI-powered customer service stacks up
against human agents in the Indian e-commerce sector. Here, it is obvious that between the
expectation of customers and the deliverables of current service solutions lies an invisible chasm
that this study looks to bridge. Culling from the growing desire for quicker, smoother, and more
satisfying customer experiences, this study examines the manner in which these applications
fulfill such needs. The findings of this study also will be strategic in providing insights to the e-
commerce companies in India about how the use of AI might improve the workings of the
customer service operation, boosting satisfaction Information worth any business moving
through this rapidly evolving market.
AI-based customer service is a very interesting topic of choice, given the exponential growth of
online shopping in India, which has fundamentally changed consumer expectations and demands
of customer support. As e-commerce continues to conquer, consumers increasingly expect fast
and streamlined as well as accessible service. The traditional models for customer service, often
restrained by human weaknesses, might not be able to meet growing expectations on a greater
scale. AI-driven platforms, including chatbots and virtual assistants, may seem to be the future
solution that can provide scale and response, but how well these efforts will play out remains to
be seen.

3|Page
Literature Review
In the e-commerce sector, for instance, artificial intelligence becomes a game changer by
providing cost-cutting as routine work like order processing, stock management, and support
work becomes automated. In the words of Srivastava (2024)1 "AI tools are used to optimize
workflows. This helps businesses use their resources with much more sense, in that end, there is
an increase in efficiency through these tools as human employees get freed up for higher-level
decision-making activities, thereby bringing greater productivity and satisfaction from
customers.".

India's rapidly growing e-commerce space feels the sense of urgency for efficiency much more.
As Srivastava mentions, AI automates time-consuming but very important tasks and allows e-
commerce companies to continue fighting for their market share in such a high-growth
environment. Improving resource management and maintaining a minimum level of constant
manual oversight, AI-driven systems make operations more efficient and cut costs. This is an
important competitive advantage for Indian e-commerce companies: thus, they are able to
maintain profitability and extend growth in a highly volatile marketplace.

In the e-commerce scenario of India, AI-based chatbots provide 24/7 support. They resolve basic
queries related to delivery or return dates or even provide information about a particular product
that would help in enabling customers to get answers to their queries faster. According to Bhatia
(2023)2, such chatbots lessen the waiting time and increase customer satisfaction for most of the
usual interactions. By answering repetitive questions, e-commerce companies can maintain low
operational costs but offer consistent customer service at any given time of the day, especially in
India. E-commerce often serves a widely spread and culturally diverse customer base in India.
AI is also modifying the search functionalities of Indian e-commerce by introducing voice and
visual search options that make shopping easier. Bhatia highlights that voice search will be a
game changer for those users who will start speaking regional languages or do not want to type.
It will bring online shopping to a larger populace. With such innovations in AI, Indian e-
commerce companies are not only enhancing the shopping experience but also making it more
inclusive for the diversified requirements of the users in this fast-growing digital market.

1
https://www.researchgate.net/profile/Ambar-Srivastava/publication/356635263_The_Application_Impact_of_Artificial_Intelligence_AI_on_E-
Commerce/links/61a6046a6864311d938a92b1/The-Application-Impact-of-Artificial-Intelligence-AI-on-E-Commerce.pdf
2
https://www.universalai.in/wp-content/uploads/2020/03/IMPACT-OF-AI-ON-CUSTOMER-RELATIONSHIP-MANAGEMENT-FOR-A-
SUSTAINABLE-FUTURE-%E2%80%93-A-COMPARATIVE-STUDY-BETWEEN-RETAIL-INDUSTRY-AND-E-COMMERCE-INDUSTRY-
IN-INDIA-KRI160621.pdf

4|Page
Over the last few years, AI-based customer service has really moved forward through the
effective use of advanced technologies such as NLP, machine learning, and deep learning. These
tools help AI systems understand exactly what the customer wants and, therefore, are able to
give accurate responses in real-time. Xiao (2024) 3 points out that this AI-based approach-very
often popular in finance and e-commerce industries-analyzes large volumes of data for
contextually relevant answers related to improving customer support. Outcome: Smoother
interactions would be more efficient and thus able to keep pace with demanding sectors that are
busy and in high traffic.

Xiao further feels that the AI chatbots need to become smart with diversified, context-aware
recommendation strategies. So, for instance, a chatbot may ask a customer to buy something
specific or suggest a few related items during the transaction because that is what the customer
may be looking out for, thus offering a more personalized and engaging experience. Such
personalized and relevantly focused recommendations can transform the very interactions with
customers into more engaging aspects, which eventually help to build much closer customer
relationships and enable such an e-commerce business to differentiate itself in its market.

Emotional comfort provided to customers by AI-driven customer service chatbots can only
provide the customer with a feeling of satisfaction in the case where complaints are addressed.
Song et al. (2021) 4 have presented a good example in the reference made about Alibaba's AliMe
Assist: this was a smart assistant which was developed for addressing the grievances of
customers on specific issues such as delayed delivery or low-quality products. The emotional
comfort framework, which AliMe Assist will be based upon, is classified in emotion and context-
sensitive responses in a bid to better evince human-like empathy, thus giving the customer a
feeling that the interaction is warm and supportive.
This framework combines multiple deep text classification techniques, including word
embeddings, n-grams, and sentence-level analysis, to identify very high accuracies of emotional
cues in customer messages and pinpoint topics precisely. In what is now being referred to as the
multi-Level feature-based Classification approach, or MLC for short, this chatbot is even more
capable of responding to those complex emotional states. Definitely a good asset in high-stress
situations when customer dissatisfaction is high. Such an effect occurs: with a satisfaction rate
rising by 8.7% once negative user emotions are detected, and it supports the idea of emotionally
intelligent AI as an important element to promote customer experiences. Since this method
increases quick satisfaction and fosters trust in AI-powered customer services, it is actually a
very important step forward for the online retailing industry.

3
https://drpress.org/ojs/index.php/EHSS/article/view/837
4
https://aclanthology.org/2021.naacl-industry.17.pdf

5|Page
Customer trust in AI-driven choices has increasingly been identified as an influential factor in
consumer behavioral intentions. Customer trust is defined as the expectation an individual has
about a company's honesty and action; thus, Muthuswamy (2024)5 postulated that the
effectiveness of AI-based decision-making, personalization, and service delivery will largely
depend on customer trust in engaging consumers and providing satisfaction. The study indicates
that increased trust in AI results in less perceivable risks in automated recommendations, hence
increasing openness and responsiveness to AI-driven interactions by the customers. Trust thus
forms an important factor that strengthens the salutary effects that AI triggers in consumers and
lowers psychological barriers in adopting AI.

Besides trust, facilitating conditions such as ease of access to AI interfaces, availability of

resources, and efficient provisions of support present a crucial factor in influencing consumers'
responses to AI. As indicated by the regression conducted by Muthuswamy, supportive
conditions are directly related to positive consumer behavior intentions, which again supports
that easy and intuitive AI interactions can significantly boost the customer experience.
Theoretically, this would mean that the more consumers are alleviated of barriers to interaction
with AI systems, the better their attitude toward AI recommendations becomes and, in turn, the
more positive their perception of services enabled by AI systems. Conclusion Being able to
incorporate trust-building strategies and supportive facilitating conditions is crucial in optimally
leveraging the contribution of AI to consumer intentions in e-commerce and similar industries.

AI-powered chatbots have become an inherent part of e-commerce today, maximizing scalability
and reducing response times while facilitating greater, more personalized interaction with
customers. According to Rahevar (2024)6, chatbots are effective when presented with high
volumes of interactions, as they support rapid, automated responses, making it possible for
platforms to provide responses each time customers demand them, which a human agent would
often find hard to continuously do over and over. Chatbots using NLP and machine learning
personalize product suggestions based on customer preferences and browsing history - they
enable a seamless and intuitive shopping experience. It is then based on 'tailored, streamlined,
intuitive product discovery' that leads the way to connecting with the customers based on their
tastes.

AI chatbots have a profoundly significant impact on customer satisfaction and trust. Since rapid
response times and accurate recommendations contribute to making the experience faster and
more enjoyable, it engenders reliability, which subsequently helps tighten brand loyalty. As seen
in Rahevar's study, an overwhelming connection exists between the level of accuracy of chatbot
recommendations and that of customer trust; it turns out well-constructed interactions in the
purpose-built chatbots raise significantly the level of retention in this e-commerce competitive
environment.
5
https://agbioforum.org/sobiad.org/menuscript/index.php/ijebeg/article/view/1942/610
6
https://ijmec.org.in/index.php/ijmec/article/view/48/33

6|Page
AI has gradually proven to be a helpful resource in customer service- crafting customer
experiences and providing prompt and automated responses. According to Abu Daqar and
Smoudy (2019)7, AI accounts for 26.4 percent of variance in the customer experience, thus it is
despite being moderate strength, its influence is nonetheless potent. Its strongest impact is on the
scene of personal service, with an explanatory power of 22.9 percent, which evidently shows
how it gives shape to the sense of experiencing through the analysis of prediction and real-time
personalization. But here, what is more critical is the relatively low explanatory power of AI at
after-sales support, accounting for only 7% variance- indicating human support or improving AI
methods to retain relationships post purchase.

While AI-driven service has a massive, positive impact on purchase decisions-69% of US

customers prefer companies offering better AI service, personalization efforts should be handled
with care. Close to 75% of surveyed customers said they were uncomfortable with targeted ads-
that's heralding in privacy issues. With immense ability to alter the customer experience,
businesses have to take pains to care about privacy when it comes to its implementation for it to
reach maximum potential as a competitive advantage.

Integration of AI chatbots has transformed service efficiency and personalization to new heights
and customer satisfaction. Chatbots are, according to Acharya and Shetty (2024) 8, basically AI-
endowed tools that maintain essential customer needs in real-time by providing instant, 24/7
support, which decreases the demand for human agents and increases the overall business
process efficiency. Constant availability will always meet today's expectations from consumers
and amounts to fast and reliable service. Customer satisfaction and better management of
processes involved in the customer service will increase because of this.

Artificially intelligent algorithms allow a chatbot to make its interaction with a user unique based
on what a user would want and behave, which strengthens a customer-brand relationship.
Simultaneously, the personalized chats of the chatbots increase customer satisfaction but also
raise loyalty and support toward the brand as the user will always be in preference of getting a
customized response rather than getting a generic response. Being two of the prime features for
any users, chatbots are most of the times meeting this expectation efficiently. However, users
appear to perceive human intervention as a core component of dealing with complex issues,
which could imply that there is a particular complementary role for AI and human agents in
providing the full breadth of customer experience.

According to Hanif (2023)9, Key performance indicators such as KPIs provide insight into the
effectiveness of AI in e-commerce in measuring customer engagement, or even operational
efficiency in respect to business growth. Other metrics applied are Customer Retention Rate and

7
https://www.econjournals.com/index.php/irmm/article/view/8166
8
https://sdmimd.ac.in/marketingconference2024/papers/IMC2466.pdf
9
https://www.jetir.org/papers/JETIR2309413.pdf

7|Page
Customer Lifetime Value (CLTV) to depict the kind of AI role in encouraging customer loyalty
by tailoring recommendations or targeted communications that lead to repurchase and long-term
relationships.

Sales Growth and Average Order Value are monetary harvesting from AI-driven upselling and
cross-selling activities. Customer Engagement Rate measures customer interactions with AI
functionalities, including the engagements associated with chats and targeted advertisements,
thus providing a more complex user experience. Operationally, AI's effects are seen in the
management of stock, as it impacts the inventory turnover ratio and the precision of demand
forecasting. Hence, the operational efficiency and cost savings driven by AI can be perceived in
aspects like automation of customer services and inventory management, where the reduction in
labor costs and human error contributes to efficiency. Customer satisfaction scores are how
customers perceive AI-powered support and shopping experiences. All of these KPIs reflect the
overall impact of AI in numerous e-commerce functions: customer experience, operational
processes, and revenue streams.

Digital Interactive Platforms have gained considerable popularity recently to enhance the
interaction between customers and the retailing industry. As suggested by Roy et al. (2023) 10,
DIPs, such as AI chatbots, transform how customers interact with a company. Using self-
determination theory, they prove that DIPs satisfy three basic psychological needs of autonomy,
competence, and relatedness, which enhances user satisfaction and subsequently engagement.
Such an alignment with SDT underlines how DIPs create person-specific experiences in order to
create a deeper sense of satisfaction and loyalty, more applicable in diversified markets like
India, where personalized and interactive digital experiences are becoming progressively valued.

Additional, from the perspective of S-D logic, DIPs play a role as a shift from a passenger type
of consumer to the active role in value co-creation. Roy et al. (2023) continue that AI-driven
platforms promote direct involvement of customers in the platform to co-create services and
evolve service experience based on a customer's meaningfully situation rather than transaction-
based interactions wherein collaboration between both a consumer and a brand can be fostered.
The transformation empowers the customers to co-create their digital retail experiences, thereby
increasing engagement and embedding users as integral contributors to the ecosystem of e-
commerce. Value co-creation with personalization and engagement may be used by DIPs toward
providing a framework for sustained involvement and stronger relationships between customers
and brands in the e-commerce segment.

Research Methodology

This is fundamentally a primary as well as secondary research study.

10
https://www.sciencedirect.com/science/article/pii/S0148296323003594

8|Page
This study adopted a cross-sectional research design and collected data at one point of time. For
this research, the data was collected using a convenient sampling method whereby the sample
respondents for the research were drawn. Data was drawn from a sample of 54 respondents
belonging to different demographic profiles across India. The research instrument used in this
study was a Structured Questionnaire.

Sampling and Data Collection

We will be doing a convenience sampling method which is a non-probabilistic sampling

technique. Convenience Sampling is a non-probability sample in which participants are selected
because they are more available or easier to recruit, rather than using a truly random selection.
This involves the data collection among people who are available and most convenient for the
researcher.
Reasons for selecting this sampling technique:
 Cost and time constraints: Convenience sampling will be used in this research due to
limited funds and a tight timeline. Convenience sampling allows the researcher to sample
those participants that are most accessible with little cost for broad outreach or survey
distribution. This method minimizes such expenses that result from more far-reaching
sampling and thus is an effective means for beginning a research study.
 Initial Findings in a Diverse Market: Demography, platforms, and consumer behavior
across the e-commerce industry of India are so wide that their wholesome representation
cannot be imagined without substantial resources. Convenience sampling can allow the
collection of initial findings from easily accessible customers. Such may reveal trends,
satisfaction levels, and general patterns in AI-based versus human-led customer service,
thereby leading to a foundation for future, more comprehensive research.
 Practical Accessibility: In the case of e-commerce, issues of data privacy policies and
accessibility to specific platforms usually restrict representative samples. Convenience
sampling targets participants in their accessible and willing states-for example, on certain
social media platforms or through their prior consumer networks. This avoids the barrier
of privacy and allows rapid data collection, with respondents who have experience with
AI customer service tools.

Ideal Scenario:
Stratified random sampling would be ideal in this particular study of AI-based customer service
in Indian e-commerce, since it represents the different strata of population segments. Stratified
sampling will ensure that there is representation in due proportion in each relevant stratum, such

9|Page
as e-commerce channels like Amazon India, Flipkart, and Myntra; types of AI-based tools which
may include chatbots, virtual assistants, predictive support AI; customer demographics by age,
region, gender, and income levels, and finally, types of customer queries-or actually, properly
speaking, ordinarily related to order status, returns, and payments. This becomes highly useful
when there are differences across customer segments as well as touch-points besides service
tools and query types.
The advantages come from the better representativeness and comparability of the results.
Because coverage is guaranteed for each stratum in this design, it picks up variations among the
Indian customer base and thus allows the analysis of its subgroups and testing the performance
of AI across platforms and customer demographics. This allows the testing of AI and human
customer services with regard to different scenarios, and the resultant insight into things like
response times, satisfaction levels, and effectiveness of resolutions across various demographic
or behavioral groups is even more profound.
The complication regarding the application of stratified sampling is that it can be very difficult. It
becomes too cumbersome or expensive in terms of the amount of information required to collect
detailed data for each stratum. Furthermore, since the stratification itself is enormously large in
number, besides demanding a sufficiently large sample size within each stratum, it also makes
this method rather expensive and time-consuming. This would be the strongest decision, given
the challenges faced, to obtain an overall and differentiated view of AI-based effectiveness in
customer service in the Indian e-commerce market.

Survey instrument
The most appropriate research tool for a survey aimed at determining the effectiveness of AI-
based customer service in the Indian e-commerce market must be a structured questionnaire. It
should contain questions that are both closed-ended and open-ended to collect relevant
quantitative and qualitative data to promote the possibility of providing measurable insights and
rich feedback.
SURVEY ADMINISTRATION CHALLENGES
• Competency of the respondents: This can lead to heterogeneous user competency with AI tools,
wherein Indian e-commerce customers have a vast range of proficiency in digital tools. Some
respondents, who don't know AI-specific terms like "NLP" or "emotionally aware chatbots," may
not fully understand the differences between AI and human- driven customer service, creating
potential inconsistencies in answers or even further requiring introduction before being
answered.
• Ultra-high dropout rates for long surveys: The number of questions spread over numerous
topics such as satisfaction, response time, and emotional engagement were simply getting too
much for participants. You would most likely have to reduce the length of the survey

10 | P a g e
significantly or implement some kind of motivational cue along the way in the survey to balance
this.
• E-commerce Platform-Experience Navigation: As Amazon, Flipkart, and other e-commerce
websites work on separate AI tools, it is difficult to generalize experience across the platforms.
Therefore, the outcome of various AI tools varies, and interpretation may be different. You may
need to ask questions that relate to specific platforms or group answers based on specific e-
commerce website for coherent interpretation of the data.
• Reaching diverse demographics: Even with convenience sampling, it would have been difficult
to ensure extensive enumeration from representative groups of diverse demographics, including
axes of age, language, or regional differences. Most the respondents fall within one range of age.
The challenge this might pose to response representativeness may require extended outreach
strategies or translations.

Limitations of using ideal sample search

While stratified random sampling proves very effective in attaining representativeness, it

does suffer from several drawbacks, especially when used in more complex studies such
as assessing AI-based customer services across various e-commerce platforms. Here's an
expanded look at the potential challenges:

1. Data Collection Problem: Stratified sampling requires all strata to be clearly defined and
discriminable. For instance, classification along the line of customer demographics, type
of AI tool, and query type require a lot of preliminary data collection efforts. But
collecting accurate data on the nature of customer preference, how often they interact,
and certain AI tool types, like chatbots or virtual assistants, is very hard if these variables
have not uniformly been reported or available on different e-commerce sites.
2. Challenges in Defining the Strata and Overlaps Overlap and/or vague categories can be
many, especially when using e-commerce platform, AI tool type, demographic segment,
and query type for stratification. This will find the customer to use more than one
platform or perform various types of interactions with different tools and queries. As
such, it becomes challenging to segment them- this calls for careful planning for the
definition of unique strata based on the distinct characteristics experienced with the
service by every group, which, in turn, can bring about significant complexity.
3. Potential Extremely Low Counts in Some Strata: When some sections of the population
are underrepresented-for example, users of single niche e-commerce sites or specific AI
tools-the danger is that one may not have a large enough dataset to support those strata.
This may force the need to over-sample in certain groups to strengthen the sample in
those areas, which is costly and logistically demanding.
4. Access and Privacy Issues: There may be limited access to the requisite information to
build strata, especially while working on e-commerce and customer service, due to

11 | P a g e
 privacy policies and data protection acts. Sometimes, high privacy restrictions might not
allow demographic information or types of AI-based interactions, thus resulting in an
unrepresentative sample over all the defined segments.
5. Inflexibility of the Scope of Changing Study: Stratified sampling assumes strata as
pertinent at the onset of carrying out the research for the study's purpose. If the scope of
research changes or if preliminary results suggest that more variables need to be looked
into are necessary-for example, different patterns of customer behavior-changing strata
by bringing in new variables can prove to be relatively inflexible. This may erode
flexibility toward accommodating emergent insights or findings that may unintentionally
bring out critical differences.
6. Greater complexity in analysis-In order to determine the final inferences with a high
degree of accuracy, often higher complexities of statistical techniques are required while
analyzing a stratified sample. For the same reason, that each stratum needs to be weighted
or aggregated to appropriately reflect the information in the whole population, this further
adds to the overall complexity of the entire process of data analysis, which could then be
more problematic with regard to expertise or resources.

Hypothesis Testing

Null Hypothesis:

1.Response Time

H0: There is no significant difference in the average response time between AI-based customer
service solutions and human agents in the Indian e-commerce sector.
Data Type: Continuous (since response time is typically measured in seconds or minutes, which
allows for a range of values).
Related question from the Survey:
 How satisfied are you with the response time of AI-based customer service solutions
versus human agents?
 How satisfied are you with the response time of AI-based customer service solutions
versus AI based chatbots?
Recommended test to be Used:
 Independent Samples t-Test: If the response times are approximately normally
distributed, use this test to compare the mean response times between AI and human
agents.

12 | P a g e
  Mann-Whitney U Test: This is a type of non-parametric test that is used in lieu for non-
normally distributed data, for an approximation to compare the median response times of
both groups.
Each of these tests would help determine if there is a statistically significant difference in
average response times.

2.Customer Satisfaction for AI Vs Human Agents

H0: No significant difference exists between interactions managed by AI-based customer service
solutions and human agents in terms of customer satisfaction levels across the Indian e-
commerce market.
Related question from the survey:
 On a scale of 1-5, how satisfied were you with the resolution provided by Human Agents.
 On a scale of 1-5, how satisfied were you with the resolution provided by AI-based
Chatbots
Data Type: Customer satisfaction scores are likely continuous, on a scale (e.g., 1 to 5 or 1 to
10).
Recommended test to be Used:

 Independent Samples t-Test: If the response times are approximately normally

distributed, use this test to compare the customer satisfaction times levels between AI and
human agents.
 Mann-Whitney U Test: If the data are not normally distributed, this non-parametric test
is an alternative to compare the median response times between the two groups.

3.Query Resolution Rate

H0: There is no significant difference between AI-based e-commerce customer service solutions
and human agents on the query resolution rate in the Indian e-commerce market.
Related questions from the survey:
 Was your issue fully resolved by the AI-based chatbot?
 Was your issue fully resolved by Human Agent

Data Type: Resolution rate is typically a binary outcome (resolved or not resolved for each
query), which can be expressed as a proportion for each group.
Recommended test to be used: Chi-Square Test for Independence or Z-Test for Proportions.
 Chi-Square Test: This test compares the observed resolution frequencies (resolved vs.
unresolved) for chatbots type vs humans to determine if there is a significant association.

13 | P a g e
  Z-Test for Proportions: This test is suitable if you have two proportions (the resolution
rates for each chatbot type) and want to determine if they differ significantly.

4.Escalation Rate

H0: The escalation rate (percentage of queries that need to be escalated to higher-level support)
does not differ significantly between AI-based customer service solutions and human agents.
Related questions from the survey:
 Did your query need to be escalated to a higher-level support agent? (For Human Agents)
 Did your query need to be escalated to a higher-level support agent? (For AI)
Data Type: Binary outcome (escalated or not escalated), which can be expressed as a proportion
for each group.
Recommended test to be used: Chi-Square Test for Independence or Z-Test for Proportions.
 Chi-Square Test: This can be used to compare the escalation rates for chatbots and
humans to determine if they are associated with a higher or lower escalation likelihood.
 Z-Test for Proportions: This test is used when the two proportions are directly to be
compared (escalation rates). It determines if the difference in escalation rates between the
Humans and chatbots is statistically significant.

5. Customer satisfaction across demographics

H0: Customer satisfaction with AI-based customer service solutions does not differ significantly
across different demographic groups (such as age, region, and digital proficiency).
Related questions from the survey
 Rate your experience (for AI Agents)
 Rate your experience (for Human Agents)
Recommended test to be used:
1.ANOVA (Analysis of Variance)
Purpose: ANOVA tests if there are statistically significant differences in mean customer
satisfaction scores across different demographic groups.
Application: Use ANOVA if each demographic variable (e.g., age groups, regions, or levels of
digital proficiency) has three or more categories.
2. Kruskal Wallis test if the data is non parametric in nature for comparing more than two means.
3.T-Test (for two demographic groups only)
Purpose: If a demographic variable has only two categories (e.g., high vs. low digital
proficiency), a t-test can determine if there’s a statistically significant difference in satisfaction
scores between these two groups. If the data is non parametric, the suggested test would be Mann
Whittney U test.
Application: Use an independent samples t-test if you are comparing satisfaction between two
independent groups.

14 | P a g e
Summary of Tests for Each Metric:

Type of
Metric Suggested Test(s)
Data
Independent Samples t-Test or Mann-
Customer Satisfaction Continuous
Whitney U Test
Chi-Square Test for Independence or Z-
Query Resolution Rate Binary
Test for Proportions
Satisfaction across demographics
Continuous ANOVA OR t test
for AI chatbots
Chi-Square Test for Independence or Z-
Escalation Rate Binary
Test for Proportions

Other Considerations
1. Effect Size. In interpreting good findings, the effect size of t-tests should be measured by
employing Cohen's d. For chi-square, effect size should be measured using Cramér's V in order
to interpret the strength of the difference between chatbot and human agent types.
2. Report of Confidence Intervals Report confidence intervals for all metrics of interest, to give
an estimate of the precision of differences.
3. Sample Size: Obtain adequate sample sizes that will guarantee sufficient power for each of the
tests, thus having the ability to detect true differences where it might be existing.
Such an approach, however, does deliver a stringent comparison on how emotionally aware
chatbots are compared to the default ones regarding the many types of customer service
performance metrics that might give statistical insight into performances between the two
categories.

Appendix
1. Name
2. Age Group

15 | P a g e
 3. Gender
4. Income Level
5. E-commerce Platform Used
6. Type of Customer Service Agent
7. How satisfied are you with the response time of AI-based customer service solutions
versus human agents?
8. Based on your experience, who responded faster?
9. On a scale of 1-5, how satisfied were you with the resolution provided by AI-based
Chatbots
10. How Satisfied were you with the solution offered by Human Agents? On the scale 1-5?
11. How Satisfied were you with the solution offered by AI Chatbots? On the scale 1-5?
12. Did the AI-based customer service agent seem emotionally aware or responsive to your
tone?
13. Rate your satisfaction with the emotionally aware chatbots (if applicable).
14. Was your issue fully resolved by the AI-based chatbot?
15. Was your issue fully resolved by Human Agent
16. Satisfaction with Query Resolution Rate for AI based agents?
17. Satisfaction with Query Resolution Rate for Human based agents?
18. Did your query need to be escalated to a higher-level support agent? (For AI)
19. Did your query need to be escalated to a higher-level support agent? (For Human Agents)
20. In your opinion, which agent type is more likely to escalate queries?
21. Rate your experience (for AI)
22. Rate your experience (for Human Agents)
23. After interacting with the AI-based chatbot, how likely are you to make a repeat
purchase?
24. Did your interaction with the customer service increase your likelihood of remaining
loyal to this platform? (For AI)

References

16 | P a g e
1. Srivastava, A. (2021). The application & impact of artificial intelligence (AI) on e-
commerce.
2. Bhatia, A. (2024). Impact of AI on customer relationship management for a sustainable
future: A comparative study between retail industry and e-commerce industry in India.
3. Xiao, X. (2024). E-commerce customer service satisfaction survey and intelligent
customer service development suggestion research.
4. Song, S., Wang, C., Chen, H., & Chen, H. (2021). An emotional comfort framework for
improving user satisfaction in e-commerce customer service chatbots. Alibaba Groups.
5. Muthuswamy, V. V. (2024). Impact of AI and e-business related factors on consumer
behaviour intention: Moderating role of customer trust.
6. Rahevar, M. (2024). The adoption of AI-driven chatbots into a recommendation for e-
commerce systems to targeted customer in the selection of product.
7. Abu Daqar, M. A. M., & Smoudy, A. K. A. (n.d.). The role of artificial intelligence on
enhancing customer experience.
8. Acharya, P. V., & Shetty, P. (2024). A study on AI and chatbots in customer experience.
9. Hanif, M. (2023). A study on role of artificial intelligence in e-commerce. Department of
Commerce, Vidyant Hindu PG College, Lucknow.
10. Roy, S. K. (2023). Customer engagement with digitalized interactive platforms in
retailing.

17 | P a g e