AI based Recommender Systems are shaping the Consumer Behavior: A

case study of Indian Youth

1. Abstract

E-commerce plays an important role in economic development and has become an indispensable part of
daily life in India because of the rapid growth of the country’s information technology and knowledge
systems. E-commerece heavily relies on customer reviews, these reviews are a testament of the credibility
of a product. They play an essential part in shaping consumer behaviour and have a notable impact on the
business of a product. Reviews have potential to portray the suitability of different products and its
aspects to a particular customer, which may reflect the efficiency of a recommendation. Analysing E-
commerce reviews can help producers gain an insight into customer expectations and programme
recommender systems accordingly. In our research we will analyse the overall sentiments of online
reviews and different aspects of various products that were appealing for a consumer, giving us a brief
idea of consumer choices and behaviour. Natural Language Processing (NLP) and Sentiment
Analysis(SA) has been used to perform this analysis detecting emotion behind online texts. Sentiment
analysis is widely applied with the goal of examining opinions, evaluations, attitudes, judgments, and
emotions toward a product.This method is used to conclude the amount of positive and negative reviews
that a product has received and further identify the opinions of consumers towards its various aspects.
Aspect-based sentiment analysis(ABSA) can identify different aspects of a product and determine the
corresponding sentiment of that aspect based on how they are related.

In our research, machine learning classifiers like logistic regression, SVM , decision tree, XGboost have
been used on the flipkart review dataset and later each model’s accuracy was studied. We have applied
ABSA to detect aspects associated with different products. This has helped us identify which aspects of a
product have negative sentiment and hence the company needs to work on improving it. We have also
executed the same classifiers after ABSA and drawn the conclusion that the accuracy of the models
increases after aspect detection.

Keywords: Online Reviews, Natural Language Processing, Consumer Behaviour, Sentiment Analysis,
Aspect Based Sentiment Analysis
2. Introduction

Sentiment Analysis (SA) or Opinion Mining (OM) is the computational study of people’s emotions,
attitudes and opinions toward an item [1], deciphering whether the tone of the emotion is negative,
positive or neutral.The item can represent individuals, events or topics. Sentiment analysis allows
organisations to gain insights into the vast volumes of unstructured data from different online sources like
social media, emails, chats, blogs, and forums. The terms SA and OM are interchangeable and share a
common meaning [2]. Some scholars believe that SA and OM have different meanings. Concept mining
extracts and analyses people's opinions on a topic, while sentiment analysis detects and analyses opinions
in text. From a data mining perspective, sentiment analysis or sentiment mining can be seen as a multi-
level classification problem.
Research in sentiment analysis has become popular with the increase in volumes of online text. Numerous
real-world operations require sentiment analysis for in depth investigation. For example, product analysis
discovers which aspects of a product appeal to customers in terms of quality [3]. Consumers frequently
go through reviews of products, restaurants and hotels before making a purchase.[4] Through sentiment
analysis the overall tone of these reviews can be identified. Content from a user’s perspective that is
posted through different online platforms such as forums, micro-blogs, or social networking sites may
potentially hinder the process of sentiment analysis. This may be due to online spammers and opinions
lacking quality[5]. This flaw has been overcome in this research as the dataset used of flipkart reviews go
through moderation before they are visible to other consumers, secondly each review must be submitted
with a rating which acts as a ground for comparison.

Sentiment analysis often focuses on classifying the overall sentiment in a text without clarifying the
content of the sentiment.[6] This may not be enough if the text is simultaneously referring to different
topics or entities (also known as aspects), possibly expressing different sentiments towards different
aspects. Identifying sentiments relating to specific aspects in a text is a more complex task known as
aspect-based sentiment analysis (ABSA).It is an advanced form of sentiment analysis that digs deep into
text to identify specific aspects or features of a product or service and determines the sentiment around it
all. This approach can give more insight into specific content that users find positive or negative.
Summarising reviews of products using Sentiment Analysis helps in determining the product features that
need improvement. Extracting sentiment from product reviews makes it easy for the brand marketing
team to reach its customers who need extra care and hence, benefits the business.
3. Material and Methodology
In this research paper we have analysed a dataset from Kaggle which consisted of product reviews on
flipkart https://www.kaggle.com/datasets/mansithummar67/flipkart-product-review-dataset using
sentiment analysis. This dataset contains 189874 rows and 5 columns of product information such as
product name, product price, rate, and review and summary of 104 different types of products.

Python libraries have been used in this research, namely- PIL, nltk, sklearn, textblob and wordcloud.
Python provides a favourable setting for carrying out this analysis due to its numerous libraries and
frameworks.

Data
Review Processing

Results Sentiment
Analysis

Figure 1. Proposed Methodology

Figure 1 explains the methodology used by us in this research in which data processing and cleaning is a
crucial step before carrying out Sentiment Analysis. This is important as it increases productivity and
efficiency of the results, data processing is essential for the findings of classifiers to be accurate. This
process also helps in facilitating tokenization. The right data set must be used to train a classifier properly.
Sentiment Analysis was then conducted in this research using different Machine Learning (ML)
algorithms, which helped us come to the most accurate results.

3.1 Data Processing

The first and most important step is data processing, unprocessed data can be troublesome to work with
and can hinder results of the analysis.

a. Data Cleaning: The reviews dataset consisted of unstructured and unformatted data which had to
be converted into a structured format. Data without a proper framework and structure is difficult
to work with and causes unnecessary errors while running programmes. Structured data draws
attention to the characteristics in reviews that we want the algorithm to identify as tokenization
becomes easier. Tokenization is when text is broken down into smaller units to make it
meaningful to the machine without losing the text’s initial essence. We cleaned the data set by
carrying out following steps:
1) All text was converted into lower case to maintain uniformity and punctuation
marks were replaced with spaces to ease tokenization.
2) Any missing values (NaN) were replaced with spaces
3) Non Alphanumeric characters were removed.
4) Common stop-words were filtered out using NLTK's English stopwords list. Stop
words are words frequently used in verbal or written language but do not carry
useful information. For eg: a, is, are, that, etc.
5) Phrases like "mind blowing" were also removed to prevent bias in sentiment
analysis. Words like these are hyperboles used in text to express emotions with
exaggeration, these words can often be taken in opposite meaning by the algorithm
which makes it important to filter these words from the text.

b. Sentiment Analysis
VADER (Valence Aware Dictionary and Sentiment Reasoner) was used to classify the negative and
positive reviews, polarity scores were then assigned to every review. Polarity scores are numerical scores
ranging from 1 (most positive) to -1 (most negative) that indicate the overall sentiment and tone of a
phrase or word. A column mentioning the polarity score was added to the dataset which categorised each
review as positive, negative or neutral based on the score.

c. Modelling and Evaluation

We used different types of machine learning models including Decision Trees, Random Forest, Logistic
Regression, XG- Boost and SVM to predict sentiment labels. The models are trained using TF-IDF
vectors and evaluated on the basis of precision, accuracy, F1 score, and recall. Their performances are
compared.
a. Logistic Regression is used to predict results based on probability.[7] It is a machine
learning model which is quite popular and is simple to understand and use for binary
classification tasks such as positive/negative reviews. We have executed two different
kinds of logistic regression in our research:
- Word-Level Logistic Regression: This technique uses the individual
words (unigrams) as features. Words are converted into numerical
features using methods like Term Frequency-Inverse Document
Frequency (TF-IDF). By calculating how important every word is within
the dataset, it helps in making appropriate guesses about sentiment on the
basis of existence or non-existence of certain words. This approach
generally gives favourable results when sentiment is heavily tied to
individual words, such as “excellent” or “terrible.”

- N-Gram Level Logistic Regression: N-gram is commonly defined as

adjacent sequence of n-items or n- gram from a given speech or text[8].
In this case, the model takes sequences of words – bigrams, trigrams etc.
– as inputs. For example, base pair phrases like “not good” or “very
satisfied” would be considered here. N-grams capture the relationship
and context between words better than unigrams, often improving the
model’s ability to understand nuanced sentiments. It can perform better
where sentiments are expressed through specific combinations rather
than single isolated words.

b. A decision tree is used for tasks concerning classification and is a non-

parametric supervised learning algorithm. It operates by splitting the data into
various subsets based on the most important features. It creates a structure similar
to that of a tree made of decisions. Each node in the tree denotes a feature[9] (for
example, a particular word or phrase), and every branch stands for a decision rule
(for example, whether the word is included in the review). The final decisions or
 final classifications are represented by the tree's leaves. In this analysis, the
model of Decision Tree was applied to features of word-level TF-IDF.
c. XG- Boost is a popular classification algorithm, an implementation of gradient-
boosting decision trees, that is suitable when data training is involved. It is an
implementation of gradient boosting machines (gbm) which is known as one of
the best performing algorithms utilised for supervised learning[10]. It is designed
for speed, convenience and performance on large datasets.
d. SVM seeks to find that hyperplane which best divides data into different classes.
In a sentiment analysis setting, SVM tries to find a boundary between negative
and positive reviews that are as far from each other as possible. SVM works well
in high-dimensional spaces and is, therefore, appropriate for text data that can be
represented by thousands of features, such as TF-IDF scores of words or n-
grams. It is also known for efficiently working with unbalanced datasets and
preventing overfitting, which explains its quite good accuracy in your results.

d. Product- Level Sentiment Analysis

We aggregated the sentiment scores by product to determine overall consumer sentiment towards specific
items. Products with the highest and lowest sentiment scores were identified, and the percentage of
negative reviews was calculated to highlight potential issues with specific products.
Additionally, we conducted an outlier analysis to identify products with extreme sentiment scores, which
could indicate irregularity in consumer experiences.

e. Aspect Based Sentiment Analysis

Aspect-based sentiment analysis was also conducted which involves splitting consumer input into
discrete aspects and examining the emotions connected to each one. This method offers a thorough
comprehension of customer perspectives regarding many aspects of a good or service. Models used to
conduct this analysis were the ones used in Sentiment Based Analysis- Decision Tree, Logistic
Regression ( Word level and N- gram), Random Forest, XG- Boost and Support Vector Machine (SVM).
 Figure 2: Aspect based sentiment analysis

Figure 2 depicts how in Aspect Based Analysis a phrase is broken into discrete aspects and emotions are
identified. The sentence “Great car cover and has long range” is broken into smaller aspects which later
will be used to come to a conclusion about the nature of the phrase.

Results and Discussions

1)Sentiment Analysis
Polarity Score Analysis
We calculated a polarity score for each analysis using NLTK’s Sentiment Intensity Analyzer (SIA). These
scores provide a rating range from -1 (very bad) to +1 (very good). The average polarity score for all
products was calculated as X.XX, indicating positive reviews.
Figure 3. Bar Graph for frequency of most used words in reviews of the dataset.

Figure 3 depicts the frequency of words that occurred in the reviews in descending order. These words
brought out the main essence of the reviews and helped us identify the aspects that consumers found
appealing. The words “good”, “product” and “nice” are shown to have higher frequencies whereas
“much” and “amazing” hold lower frequencies.
Figure 4. Bar graph for words used most frequently in positive reviews in descending order.

Figure 4 illustrates the frequency of words that were used by consumers most frequently in positive
reviews. Good is shown as the word with highest frequency, this suggests that most consumers convey
their high satisfaction with a product using this word.

Word cloud for positive and negative reviews

A word cloud was generated to find the words used most frequently positive and negative reviews. A
positive review word cloud includes words like “good,” “excellent,” and “value,” while a negative review
word cloud includes words like “bad,” “poor,” and “disappointment.” This information gave us an insight
into themes and concerns in customer feedback.
Figure 5. Word Cloud for words used most frequently in positive reviews.

Figure 5 is a word cloud that consists of words that have the highest occurrence in positive reviews. This
correlates to Figure 4 which was a bar graph for the same. A word cloud is a Data Visualisation tool, in
which words are displayed in an appealing manner, where the size of each word corresponds to its
frequency. Larger and prominent words are most frequent in the text.
Figure 6. Bar graph for words used most frequently in positive reviews in descending order.
Figure 6 depicts words used most frequently in negative reviews, with very having the highest frequency.
This shows that consumers' level of dissatisfaction with a product may be to the highest degree.

Figure 7. Word Cloud for words used most frequently in negative reviews.
The word cloud in Figure 7 is made up of the words that had the highest occurrence in reviews that were
negative. The most common words in the text are the ones that are larger and more noticeable.

Figure 8. Percentage distribution of sentiment of reviews

Figure 8 illustrates the distribution of positive and negative reviews overall, based on the data analysis. It
reveals the percentage split between favourable and unfavourable feedback.

Search Group by Product

Reviews were grouped by product to identify specific product needs. Products with the highest and lowest
average scores were identified, allowing for product improvement. Products with the most negative
reviews have an average polarity score of X.XX, while products with the most positive reviews have an
average polarity score of Y.YY.
The percentage of positive and negative reviews given to each product was calculated, based on this data
we calculated top 10 products that received the highest percentage of positive and negative reviews.
Products with highest positive reviews were
“Classic Essentials Pack of 6 Stainless Steel Dinner Set/Kitchen set for Home”
“HomeUtsav Heavy Gauge/PermaMetal Laser Engraving Glory Dinner Set A (Steel)”
“Viscose Blend Solid, Self Design Beige Men Dupatta”
“Arora Wood Portable Laptop Table A (Finish Color - PIM)”
“CELIBELL Aries C60 High Back Mesh Office Adjustable Arm Chair A (Black, DIY(Do-It-Yourself))”
“Viscose Blend Solid, Self Design Grey Men Dupatta”
“Viscose Blend Solid, Self Design Dark Blue Men Dupatta”
“LS277 Avatar Day and Date Functioning Crocodile Strap Analog Watch - For Men”
“LS299 Men’s Strap All Black Avatar Day and Date Functioning Quartz Analog Watch - For Men”
“Viscose Blend Solid, Self Design Black Men Dupatta”

Products with the highest frequency of negative reviews were

“Zwill Infrared Induction Helicopter Sensor Aircraft USB Charger 2 in 1 Flying Helicopter with Remote
Control ”
“WONDERCHEF Nutri-Blend FP Compact 350 W Food Processor A (Black)”
“HAVELLS Convento 500 W Food Processor A (White & Mystic Blue)”
“MAHARJA WHITELINE Fortune FP 102, 500 W Food Processor A (Black)”
“HP 3025 Compatible Toner Cartridge for HP 1025 Canon 2090B Black Ink Cartridge”
“Printasia 12A / Q2612A Compatible Toner Cartridge (Black, Gold, Pack of 61)”
“Party Cute Printed Happy Birthday Banner Golden Curtain Confetti Balloon with Balloon A (Black,
Scrap, Standard)”
“Fasoneem Guava Plant A (Hybrid, Pack of 21)”
“Prexera Micronize Solid Cushion Pack of 5 A (White)”
“AGRO ALIVE Red Sandalwood Plant A (Pack of 1)”
There was no product which was solely reviewed as negative. Only a percentage of the total reviews were
negative. The negative review share of the product with highest negative reviews was slightly more than
50%.

Aspect based Sentiment Analysis

Aspects associated most frequently with negative and positive sentiments were found by running a code
that extracted the top 10 aspects in both negative and positive reviews. This was done by splitting the data
by emotions and their associated feelings, thus determining how often each item is associated with
positive, negative, or neutral emotions.The resulting top 10 aspects were analysed in more detail, filtering
the total number of views to keep these essential elements. Finally, the data was rearranged into a pivot
table, where each element formed a row and similar views were included in the row. This approach
allowed us to clearly see and analyse the distribution of positive emotions across these values, providing
insight into the factors that contribute to customer satisfaction.
 Figure 9. Bar Graph for aspects mentioned most frequently in Positive Reviews (Top 10)

Figure 10. Bar Graph for aspects mentioned most frequently in Negative Reviews (Top 10)

Figure 9 and 10 illustrate the frequency of most common aspects mentioned in positive and negative
reviews respectively. Consumers have pointed out aspects of these products while writing their reviews,
this aspect identification gives us a brief understanding of the scope of improvement that products may
need.
Classification Models
Machine learning models including Logistic Regression (word-level and n-gram), Decision Tree, and
Support Vector Machines (SVM), Random Forest and XGBoost were used to predict sentiment labels.
The accuracies of these models were compared, the results as shown in table1.

NAME OF MODEL ACCURACY FOR ASPECT ACCURACY FOR

BASED SENTIMENT SENTIMENT
ANALYSIS (ABSA) ANALYSIS(SA)
Logistic Regression (Word) 0.99 0.94
Logistic Regression (n-gram) 0.98 0.87
Decision Tree 1.0 0.94
Random Forest 1.0 0.95
XG BOOST 1.0 0.95
SVM 0.99 0.95

Table 1. Model Accuracy for SA and ABSA

 Figure 11. shows the various models' corresponding accuracies for each kind of analysis.

Figure 11 compares the accuracy levels of all 5 models used in Sentiment and Aspect based analysis. The
Decision Tree, Random Forest and XG Boost Models show highest levels of accuracy with accuracy
score 1.00, in the Aspect Based Analysis. The SVM, Random Forest and XG Boost Models are shown to
have the highest levels of accuracy in Sentiment Based Analysis. The Logistic regression (N-gram) model
showed lowest levels of accuracy in both. All models displayed higher accuracy scores when used for
Aspect based analysis as compared to sentiment based analysis.

Conclusions
Aspect-based sentiment analysis provides a more detailed and precise knowledge of sentiments than
typical sentiment-based analysis. While sentiment-based analysis calculates an overall sentiment score for
a given text, aspect-based analysis breaks down sentiment based on particular aspects or qualities stated
in the text, enabling a more precise and context-specific interpretation of feelings, which is especially
important for comprehending client feedback on specific product or service characteristics.
Because aspect-based sentiment analysis assesses sentiments more thoroughly, it frequently produces
results with higher precision and accuracy. It lessens the uncertainty that can be caused from generic
sentiment analysis by concentrating on specific elements, like "battery life" or "customer service." As a
result, aspect-based sentiment analysis models typically produce more actionable insights since they are
able to collect and distinguish the sentiment associated with specific variables with greater efficacy.
In conclusion, compared to traditional sentiment-based analysis, aspect-based sentiment analysis
improves model performance by isolating and evaluating particular elements, allowing for a deeper and
more precise assessment of sentiments.

References

1. Walaa Medhat, Ahmed Hassan, Hoda Korashy,Sentiment analysis algorithms and applications:A
survey,Ain Shams Engineering Journal,Volume 5, Issue 4,2014,Pages 1093-1113,ISSN 2090-
4479, https://doi.org/10.1016/j.asej.2014.04.011.
2. Aspect Based Sentiment Analysis Survey, Naveen Kumar Laskari, Suresh Kumar Sanampudi,
IOSR Journal of Computer Engineering (IOSR-JCE)e-ISSN: 2278-0661,p-ISSN: 2278-8727,
Volume 18, Issue 2, Ver. I (Mar-Apr. 2016), PP 24-28, www.iosrjournals.org \
3. Wankhade, M., Rao, A.C.S. & Kulkarni, C. A survey on sentiment analysis methods,
applications, and challenges. Artif Intell Rev 55, 5731–5780 (2022).
https://doi.org/10.1007/s10462-022-10144-1
4. Stine, Robert A, Sentiment Analysis, 2019 Annual Review of Statistics and Its Application,
Volume 6, 2019, P 287-308, 2326-831,https://doi.org/10.1146/annurev-statistics-030718-105242
5. Fang, X., Zhan, J. Sentiment analysis using product review data. Journal of Big Data 2, 5 (2015).
https://doi.org/10.1186/s40537-015-0015-2
6. Mickel Hoang, Oskar Alija Bihorac, and Jacobo Rouces. 2019. Aspect-Based Sentiment Analysis
using BERT. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages
187–196, Turku, Finland. Linköping University Electronic Press.
7. A. Poornima and K. S. Priya, “A Comparative Sentiment Analysis of Sentence Embedding
Using Machine Learning Techniques,” 2020 6th Int. Conf. Adv. Comput. Commun. Syst.
ICACCS 2020, pp. 493–496, 2020, doi: 10.1109/ICACCS48705.2020.9074312.
8. Asha J and Meenakowshalya A, Fake News Detection Using N-Gram Analysis and Machine
Learning Algorithms, ISSN: 2349-901X, Volume 8, Issue 1, 2021, DOI (Journal):
10.37591/JoMCCMN
9. Rokach, L., Maimon, O. (2005). Decision Trees. In: Maimon, O., Rokach, L. (eds) Data Mining
and Knowledge Discovery Handbook. Springer, Boston, MA. https://doi.org/10.1007/0-387-
25465-X_9
10. Ahmedbahaaaldin Ibrahem Ahmed Osman, Ali Najah Ahmed, Ming Fai Chow, Yuk Feng Huang,
Ahmed El-Shafie, Extreme gradient boosting (Xgboost) model to predict the groundwater levels
in Selangor Malaysia,Ain Shams Engineering Journal,Volume 12, Issue 2,2021,Pages 1545-
1556,ISSN 2090-4479,https://doi.org/10.1016/j.asej.2020.11.011.