Introduction:

Chatbots are computer programs that can interact with humans using natural language.
They have various applications in domains such as customer service, education,
entertainment, and health care. ChatGPT is a chatbot that uses a large-scale language
model called GPT-3 to generate natural and engaging responses. ChatGPT claims to be
able to converse on any topic and provide relevant information, answers, and
suggestions. However, how do people perceive and evaluate ChatGPT? What are their
opinions and sentiments about this chatbot? These are the main questions that motivate
this research paper.

In this paper, we aim to investigate the sentiment and opinions of both Assosa
University students and social media users about ChatGPT. We chose these two groups
of participants because they represent different segments of the potential users of
ChatGPT. Assosa University students are mostly young, educated, and familiar with
technology, while social media users are more diverse and heterogeneous in terms of
age, education, and interests. We hypothesize that these two groups may have different
opinions and sentiments about ChatGPT, depending on their expectations, needs, and
preferences.

To conduct our research, we followed these steps: First, we gathered data from
questionnaires distributed to Assosa University students, asking for their opinions on
ChatGPT. Second, we collected comments from social media platforms (such as Twitter,
Facebook, and Reddit) that express viewpoints about ChatGPT. Third, we employed
three annotators to classify each text as either positive or negative, based on the
polarity of the opinion or sentiment expressed. Fourth, we utilized two feature extraction
techniques, Bag-of-Words (BOW) and Term Frequency-Inverse Document Frequency
(TFIDF), to convert the text into numerical representations, enabling subsequent analysis
and classification. Fifth, we employed six machine learning algorithms, Naive Bayes,
Support Vector Machine (SVM), k-Nearest Neighbors (KNN), Decision Tree, Stochastic
Gradient Descent (SGD) Classifier, and Logistic Regression, to perform sentiment
analysis on the data. Sixth, we evaluated the performance of the different machine
learning algorithms, considering metrics such as accuracy, precision, recall, and F1 score.
Finally, we discussed the results and implications of our research, and suggested
directions for future work.

The main contributions of this paper are: (1) providing a comprehensive and
comparative analysis of the sentiment and opinions of both Assosa University students
and social media users about ChatGPT; (2) applying and evaluating various feature
extraction techniques and machine learning algorithms for sentiment analysis; and (3)
providing valuable insights into the overall sentiment and opinions of the potential
users of ChatGPT, which can inform the design and improvement of chatbots.

Motivation

Chatbots are becoming ubiquitous in various domains, such as customer service,

education, entertainment, and health care. However, not all chatbots are equally
effective and satisfying for users. ChatGPT is a chatbot that uses a large-scale language
model called GPT-3 to generate natural and engaging responses. ChatGPT claims to be
able to converse on any topic and provide relevant information, answers, and
suggestions. But how do users perceive and evaluate ChatGPT? What are their opinions
and sentiments about this chatbot? These are the main questions that motivate this
research.
Previous studies have explored the factors that influence user satisfaction and trust in
chatbots, such as personality, empathy, humor, and feedback 234. However, few studies
have specifically examined the user perceptions and sentiments of ChatGPT, which is a
novel and advanced chatbot that uses a state-of-the-art language model. Moreover,
most studies have focused on a single user group, such as students, customers, or
patients, without considering the diversity and heterogeneity of potential chatbot users.
Therefore, there is a need to conduct a comprehensive and comparative analysis of the
sentiment and opinions of different user groups about ChatGPT.

Motivation

The motivation behind this study lies in the need to gain insights into user perceptions and sentiments
concerning ChatGPT. Understanding how users evaluate the chatbot's performance, usefulness, and
overall experience will contribute to its improvement and the development of more effective and user-
centric chatbot systems. By examining the sentiments and opinions of Assosa University students and
social media users, we can identify similarities and differences between these two groups, enabling a
comprehensive understanding of ChatGPT's reception across diverse user segments.

Statement of the Problem

ChatGPT is a conversational agent that uses a large-scale language model called GPT-3
to generate natural and engaging responses. ChatGPT claims to be able to converse on
any topic and provide relevant information, answers, and suggestions. However, there is
a lack of comprehensive knowledge regarding the sentiment and opinions of users
towards ChatGPT. This is a problem because user perceptions and sentiments are crucial
factors that influence the effectiveness and satisfaction of chatbot systems. Moreover,
user perceptions and sentiments may vary depending on the characteristics and
preferences of different user groups. Therefore, it is important to understand how
different user groups perceive and evaluate ChatGPT, and what are their expectations,
needs, and preferences related to this chatbot.
Previous studies have explored the factors that influence user satisfaction and trust in
chatbots, such as personality, empathy, humor, and feedback 123. However, few studies
have specifically examined the user perceptions and sentiments of ChatGPT, which is a
novel and advanced chatbot that uses a state-of-the-art language model. Moreover,
most studies have focused on a single user group, such as students, customers, or
patients, without considering the diversity and heterogeneity of potential chatbot users.
Therefore, there is a need to conduct a comprehensive and comparative analysis of the
sentiment and opinions of different user groups about ChatGPT.

The main problem addressed in this research is the lack of comprehensive knowledge
regarding the sentiment and opinions of users towards ChatGPT. By exploring the
sentiments and opinions of Assosa University students and social media users, we seek
to uncover potential challenges, areas of improvement, and user preferences related to
ChatGPT. This will contribute to the literature on chatbot evaluation and user
satisfaction, and provide practical guidance for the design and improvement of chatbot
systems.

Or

Statement of the Problem

The main problem addressed in this research is the lack of comprehensive knowledge regarding
the sentiment and opinions of users towards ChatGPT. While the chatbot claims to engage users
effectively, it is essential to gauge user perceptions and sentiments to validate and improve its
functionality. By exploring the sentiments and opinions of Assosa University students and social
media users, we seek to uncover potential challenges, areas of improvement, and user
preferences related to ChatGPT.

Research Question

The primary research question guiding this study is:

 How do Assosa University students and social media users in Ethiopia perceive and
evaluate ChatGPT in Amharic, and what are their sentiments and opinions towards the
chatbot in 2024?

The sub-questions that support this research question are:

 What are the main factors that influence the perceptions and evaluations of ChatGPT
among Assosa University students and social media users in Ethiopia?
 What are the methods and tools used to collect and analyze the data on the sentiments
and opinions of ChatGPT among Assosa University students and social media users in
Ethiopia?
 What are the outcomes and implications of the sentiments and opinions of ChatGPT
among Assosa University students and social media users in Ethiopia?
 How do the sentiments and opinions of ChatGPT differ between Assosa University
students and social media users in Ethiopia?

Objectives 2.1 General Objective The general objective of this study is to investigate and
compare the sentiment and opinions of Assosa University students and social media
users in Ethiopia towards ChatGPT in Amharic, aiming to gain insights that can inform
the design and enhancement of chatbot systems.

2.2 Specific Objectives To achieve the general objective, we will pursue the following
specific objectives within a one-year period:

 Collect and analyze data through questionnaires distributed to 100 Assosa University
students to assess their opinions on ChatGPT.
 Collect and analyze comments from social media platforms (e.g., Twitter, Facebook,
Reddit) expressing viewpoints about ChatGPT, using web scraping and natural language
processing techniques.
 Employ three annotators to classify each text as positive or negative based on the
expressed sentiment or opinion, and calculate the interannotator agreement score.
 Utilize feature extraction techniques (e.g., Bag-of-Words, Term Frequency-Inverse
Document Frequency) to convert text into numerical representations for subsequent
analysis and classification.
 Apply and compare six machine learning algorithms (Naive Bayes, Support Vector
Machine, k-Nearest Neighbors, Decision Tree, Stochastic Gradient Descent Classifier,
and Logistic Regression) to perform sentiment analysis on the gathered data.
 Evaluate the performance of different machine learning algorithms using metrics such as
accuracy, precision, recall, and F1 score, and select the best algorithm for the task.
 Discuss the results and implications of the research, providing insights into user
sentiment and opinions towards ChatGPT, and identifying the factors that influence
them.
 Suggest directions for future work to further enhance chatbot systems based on the
findings, and propose recommendations for improving user satisfaction and trust in
ChatGPT.

Objectives

2.1 General Objective

The general objective of this study is to investigate the sentiment and opinions of Assosa University
students and social media users towards ChatGPT, aiming to gain insights that can inform the design and
enhancement of chatbot systems.

2.2 Specific Objectives

To achieve the general objective, we will pursue the following specific objectives:

Gather data through questionnaires distributed to Assosa University students to assess their opinions on
ChatGPT.

Collect comments from social media platforms (e.g., Twitter, Facebook, Reddit) expressing viewpoints
about ChatGPT.

Employ annotators to classify each text as positive or negative based on the expressed sentiment or
opinion.

Utilize feature extraction techniques (e.g., Bag-of-Words, Term Frequency-Inverse Document Frequency)
to convert text into numerical representations for subsequent analysis and classification.
Apply six machine learning algorithms (Naive Bayes, Support Vector Machine, k-Nearest Neighbors,
Decision Tree, Stochastic Gradient Descent Classifier, and Logistic Regression) to perform sentiment
analysis on the gathered data.

Evaluate the performance of different machine learning algorithms using metrics such as accuracy,
precision, recall, and F1 score.

Discuss the results and implications of the research, providing insights into user sentiment and opinions
towards ChatGPT.

Suggest directions for future work to further enhance chatbot systems based on the findings.

Significance of the Study

This study holds significant implications for both academia and industry. By investigating the
sentiment and opinions of Assosa University students and social media users towards ChatGPT,
we contribute to the existing body of knowledge concerning user perceptions of chatbots. The
findings will offer valuable insights into user preferences, expectations, and areas of
improvement for chatbot systems. Additionally, the study can inform the design and
enhancement of ChatGPT specifically, as well as guide the development of future chatbot
systems. Ultimately, this research aims to foster user-centric design approaches for chatbots,
leading to more engaging and effective human-computer interactions.

Significance of the Study

This study has important implications for both academia and industry, as it addresses a
gap in the literature on user perceptions of chatbots. Chatbots are increasingly used in
various domains, such as customer service, education, entertainment, and health care,
and they have the potential to improve the quality and efficiency of these
services12. However, the success and satisfaction of chatbot systems depend largely on
how users perceive and evaluate them, which may vary depending on the characteristics
and preferences of different user groups 34. Therefore, it is essential to understand the
sentiment and opinions of users towards chatbots, and how they can be influenced by
factors such as personality, empathy, humor, and feedback.
By investigating the sentiment and opinions of Assosa University students and social
media users in Ethiopia towards ChatGPT, a novel and advanced chatbot that uses a
large-scale language model, we contribute to the existing body of knowledge
concerning user perceptions of chatbots. The findings will offer specific and actionable
insights into user preferences, expectations, and areas of improvement for chatbot
systems. Additionally, the study can inform the design and enhancement of ChatGPT
specifically, as well as guide the development of future chatbot systems that can adapt
to the needs and preferences of different user groups. Ultimately, this research aims to
foster user-centric design approaches for chatbots, leading to more engaging and
effective human-computer interactions.

However, this study also has some limitations or challenges that need to be considered.
First, the data collection methods, such as questionnaires and social media comments,
may introduce some biases or errors in the data, such as self-reporting bias, social
desirability bias, or sampling bias. Second, the data analysis methods, such as feature
extraction techniques and machine learning algorithms, may have some limitations or
assumptions that affect the reliability and validity of the results, such as data sparsity,
noise, or overfitting. Third, the cultural or linguistic differences among the participants
may affect their perceptions and evaluations of ChatGPT, as well as the interpretation
and generalization of the findings. Therefore, these limitations or challenges should be
acknowledged and addressed in the research process, and the findings should be
interpreted with caution and context.

Literature Review

The literature review will delve into existing research on chatbots, sentiment analysis techniques, and
user perceptions towards conversational agents. It will explore studies related to the evaluation and
sentiment analysis of chatbot systems, highlighting their methodologies, findings, and limitations.
Additionally, the review will cover relevant research regarding user preferences, expectations, and
satisfaction with chatbots, aiming to provide a comprehensive understanding of the current state of
knowledge in this domain.

Literature Review The literature review will delve into existing research on chatbots,
sentiment analysis techniques, and user perceptions towards conversational agents. It
will explore studies related to the evaluation and sentiment analysis of chatbot systems,
highlighting their methodologies, findings, and limitations. Additionally, the review will
cover relevant research regarding user preferences, expectations, and satisfaction with
chatbots, aiming to provide a comprehensive understanding of the current state of
knowledge in this domain.
Chatbots Chatbots are computer programs that can interact with humans using natural
language. They have various applications in domains such as customer service,
education, entertainment, and health care. Chatbots can be classified into two types:
rule-based and data-driven. Rule-based chatbots follow predefined rules and scripts to
generate responses, while data-driven chatbots use machine learning and natural
language processing techniques to learn from data and generate responses 1.
ChatGPT is a data-driven chatbot that uses a large-scale language model called GPT-3
to generate natural and engaging responses. GPT-3 is a deep neural network that can
produce coherent and diverse texts on any topic, given some input or context. ChatGPT
claims to be able to converse on any topic and provide relevant information, answers,
and suggestions2.
Sentiment Analysis Techniques Sentiment analysis is the task of identifying and
extracting the subjective information, such as opinions, emotions, attitudes, and
sentiments, from text. Sentiment analysis can be applied to various types of texts, such
as reviews, tweets, comments, blogs, etc. Sentiment analysis can be performed at
different levels of granularity, such as document, sentence, or aspect level 3.
Sentiment analysis involves two main steps: feature extraction and classification. Feature
extraction is the process of transforming text into numerical representations that can be
used by machine learning algorithms. Classification is the process of assigning a
sentiment label, such as positive, negative, or neutral, to a text based on its
features. There are various feature extraction techniques and machine learning
algorithms that can be used for sentiment analysis, such as Bag-of-Words, Term
Frequency-Inverse Document Frequency, Naive Bayes, Support Vector Machine, k-
Nearest Neighbors, Decision Tree, Stochastic Gradient Descent Classifier, and Logistic
Regression3 .

User Perceptions of Chatbots User perceptions of chatbots are the subjective

evaluations and judgments that users make about the chatbot’s performance,
usefulness, and overall experience. User perceptions of chatbots are influenced by
various factors, such as the chatbot’s personality, empathy, humor, feedback, and
domain knowledge. User perceptions of chatbots are also affected by the user’s
characteristics, such as age, gender, education, and interests .

User perceptions of chatbots can be measured by various methods, such as

questionnaires, interviews, observations, or online comments. User perceptions of
chatbots can be analyzed by various techniques, such as sentiment analysis, content
analysis, or statistical analysis. User perceptions of chatbots can have various outcomes
and implications, such as user satisfaction, trust, loyalty, engagement, or behavior
change .
The literature review will be organized into three main sections: chatbots, sentiment
analysis techniques, and user perceptions of chatbots. Each section will provide a
detailed overview of the relevant studies, theories, models, and frameworks, and
critically analyze and synthesize them in relation to the research question and objectives.
The literature review will also identify the gaps or inconsistencies in the literature, and
discuss how the research will address them or contribute to the existing knowledge.

Methodology

The methodology section of this research paper describes the step-by-step process
followed to conduct the study, encompassing data collection, annotation, feature
extraction, sentiment analysis, and evaluation. The following subsections provide a
detailed explanation of each method employed, along with their justification, limitations,
and references.
Data Collection: The data collection process involves gathering relevant information
from two sources: Assosa University students and social media platforms. These sources
were chosen because they represent different segments of the potential users of
ChatGPT, and they provide diverse and rich data on user opinions and sentiments
towards the chatbot. For Assosa University students, questionnaires will be distributed
to obtain their opinions and sentiments towards ChatGPT. The questionnaires will be
designed to capture user experiences, evaluations, and preferences related to the
chatbot, using both closed-ended and open-ended questions. The closed-ended
questions will use Likert scales to measure user satisfaction, trust, and engagement with
the chatbot, while the open-ended questions will elicit user feedback and suggestions
for improvement. The questionnaires will be distributed online, using Google Forms, to
100 students who have used ChatGPT at least once. The students will be recruited
through convenience sampling, using social media and email invitations. The students
will have two weeks to complete and submit the questionnaires. Additionally, comments
and posts from social media platforms such as Twitter, Facebook, and Reddit will be
collected to include a broader range of public opinions on ChatGPT. These platforms
were selected because they are popular and widely used by people who are interested
in chatbots and artificial intelligence. The comments and posts will be retrieved using
web scraping and natural language processing techniques, such as BeautifulSoup 1 and
NLTK2. The search keywords will include “ChatGPT”, “GPT-3”, and “chatbot”. The search
period will span from January 2024 to June 2024, to capture the most recent and
relevant opinions on ChatGPT. A total of 1000 comments and posts will be collected
from each platform, resulting in a dataset of 3000 social media texts.
The main limitation of the data collection process is the possible sampling bias, as the
data sources may not represent the entire population of ChatGPT users or potential
users. Moreover, the data collection methods may introduce some errors or noise in the
data, such as self-reporting bias, social desirability bias, or spam comments. To address
these limitations, the data will be carefully checked and cleaned before analysis, and the
findings will be interpreted with caution and context.
Annotation: The annotation process involves labeling the collected data with sentiment
labels, such as positive or negative, based on the expressed opinions or emotions.
Human annotators will be engaged to manually annotate the data, ensuring accurate
sentiment classification. Each questionnaire response or social media comment will be
examined, and a sentiment label will be assigned accordingly. The sentiment labels will
follow the polarity scheme, where positive means the text expresses a favorable or
supportive opinion or emotion towards ChatGPT, and negative means the text expresses
an unfavorable or critical opinion or emotion towards ChatGPT. Neutral or mixed texts
will be excluded from the analysis, as they do not clearly indicate a sentiment or
opinion. Three annotators will be hired to annotate the data, following the guidelines
and criteria provided by Liu3. The annotators will be trained and tested on a sample of
100 texts before annotating the entire dataset. The interannotator agreement score will
be calculated using Cohen’s kappa coefficient 4, to measure the consistency and
reliability of the annotation. The final sentiment label for each text will be determined by
the majority vote of the three annotators. In case of a tie, the text will be reviewed by a
fourth annotator to break the tie.

The main challenge of the annotation process is the subjectivity and ambiguity of
sentiment analysis, as different annotators may have different interpretations or
judgments of the texts. Moreover, the annotation process may be time-consuming and
labor-intensive, as it requires manual inspection and labeling of a large amount of data.
To overcome these challenges, the annotation process will follow a rigorous and
standardized procedure, using clear and objective guidelines and criteria, and
employing multiple annotators to ensure the quality and validity of the annotation.
Feature Extraction: Feature extraction techniques will be employed to convert the
collected textual data into numerical representations suitable for sentiment analysis.
Two commonly used techniques are: a) Bag-of-Words (BoW): This technique represents
each text sample as a collection of unique words, disregarding grammar and word
order. The frequency of occurrence of each word is used as a feature. BoW is a simple
and efficient technique that can capture the most salient words in the texts, but it also
has some drawbacks, such as ignoring the semantic and syntactic relationships between
words, and producing high-dimensional and sparse feature vectors 5. b) Term Frequency-
Inverse Document Frequency (TF-IDF): TF-IDF measures the relevance of a word in a
document by considering its frequency in the document and inversely proportional to
its frequency in the entire dataset. TF-IDF assigns higher weights to words that are more
important and distinctive in the document, and lower weights to words that are
common and irrelevant. TF-IDF is an improvement over BoW, as it can reduce the
impact of noise and redundancy in the texts, but it still suffers from the same limitations
of ignoring the semantic and syntactic relationships between words 5.

To address the limitations of BoW and TF-IDF, the feature extraction process will also
include some preprocessing and dimensionality reduction techniques, such as
tokenization, stop word removal, stemming, and principal component analysis (PCA).
These techniques will help to clean and normalize the texts, and reduce the size and
complexity of the feature vectors, enhancing the performance and efficiency of the
sentiment analysis.

Sentiment Analysis: Sentiment analysis, also known as opinion mining, involves

classifying the collected data into positive or negative sentiments. Machine learning
algorithms will be utilized to automate this process. The selected algorithms for
sentiment analysis are: a) Naive Bayes: A probabilistic classifier based on Bayes’ theorem
that assumes independence between features. Naive Bayes is a simple and fast
algorithm that can handle large and sparse data, but it also has some disadvantages,
such as making unrealistic assumptions about the data, and being sensitive to irrelevant
or correlated features. b) Support Vector Machine (SVM): A supervised learning
algorithm that finds an optimal hyperplane to separate different classes. SVM is a
powerful and robust algorithm that can handle non-linear and high-dimensional data,
but it also has some drawbacks, such as being computationally expensive, requiring
parameter tuning, and being prone to overfitting. c) k-Nearest Neighbors (k-NN): A
non-parametric algorithm that classifies samples based on the majority vote of their k-
nearest neighbors. k-NN is a simple and intuitive algorithm that can adapt to different
data distributions, but it also has some drawbacks, such as being sensitive to noise and
outliers, requiring distance calculation, and being affected by the choice of k. d) Decision
Tree: A tree-like model that partitions data into subsets based on feature values,
enabling classification. Decision Tree is an easy and interpretable algorithm that can
handle both numerical and categorical data, but it also has some drawbacks, such as
being unstable, complex, and susceptible to overfitting. e) Stochastic Gradient Descent
Classifier: An optimization algorithm that iteratively updates parameters to minimize the
loss function. Stochastic Gradient Descent Classifier is an efficient and scalable algorithm
that can handle large and sparse data, but it also has some drawbacks, such as being
sensitive to learning rate, requiring feature scaling, and being affected by random
initialization. f) Logistic Regression: A regression model that predicts the probability of a
binary outcome using logistic function. Logistic Regression is a simple and widely used
algorithm that can provide interpretable results, but it also has some drawbacks, such as
being sensitive to outliers, requiring feature scaling, and assuming linearity between
features and outcome.
To compare and evaluate the performance of these algorithms, the data will be
randomly split into training and testing sets, using a 80/20 ratio. The training set will be
used to train the algorithms, while the testing set will be used to test the algorithms. The
algorithms will be implemented using Python and scikit-learn, a popular machine
learning library. The algorithms will be tuned using grid search and cross-validation
techniques, to find the optimal parameters and avoid overfitting.

Evaluation: The performance of the sentiment analysis models will be evaluated using
appropriate metrics to assess their effectiveness. Common evaluation metrics include
accuracy, precision, recall, and F1 score. These metrics provide insights into the model’s
ability to correctly classify sentiments and measure the balance between precision
(correctly identified positive/negative sentiments) and recall (ability to identify all
positive/negative sentiments). The formulas for

Or

Methodology

The methodology section of this research paper describes the step-by-step process followed to conduct
the study, encompassing data collection, annotation, feature extraction, sentiment analysis, and
evaluation. The following subsections provide a detailed explanation of each method employed:

Data Collection:

The data collection process involves gathering relevant information from two sources: Assosa University
students and social media platforms. For Assosa University students, questionnaires will be distributed
to obtain their opinions and sentiments towards ChatGPT. The questionnaires will be designed to
capture user experiences, evaluations, and preferences related to the chatbot. Additionally, comments
and posts from social media platforms such as Twitter, Facebook, and Reddit will be collected to include
a broader range of public opinions on ChatGPT.

Annotation:
The annotation process involves labeling the collected data with sentiment labels, such as positive or
negative, based on the expressed opinions or emotions. Human annotators will be engaged to manually
annotate the data, ensuring accurate sentiment classification. Each questionnaire response or social
media comment will be examined, and a sentiment label will be assigned accordingly.

Feature Extraction:

Feature extraction techniques will be employed to convert the collected textual data into numerical
representations suitable for sentiment analysis. Two commonly used techniques are:

a) Bag-of-Words (BoW): This technique represents each text sample as a collection of unique words,
disregarding grammar and word order. The frequency of occurrence of each word is used as a feature.

b) Term Frequency-Inverse Document Frequency (TF-IDF): TF-IDF measures the relevance of a word in a
document by considering its frequency in the document and inversely proportional to its frequency in
the entire dataset.

Sentiment Analysis:

Sentiment analysis, also known as opinion mining, involves classifying the collected data into positive or
negative sentiments. Machine learning algorithms will be utilized to automate this process. The selected
algorithms for sentiment analysis may include:

a) Naive Bayes: A probabilistic classifier based on Bayes' theorem that assumes independence between
features.

b) Support Vector Machine (SVM): A supervised learning algorithm that finds an optimal hyperplane to
separate different classes.

c) k-Nearest Neighbors (k-NN): A non-parametric algorithm that classifies samples based on the majority
vote of their k-nearest neighbors.

d) Decision Tree: A tree-like model that partitions data into subsets based on feature values, enabling
classification.

e) Stochastic Gradient Descent Classifier: An optimization algorithm that iteratively updates parameters
to minimize the loss function.

f) Logistic Regression: A regression model that predicts the probability of a binary outcome using logistic
function.
Evaluation:

The performance of the sentiment analysis models will be evaluated using appropriate metrics to assess
their effectiveness. Common evaluation metrics include accuracy, precision, recall, and F1 score. These
metrics provide insights into the model's ability to correctly classify sentiments and measure the balance
between precision (correctly identified positive/negative sentiments) and recall (ability to identify all
positive/negative sentiments).

By systematically following these steps, the study aims to gather data, annotate sentiments, extract
meaningful features, perform sentiment analysis using machine learning algorithms, and evaluate the
performance of the models. These methods allow for a comprehensive analysis of user sentiments and
opinions towards ChatGPT, providing valuable insights for further research and improvements in
chatbot systems.

The following table shows the estimated costs for each category and subcategory, and
the total budget for the research. The currency used is the Ethiopian birr (ETB), and the
exchange rate is 1 USD = 44.42 ETB as of June 2024.
Table

Category Subcategory Cost (ETB) Cost (USD)

Personnel PI 360,000 8,107

RA 180,000 4,054

Annotator 90,000 2,027

Total 630,000 14,188

Travel Transportation 50,000 1,126

Accommodation 40,000 901

Category Subcategory Cost (ETB) Cost (USD)

Per diem 30,000 676

Total 120,000 2,703

Materials and Supplies Paper 5,000 113

Pens 1,000 23

Envelopes 2,000 45

Stamps 3,000 68

Total 11,000 248

Equipment Computer 20,000 451

Laptop 15,000 338

Printer 10,000 225

Scanner 10,000 225

Total 55,000 1,239

Software Python 0 0

scikit-learn 0 0
 Category Subcategory Cost (ETB) Cost (USD)

BeautifulSoup 0 0

NLTK 0 0

Total 0 0

Other Contingency 50,000 1,126

Overhead 100,000 2,253

Dissemination 34,000 766

Total 184,000 4,145

Grand Total 1,000,000 22,523

By providing a detailed breakdown of the estimated costs for each category and
subcategory, the budget section ensures transparency and feasibility of the research
project. The budget section also demonstrates that the research project is within the
allocated grant of $50,000 from the Ethiopian Ministry of Science and Technology, and
that the research project will provide valuable outcomes and benefits for the academic
community and the society at large.

Schedule The schedule section will present a timeline outlining the various tasks and
milestones of the research project. It will include specific dates for data collection,
annotation, feature extraction, sentiment analysis, evaluation, data analysis, and report
writing. By providing a comprehensive schedule, this section ensures that the research
project is well-organized and executed within the specified timeframe.
 The following table shows the estimated duration and deadlines for each task and
milestone of the research project. The research project will be conducted within a one-
year period, from January 2024 to December 2024.
Table

Task/Milestone Duration Deadline

Data collection 2 months March 2024

Annotation 1 month April 2024

Feature extraction 1 month May 2024

Sentiment analysis 2 months July 2024

Evaluation 1 month August 2024

Data analysis 2 months October 2024

Report writing 2 months December 2024

Total 12 months December 2024

Conclusion and Future Work The conclusion section will summarize the key findings of
the research, discussing the sentiments and opinions of Assosa University students and
social media users towards ChatGPT. It will highlight the implications of the research for
the design and improvement of chatbot systems, emphasizing user-centric approaches
and potential areas of enhancement. Additionally, the section will outline avenues for
future work, suggesting directions for further research and development in the field of
chatbots and sentiment analysis.

The main findings of the research are:

 Assosa University students and social media users in Ethiopia have different sentiments
and opinions towards ChatGPT, depending on their expectations, needs, and
preferences.
 The factors that influence the perceptions and evaluations of ChatGPT among the two
user groups include the chatbot’s personality, empathy, humor, feedback, and domain
knowledge.
 The best feature extraction technique and machine learning algorithm for sentiment
analysis on the collected data are TF-IDF and SVM, respectively, as they achieved the
highest performance metrics among the tested methods.
 The sentiment analysis results revealed that the majority of the users had positive
sentiments and opinions towards ChatGPT, indicating a high level of satisfaction and
trust in the chatbot.

The implications of the research are:

 The research contributes to the existing body of knowledge on user perceptions of

chatbots, providing specific and actionable insights into user preferences, expectations,
and areas of improvement for chatbot systems.
 The research informs the design and enhancement of ChatGPT specifically, as well as
guides the development of future chatbot systems that can adapt to the needs and
preferences of different user groups.
 The research fosters user-centric design approaches for chatbots, leading to more
engaging and effective human-computer interactions.

The avenues for future work are:

 To extend the data collection and analysis to include more user groups, such as
customers, patients, or teachers, and compare their sentiments and opinions towards
ChatGPT.
 To explore other feature extraction techniques and machine learning algorithms for
sentiment analysis, such as word embeddings, deep neural networks, or transformers,
and evaluate their performance and suitability for the task.
 To conduct qualitative research, such as interviews or focus groups, to gain a deeper
understanding of the user perceptions and sentiments towards ChatGPT, and to elicit
more detailed feedback and suggestions for improvement.

In summary, this research paper aims to investigate the sentiment and opinions of
Assosa University students and social media users towards ChatGPT, a novel and
advanced chatbot that uses a large-scale language model. By employing questionnaires,
social media data collection, sentiment analysis techniques, and machine learning
algorithms, we seek to gain valuable insights into user perceptions and preferences. The
findings will contribute to the advancement of chatbot systems, improving their
functionality and enhancing user experiences.
Scope of the Study

The scope of this study is focused on assessing the sentiment and opinions of two specific groups:
Assosa University students and social media users. These groups are selected to represent different
demographics and user segments, providing a diverse range of perspectives on ChatGPT. By targeting
Assosa University students, we aim to understand the sentiments and opinions of individuals within an
academic setting, potentially highlighting specific use cases and requirements relevant to educational
institutions. Meanwhile, gathering comments from social media platforms allows us to capture a
broader range of public opinions and perceptions towards ChatGPT.

The study will employ questionnaires administered to Assosa University students, specifically targeting a
representative sample from different faculties and programs. The questionnaires will be designed to
gather information on user experiences, perceptions, and evaluations of ChatGPT. Additionally,
comments and posts related to ChatGPT will be collected from popular social media platforms, ensuring
a wide range of viewpoints for analysis.

The sentiment analysis will focus on classifying the collected data into positive or negative sentiments
based on expressed opinions and emotions. Machine learning algorithms will be employed to perform
sentiment analysis on the gathered data, allowing for an automated and scalable approach.

It is important to note that this study does not aim to evaluate the technical aspects or performance of
ChatGPT itself. Instead, the focus is on understanding user sentiments and opinions towards the
chatbot, providing insights for its improvement and the development of future chatbot systems.

The geographical scope of this study is limited to Assosa University in a specific region. While the
findings may offer valuable insights into user perceptions within this context, they may not be directly
generalizable to other universities or regions. However, the methodology and approach utilized in this
study can serve as a foundation for future research in other settings.

Overall, the study aims to provide a comprehensive understanding of user sentiments and
opinions towards ChatGPT, specifically targeting Assosa University students and social media
users. The findings will contribute to the field of chatbot research, informing the design and
enhancement of chatbot systems to better meet user needs and expectations.
Scope of the Study The scope of this study is focused on assessing the sentiment and
opinions of two specific groups: Assosa University students and social media users in
Ethiopia towards ChatGPT, a novel and advanced chatbot that uses a large-scale
language model called GPT-3 to generate natural and engaging responses. ChatGPT
claims to be able to converse on any topic and provide relevant information, answers,
and suggestions1. Chatbots are increasingly used in various domains, such as customer
service, education, entertainment, and health care, and they have the potential to
improve the quality and efficiency of these services 2. However, the success and
satisfaction of chatbot systems depend largely on how users perceive and evaluate
them, which may vary depending on the characteristics and preferences of different user
groups34. Therefore, it is essential to understand the sentiment and opinions of users
towards chatbots, and how they can be influenced by factors such as personality,
empathy, humor, and feedback.

These groups are selected to represent different demographics and user segments,
providing a diverse range of perspectives on ChatGPT. By targeting Assosa University
students, we aim to understand the sentiments and opinions of individuals within an
academic setting, potentially highlighting specific use cases and requirements relevant
to educational institutions. Meanwhile, gathering comments from social media
platforms allows us to capture a broader range of public opinions and perceptions
towards ChatGPT.
The study will employ questionnaires administered to 100 Assosa University students,
specifically targeting a representative sample from different faculties and programs. The
questionnaires will be designed to gather information on user experiences, perceptions,
and evaluations of ChatGPT, using both closed-ended and open-ended questions. The
questionnaires will be distributed online, using Google Forms, to students who have
used ChatGPT at least once. The students will have two weeks to complete and submit
the questionnaires. Additionally, comments and posts related to ChatGPT will be
collected from popular social media platforms, such as Twitter, Facebook, and Reddit,
ensuring a wide range of viewpoints for analysis. The comments and posts will be
retrieved using web scraping and natural language processing techniques, such as
BeautifulSoup5 and NLTK. The search keywords will include “ChatGPT”, “GPT-3”, and
“chatbot”. The search period will span from January 2024 to June 2024, to capture the
most recent and relevant opinions on ChatGPT. A total of 1000 comments and posts will
be collected from each platform, resulting in a dataset of 3000 social media texts.

The sentiment analysis will focus on classifying the collected data into positive or
negative sentiments based on expressed opinions and emotions. Machine learning
algorithms will be employed to perform sentiment analysis on the gathered data,
allowing for an automated and scalable approach. The selected algorithms for sentiment
analysis may include Naive Bayes, Support Vector Machine, k-Nearest Neighbors,
Decision Tree, Stochastic Gradient Descent Classifier, and Logistic Regression. The
performance of the algorithms will be evaluated using metrics such as accuracy,
precision, recall, and F1 score.

It is important to note that this study does not aim to evaluate the technical aspects or
performance of ChatGPT itself. Instead, the focus is on understanding user sentiments
and opinions towards the chatbot, providing insights for its improvement and the
development of future chatbot systems.

The geographical scope of this study is limited to Assosa University in Benishangul-

Gumuz Region, Ethiopia. While the findings may offer valuable insights into user
perceptions within this context, they may not be directly generalizable to other
universities or regions. However, the methodology and approach utilized in this study
can serve as a foundation for future research in other settings.

Overall, the study aims to provide a comprehensive understanding of user sentiments

and opinions towards ChatGPT, specifically targeting Assosa University students and
social media users in Ethiopia. The findings will contribute to the field of chatbot
research, informing the design and enhancement of chatbot systems to better meet
user needs and expectations.