Project: CHATGPT

Project: ChatGPT
CS-250 Data Structures and Algorithms

Group Members
Afifa Attiq
Javeria Hakim
Mehtab Ameem

Submission Date: 15th January,2023

Submitted to: Ma’am Fauzia Ehsan

BS Mathematics, School Of Natural Sciences,
NUST
 Contents
Project: CHATGPT...............................................................................
Project: ChatGPT..............................................................................
.......................................................................................................

Inrtroduction
Motivation
Literature Review
Working of ChatGPT
Applications
Comparison with Google
Refrences

Introduction:

John McCarthy, the person who coined the term “artificial intelligence” (AI)
in 1955, describes it as machines that “… use language, form abstractions

Final Report PAGE 2

and concepts, solve kinds of problems now reserved for humans, and
improve themselves”. ChatGPT is one of the significant applications of AI.

ChatGPT presents an artificial intelligence chatbot which has been

developed by OpenAI. It is designed to converse with humans in a natural,
intuitive way, by using advanced machine learning techniques to generate
human-like responses to text input. It has been in the development phase for
several years and has undergone numerous updates and improvements
which made it one of the most advanced chatbots available today.

The origins of ChatGPT can be traced back to the development of GPT

(Generative Pre-training Transformer), an AI language model which was
created by OpenAI in 2018. GPT was designed to predict the next word in a
given sequence of words, using a large dataset of human-generated text to
learn about language structure and style. This technology proved to be
extremely successful and was quickly integrated into a variety of
applications, including machine translation, language generation, and even
music composition.

On the basis of the features and success of GPT, OpenAI managed to create
a chatbot that could hold natural conversations with humans. This led to the
development of ChatGPT, which was initially released in early 2020. It has
been trained on a massive dataset of hundreds of billions of words, allowing
it to generate high-quality language and perform a wide range of NLP tasks,
from text completion and translation to question answering and
summarization. Also, it has been fine-tuned for conversational language and
can be used to generate responses in a conversational context, such as in a
chatbot or virtual assistant, due to ability to generate human-like responses
to text input. By using advanced machine learning techniques, ChatGPT is
able to analyze the context and content of a conversation, and generate
appropriate responses based on this analysis. This allows it to hold natural,
flowing conversations with humans, rather than simply providing pre-
written responses to specific keywords or phrases.

Another significant feature of ChatGPT is its large knowledge base.

Namely, it has access to a vast dataset of human-generated text, which it
uses to learn about language, culture, and a wide range of topics. This
allows ChatGPT to provide accurate and detailed responses to a wide range
of questions. All that makes it useful for a range of NLP tasks, including but
not limited to text generation, question answering, and text classification.

Final Report PAGE 3

1.1 Motivation:
As we know it’s time of artificial intelligence and none is unaware of the
uses of ChatGPT, we feel to study the working and nature of the ChatGPT.
It is one of the most highly used AI products at this time. The high usage of
it leads to the development of ChatGPT 4 from ChatGPT3.

1.2 Literature Review:

The emergence of ChatGPT, a cutting-edge language model developed by

OpenAI, signifies a groundbreaking leap in the realm of natural language
processing, riding on the state-of-the-art GPT-3.5 architecture. This
sophisticated model has permeated diverse domains, demonstrating its
prowess in conversation, text generation, and information retrieval.

The GPT-3.5 architecture stands as the latest pinnacle in the evolution of

Generative Pre-trained Transformers (GPT). Renowned for their capacity to
produce coherent and contextually relevant text based on input prompts,
GPT models undergo a rigorous pre-training process. This involves
exposure to an extensive corpus of varied textual data, enabling the model to
intricately capture the subtle patterns and nuances inherent in language.

A hallmark feature of ChatGPT lies in its exceptional conversational

abilities. Unlike conventional language models, ChatGPT excels in
preserving context across prolonged interactions, culminating in the
generation of responses that exhibit a heightened level of coherence and
contextual relevance. This proficiency proves particularly critical in
applications such as chatbots, virtual assistants, and automated customer
support systems.

Final Report PAGE 4

Beyond its role in text generation, ChatGPT manifests a nuanced
understanding of context, intent, and user-specific intricacies. Its unique
capability to furnish contextually appropriate responses renders it an
invaluable tool across a spectrum of applications, spanning from content
creation and brainstorming to educational support and programming
assistance.

Furthermore, the versatility of ChatGPT is underscored by its ability to

seamlessly adapt to diverse writing styles and subjects, positioning it as an
all-encompassing tool catering to a broad user base. The underlying
transformer architecture empowers the model to discern the context of a
given prompt, generating text that aligns seamlessly with the user's input.

However, it is imperative to acknowledge that, notwithstanding its potency,

ChatGPT is not impervious to limitations. While the model can produce
responses that appear plausible, it may sometimes lack factual accuracy.
Sensitivity to the phrasing of input prompts adds another layer of
complexity, with slight alterations potentially yielding divergent or
inconsistent responses.

The widespread integration of ChatGPT into practical applications

underscores its potential to revolutionize the landscape of human-computer
interactions. As the scientific community and developers continue to delve
into avenues for enhancing the model's capabilities and mitigating its
limitations, ChatGPT persists as a vanguard in the continual evolution of
natural language processing, providing a tantalizing glimpse into the future
of intelligent and context-aware conversational agents

Final Report PAGE 5

Self-Attention Mechanism:
The self-attention mechanism enables the model to weigh the importance of different words in a sequence when
processing each word. It helps capture contextual information effectively

Fine Tuning:
Fine-tuning is a process where the pre-trained model is further trained on a specific dataset to adapt it to a
particular task or domain. Fine-tuning can enhance the model's performance for specific applications.

2.Analysis on the Working of the ChatGPT:

ChatGPT is based on ‘’Generative Pre-Trained Transformer’’ architecture, that uses transformer

neural networks. This architecture has become a fundamental building block for many natural
language processing tasks. The GPT in ChatGPT is mostly two related algorithms: GPT-3.5
Turbo and GPT-4, though the latter is only available in ChatGPT for ChatGPT Plus
Subscribers. The GPT bit stands for Generative Pre-trained Transformer, and the number
is just the version of the algorithm. The key components and algorithms that contribute to its
functioning include:

1. Transformer Architecture
2. Pre-Training
3. Tokenization
4. Self Attention Mechanism
5. Fine Tuning

Transformers Architecture:
The transformer architecture is the foundation of GPT models. It uses self-attention mechanisms to
capture dependencies between different words in a sentence, allowing the model to consider long-
range context efficiently.

Pre-Training:
ChatGPT undergoes pre-training on a large dataset. During pre-training, the model learns to predict

Final Report PAGE 6

the next word in a sentence given the context of preceding words. This process helps the model
capture general language pattern

Tokenization:
Tokenization is the process of breaking down input text into smaller units called tokens. GPT
models operate on these tokens, which can be words, sub words, or characters.

DETAILED VIEW ON IT’S ALGORITHMS

-TOKENS:
GPT-3 was trained on roughly 500 billion tokens, which allows its language models to
more easily assign meaning and predict plausible follow-on text by mapping them in
vector-space. Many words map to single tokens, though longer or more complex words
often break down into multiple tokens. On average, tokens are roughly four characters
long. OpenAI has stayed quiet about the inner workings of GPT-4, but we can safely
assume it was trained on much the same dataset since it's even more powerful. All the
tokens came from a massive corpus of data written by humans. That includes books,
articles, and other documents across all different topics, styles, and genres—and an
unbelievable amount of content scraped from the open internet. Basically, it was
allowed to crunch through the sum total of human knowledge to develop the network it
uses to generate text.

Based on all that training, GPT-3's neural network has 175 billion parameters or
variables that allow it to take an input (user’s prompt) and then, based on the values
and weightings it gives to the different parameters (and a small amount of
randomness), outputs whatever it thinks best matches your request. OpenAI hasn't said
how many parameters GPT-4 has, but it's a safe guess that it's more than 175 billion
and less than the once-rumored 100 trillion parameters. Regardless of the exact
number, more parameters doesn't automatically mean better. Some of GPT-4's
increased power probably comes from having more parameters than GPT-3, but a lot is
probably down to improvements in how it was trained.

Final Report PAGE 7

Transformer Architecture:
The network uses transformer architecture (the T in GPT) and was proposed in a research paper back in 2017
“Attention is All You Need’’. The transformer architecture is a type of neural network architecture. All this
training is intended to create a deep learning neural network that is a complex, many-layered, weighted
algorithm modeled after the human brain. It allowed ChatGPT to learn patterns and relationships in the text data
and tap into the ability to create human-like responses by predicting what text should come next in any given
sentence.

At the core of transformers is a process called "self-attention." Older recurrent neural networks (RNNs) read
text from left-to-right. This is fine when related words and concepts are beside each other, but it makes things
complicated when they're at opposite ends of the sentence. (It's also a slow way to compute things as it has to be
done sequentially.)

Transformers, however, read every word in a sentence at once and compare each word to all the others.
Transformers don't work with words, they work with "tokens," which are chunks of text encoded as a vector (a
number with position and direction). The closer two token-vectors are in space, the more related they are.
Similarly, attention is encoded as a vector, which allows transformer-based neural networks to remember
important information from earlier in a paragraph.

The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected
layers for both the encoder and decoder, shown in the left and right halves of Figure.

Final Report PAGE 8

COMPARISON BETWEEN CHATGPT AND OTHER SEARCH ENGINES:

ChatGPT Google

Final Report PAGE 9

In order to produce an output, ChatGPT analyzes Google, on the other hand, combines crawling,
input data using deep learning techniques and a indexing, ranking, and other algorithms. It gathers
transformer architecture. This enables the system to information by crawling the internet, classifies it
interpret the input's context and offer pertinent using a variety of indexing techniques, and then uses
responses. But the data was trained on limits how search engine optimization and ranking algorithms to
accurate the output may be. However, accuracy can order the results according to relevancy. As a result,
be raised. Google is able to deliver precise and pertinent results
for a variety of inquiries.
Artificial intelligence is used by ChatGPT to Conversely, Google functions as a search engine.
comprehend and respond to questions in natural With the use of this application, you can search
language. It's similar to speaking with an informed through billions of web pages to find information on
buddy who can relate to your questions and provide the internet. For information on a variety of subjects,
you valuable advice. ChatGPT is best for answering including news articles and shopping websites,
specific questions and having a conversation Google is the best resource. Google provides a list
of links to websites that contain information related
to your search.
It's an Artificial Intelligence model. It’s a worldwide search engine.
It primarily focused on generating human-like It is useful for many different things, such as
texts. language translation, picture and speech recognition,
and much more.
It uses GPT 3.5 and GPT 4 It Uses LAMDA and PALM 2 as language model.
It work on transformer architecture, tokenization and Most search engines use graphs , inverted index as
fine tuning. data structures. Graphs are kind of data structures that
are used by google. Moreover, Google uses tries to
predict our searches. Google
uses PageRank Algorithm developed by
its founders Sergey Brin and Larry Page.
Today Google’s algorithms rely on more
than 200 unique signals which include
things like the terms on websites, the
freshness of content and our region that
make it possible to guess what we might
really be looking for.

ChatGPT AND WOLFRAM:

Final Report PAGE 10

References:

1] J. McCarthy, M. L Minsky, N. Rochester, and C. E. Shannon, “A proposal for the

Dartmouth summer research project on artificial intelligence,” AI Magazine, 27(4),
2006, pp. 12-14.

[2] F. Jia, D, Sun, Q, Ma, and C. K. Looi,. “Developing an AI-Based Learning System for
L2 Learners’ Authentic and Ubiquitous Learning in English Language,” Sustainability
14, no. 23, 2022, 15527.

3) Bhavya, B., Xiong, J., and Zhai, C. Analogy generation by prompting large language
models: A case study of instructgpt. arXiv preprint arXiv:2210.04186 (2022).

4) zapier.com/blog/how-does-ChatGPT-work

5) https://doi.org/10.48550/arXiv.1706.03762 Ashish Vaswani, Noam Shazeer, Niki

Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

Final Report PAGE 11

6) ChatGPT is not all you need. A State of the Art Review of large Generative AI models
Roberto Gozalo-Brizuela, Eduardo C. Garrido-Merch´an

7) Effectiveness of ChatGPT in Essay Autograding Ahmed B. Altamimi College of

computer scienc and engineering, Univeristy of Ha’il

8) Use of ChatGPT in ESP Teaching Process Darko Kovačević University of East

Sarajevo Faculty of Electrical Engineering East Sarajevo, Bosnia and Herzegovina
[email protected]

9) www.semrush.com/blog/google-search-algorithm/

10) A Survey on ChatGPT: AI–Generated Contents, Challenges, and Solutions YUNTAO

WANG , YANGHE PAN , MIAO YAN , ZHOU SU (Senior Member, IEEE), AND TOM H.
LUAN School of Cyber Science, Engineering, Xi’an Jiaotong University, Xi’an 710049,
China CORRESPONDING AUTHOR: ZHOU SU (e-mail: [email protected]).

Final Report PAGE 12