Descriptive and MCQ questions will be asked.

Code related questions from Lemmatization and

Stemming only. Topics in Bold are important

Unit – I

• What is NLP?
NLP stands for Natural Language Processing. It's a field of artificial intelligence that deals with the
interaction between computers and human (natural) languages. In simpler terms, it's about teaching
computers to understand, interpret, and generate human language.

Here are some key areas of NLP:

 Machine Translation: Translating text from one language to another.

 Sentiment Analysis: Determining the sentiment (positive, negative, or neutral) of a piece of

text.

 Text Summarization: Creating a concise summary of a longer text.

 Question Answering: Answering questions based on a given text.

 Chatbots and Virtual Assistants: Creating conversational agents that can interact with
humans.

NLP has a wide range of applications, including:

 Customer service: Chatbots and virtual assistants can provide quick and efficient customer
support.

 Search engines: NLP algorithms can help search engines better understand and rank search
queries.

 Healthcare: NLP can be used to analyze medical records and extract relevant information.

 Social media: NLP can be used to monitor social media sentiment and identify trends.

Natural Language Processing (NLP) is a subfield of artificial intelligence that focuses on the
interaction between computers and human (natural) languages. It involves developing algorithms
and techniques that allow computers to understand, interpret, and generate human language in a
meaningful way.

Key technical aspects of NLP include:

 Tokenization: Breaking down text into individual words or tokens.

 Part-of-Speech Tagging: Identifying the grammatical category of each word (e.g., noun, verb,
adjective).

 Named Entity Recognition: Identifying named entities in text, such as people, organizations,
and locations.

 Syntax Parsing: Analyzing the grammatical structure of sentences to understand the

relationships between words.
  Semantic Analysis: Understanding the meaning of words and phrases in context.

 Machine Learning: Using statistical models and machine learning algorithms to learn
patterns and relationships in language data.

 Deep Learning: Applying deep neural networks, such as recurrent neural networks (RNNs)
and transformers, to capture complex language patterns.

NLP techniques are used in various applications, including:

 Machine Translation: Translating text from one language to another.

 Sentiment Analysis: Determining the sentiment (positive, negative, or neutral) of a piece of

text.

 Text Summarization: Creating a concise summary of a longer text.

 Question Answering: Answering questions based on a given text.

 Chatbots and Virtual Assistants: Creating conversational agents that can interact with
humans.

Challenges in NLP:

 Ambiguity: Natural language is often ambiguous, making it difficult for computers to

understand the intended meaning.

 Context: The meaning of words can change depending on the context in which they are
used.

 Diversity: Natural language is diverse, with many different dialects, accents, and styles.

Despite these challenges, NLP has made significant advancements in recent years, and it is a rapidly
growing field with many potential applications.

• Turing Test
The Turing test, as applied to NLP, is a thought experiment to determine if a machine can exhibit
intelligent behavior indistinguishable from that of a human, specifically in a conversational setting.

Key points to consider:

 Conversational Abilities: The test focuses on a machine's ability to engage in natural

language conversation, understanding and responding to prompts in a way that is
indistinguishable from a human.

 NLP Techniques: NLP techniques like tokenization, part-of-speech tagging, named entity
recognition, syntax parsing, and semantic analysis are crucial for a machine to perform well
in the Turing test.

 Challenges: Even with advancements in NLP, machines still face challenges in understanding
nuances, context, and subtle human cues that are essential for genuine conversation.
  Practical Applications: While passing the Turing test is a significant milestone, it's not the
sole measure of AI intelligence. NLP advancements have practical applications in chatbots,
virtual assistants, and language translation.

In essence, the Turing test serves as a benchmark for evaluating the progress of NLP systems. A
machine that can pass the Turing test would demonstrate a high level of natural language
understanding and generation, but it doesn't necessarily mean it possesses true consciousness or
sentience.

• Structure of Natural Language

Here's a brief overview of common POS tags:

 Nouns (N): Refer to people, places, things, or ideas.

 Verbs (V): Express actions or states of being.

 Adjectives (ADJ): Describe nouns.

 Adverbs (ADV): Modify verbs, adjectives, or other adverbs.

 Pronouns (PRN): Replace nouns.

 Prepositions (PREP): Show relationships between nouns and other words.

 Conjunctions (CONJ): Connect words, phrases, or clauses.

 Determiners (DET): Specify nouns (e.g., "the," "a," "this").

 Interjections (INTJ): Express emotions or exclamations.

• Applications of NLP
NLP has a wide range of applications across various industries. Here are some prominent examples:

Customer Service and Support:

 Chatbots and Virtual Assistants: NLP-powered chatbots can handle customer inquiries,
provide support, and even resolve issues.

 Sentiment Analysis: Analyzing customer feedback to understand their satisfaction levels and
identify areas for improvement.

Healthcare:

 Medical Record Analysis: Extracting relevant information from unstructured medical records
for research and clinical decision-making.

 Drug Discovery: Analyzing scientific literature to identify potential drug targets and side
effects.

Social Media Analysis:

  Sentiment Analysis: Understanding public sentiment towards brands, products, or events.

 Topic Modeling: Identifying trending topics and discussions on social media platforms.

Search Engines:

 Natural Language Search: Enabling users to search with more natural language queries,
improving search results.

 Semantic Search: Understanding the underlying meaning of search queries to provide more
relevant results.

Language Translation:

 Machine Translation: Translating text from one language to another, improving accuracy and
fluency.

Legal and Compliance:

 Document Analysis: Analyzing legal documents for key information and compliance issues.

 Contract Review: Identifying potential risks and inconsistencies in contracts.

Education:

 Language Learning: Providing personalized language learning experiences.

 Grading and Assessment: Automating the grading of essays and other written assignments.

Content Creation:

 Generating Creative Content: Assisting writers in generating ideas, writing outlines, or even
creating entire pieces of content.

Financial Services:

 Risk Assessment: Analyzing financial news and reports to identify potential risks.

 Customer Churn Prediction: Predicting customer churn based on their interactions and
sentiment.

These are just a few examples of the many applications of NLP. As the technology continues to
advance, we can expect to see even more innovative and impactful uses in the future.

• Knowledge of Language
• Advantage of NLP
NLP has numerous advantages that make it a valuable tool in various fields. Here are some of the key
benefits:

Improved Efficiency and Productivity:

 Automation: NLP can automate tasks that would otherwise require human intervention,
leading to increased efficiency and productivity.
  Time-Saving: NLP can quickly process large amounts of text data, saving time and effort.

Enhanced Data Analysis:

 Unstructured Data: NLP can extract valuable insights from unstructured text data, which is
often difficult to analyze using traditional methods.

 Information Extraction: NLP can identify and extract relevant information from text, such as
named entities, keywords, and relationships.

Improved Customer Experience:

 Personalized Interactions: NLP can enable personalized interactions with customers,

enhancing their overall experience.

 Efficient Support: NLP-powered chatbots can provide quick and efficient customer support,
reducing wait times.

Better Decision Making:

 Data-Driven Insights: NLP can provide data-driven insights that can inform decision-making
processes.

 Risk Mitigation: NLP can help identify potential risks and challenges by analyzing text data.

Language Accessibility:

 Translation: NLP can facilitate communication between people who speak different
languages.

 Accessibility: NLP can make information more accessible to people with disabilities, such as
those who are visually impaired or have language impairments.

Research and Development:

 Text Analysis: NLP can be used to analyze vast amounts of text data for research purposes,
such as studying language patterns, historical trends, or scientific literature.

 Knowledge Discovery: NLP can help discover new knowledge and insights from existing data.

Cost-Effectiveness:

 Scalability: NLP can be easily scaled to handle large volumes of data, making it a cost-
effective solution.

 Reduced Labor Costs: By automating tasks, NLP can reduce the need for human labor,
leading to cost savings.

These are just a few of the many advantages of NLP. As the technology continues to evolve, we can
expect to see even more innovative and impactful applications in the future.

• Disadvantage of NLP
Disadvantages of Natural Language Processing (NLP)
While NLP offers numerous advantages, it also has some inherent challenges and limitations:

Ambiguity and Contextual Understanding:

 Polysemy: Many words have multiple meanings, making it difficult for NLP systems to
accurately interpret their intended meaning in context.

 Contextual Understanding: NLP models may struggle to understand the nuances of language
and the context in which words are used.

Data Quality and Quantity:

 Data Dependence: NLP models rely on large amounts of high-quality data to train effectively.
Lack of sufficient or relevant data can limit their performance.

 Data Bias: Bias in the training data can lead to biased outputs from the NLP model.

Computational Complexity:

 Resource-Intensive: NLP models, especially deep learning-based models, can be

computationally expensive to train and run. This can limit their scalability and accessibility.

Domain Specificity:

 Generalizability: NLP models trained on one domain may not perform well on another
domain, requiring domain-specific training data and techniques.

Evaluation Challenges:

 Subjectivity: Evaluating the performance of NLP systems can be subjective, as there is no

single standard for measuring accuracy or quality.

Ethical Considerations:

 Bias and Discrimination: NLP models can perpetuate biases present in the training data,
leading to discriminatory outcomes.

 Privacy Concerns: Handling large amounts of personal data raises privacy concerns.

Language Variation:

 Dialects and Accents: NLP models may struggle to understand different dialects, accents, or
regional variations of a language.

• Components of NLP – NLU/NLG

NLP: NLU and NLG

NLP (Natural Language Processing) is broadly divided into two main components:

1. NLU (Natural Language Understanding):

 Focuses on the computer's ability to comprehend and interpret human language.

 Involves tasks like:

 o Tokenization: Breaking text into individual words or tokens.

o Part-of-Speech Tagging: Identifying the grammatical category of each word.

o Named Entity Recognition: Recognizing entities like people, places, and

organizations.

o Syntax Parsing: Analyzing the grammatical structure of sentences.

o Semantic Analysis: Understanding the meaning of words and phrases.

o Intent Recognition: Determining the underlying goal or purpose of a statement.

2. NLG (Natural Language Generation):

 Focuses on the computer's ability to generate human-readable text.

 Involves tasks like:

o Text Generation: Creating coherent and informative text based on input data.

o Summarization: Condensing longer text into shorter summaries.

o Translation: Translating text from one language to another.

o Question Answering: Generating informative responses to questions.

In essence, NLU is about understanding what is said, while NLG is about generating what should be
said.

• Phases of NLP
Phases of Natural Language Processing (NLP)

NLP typically involves several interconnected phases:

1. Text Preprocessing:

 Tokenization: Breaking text into individual words or tokens.

 Normalization: Converting text to a standard format (e.g., lowercase, removing
punctuation).
 Stemming or Lemmatization: Reducing words to their root form (e.g., "running" to "run").

2. Feature Extraction:

 Bag-of-Words: Representing text as a numerical vector where each element corresponds to

the frequency of a word.
 TF-IDF: Weighting terms based on their frequency in a document and the corpus.
 Word Embeddings: Representing words as dense vectors in a continuous space, capturing
semantic relationships.
3. Language Modeling:

 Statistical Language Models: Predicting the next word in a sequence based on previous
words.
 Neural Language Models: Using deep learning techniques to capture complex language
patterns.

4. Named Entity Recognition (NER):

 Identifying named entities in text, such as people, organizations, and locations.

5. Part-of-Speech Tagging (POS):

 Assigning grammatical categories to words (e.g., noun, verb, adjective).

6. Dependency Parsing:

 Analyzing the grammatical structure of sentences to understand the relationships between

words.

7. Sentiment Analysis:

 Determining the sentiment (positive, negative, or neutral) of a piece of text.

8. Question Answering:

 Answering questions based on given text or knowledge base.

9. Machine Translation:

 Translating text from one language to another.

10. Text Summarization:

 Creating a concise summary of a longer text.

These phases are often interconnected and can be combined in various ways to address different
NLP tasks. For example, a question answering system might involve text preprocessing, named entity
recognition, dependency parsing, and information retrieval.

• Difficulty in NLP :
Challenges in Natural Language Processing (NLP)

NLP, while a powerful field, faces several challenges:

Ambiguity and Contextual Understanding:

  Polysemy: Many words have multiple meanings, making it difficult for NLP systems to
accurately interpret their intended meaning in context.

 Contextual Understanding: NLP models may struggle to understand the nuances of language
and the context in which words are used.

Data Quality and Quantity:

 Data Dependence: NLP models rely on large amounts of high-quality data to train effectively.
Lack of sufficient or relevant data can limit their performance.

 Data Bias: Bias in the training data can lead to biased outputs from the NLP model.

Computational Complexity:

 Resource-Intensive: NLP models, especially deep learning-based models, can be

computationally expensive to train and run. This can limit their scalability and accessibility.

Domain Specificity:

 Generalizability: NLP models trained on one domain may not perform well on another
domain, requiring domain-specific training data and techniques.

Evaluation Challenges:

 Subjectivity: Evaluating the performance of NLP systems can be subjective, as there is no

single standard for measuring accuracy or quality.

Ethical Considerations:

 Bias and Discrimination: NLP models can perpetuate biases present in the training data,
leading to discriminatory outcomes.

 Privacy Concerns: Handling large amounts of personal data raises privacy concerns.

Language Variation:

 Dialects and Accents: NLP models may struggle to understand different dialects, accents, or
regional variations of a language.

• Writing System

• Basic approaches to Problem Solving for NLP Problem

Unit – II

• Text Pre-Processing/ Segmentation

Text Preprocessing/Segmentation in NLP

Text preprocessing is a crucial step in NLP, where raw text is transformed into a structured format
suitable for further analysis. Segmentation is a specific task within preprocessing that involves
breaking down text into smaller units, typically words or sentences.

Key Tasks in Text Preprocessing/Segmentation:

 1. Tokenization:

o Breaking text into individual words or tokens.

o Example: "The quick brown fox jumps over the lazy dog." becomes ["The", "quick",
"brown", "fox", "jumps", "over", "the", "lazy", "dog"].

2. Normalization:

o Converting text to a standard format. This might involve:

 Lowercasing: Converting all letters to lowercase.

 Removing Punctuation: Removing punctuation marks.

 Removing Stop Words: Removing common words (e.g., "the," "and," "a")
that often don't carry significant meaning.

 Stemming or Lemmatization: Reducing words to their root form (e.g.,

"running" becomes "run").

3. Sentence Segmentation:

o Identifying sentence boundaries. This is typically done using punctuation marks like
periods, question marks, and exclamation points.

Challenges in Text Preprocessing:

 Language-Specific Rules: Different languages have different rules for tokenization and
sentence segmentation.

 Ambiguity: Some words or phrases can have multiple interpretations, making it difficult to
determine the correct segmentation.

 Noise and Errors: Text data can contain noise, such as typos or inconsistencies, that can
affect the preprocessing process.

Tools and Libraries:

 NLTK (Natural Language Toolkit): A popular Python library for NLP tasks, including
tokenization and stemming.

 spaCy: Another Python library known for its speed and efficiency, offering features like
tokenization, part-of-speech tagging, and named entity recognition.

 Gensim: A Python library for topic modeling, document similarity, and indexing.

• Pictograms, Ideograms and Logograms

• Concepts of Parts-of speech (POS Tagging)

Concepts of Part-of-Speech Tagging (POS Tagging)
Part-of-Speech (POS) tagging is a fundamental task in natural language processing (NLP) that
involves identifying the grammatical category of each word in a sentence. This information is
crucial for understanding the syntactic structure and meaning of the text.

Key Concepts:

1. POS Tags:

o Nouns (N): Refer to people, places, things, or ideas (e.g., "dog," "house," "love").

o Verbs (V): Express actions or states of being (e.g., "run," "is," "become").

o Adjectives (ADJ): Describe nouns (e.g., "big," "red," "happy").

o Adverbs (ADV): Modify verbs, adjectives, or other adverbs (e.g., "quickly," "very,"
"often").

o Pronouns (PRN): Replace nouns (e.g., "he," "she," "it").

o Prepositions (PREP): Show relationships between nouns and other words (e.g.,
"in," "of," "with").

o Conjunctions (CONJ): Connect words, phrases, or clauses (e.g., "and," "but," "or").

o Determiners (DET): Specify nouns (e.g., "the," "a," "this").

o Interjections (INTJ): Express emotions or exclamations (e.g., "wow," "ouch").

2. Tagging Schemes:

o Penn Treebank Tagset: A widely used tagset in English NLP.

o Universal Dependencies: A cross-lingual tagset that aims to standardize POS

tagging across different languages.

3. Tagging Algorithms:

o Rule-Based Taggers: Use predefined rules to assign POS tags.

o Statistical Taggers: Use probabilistic models to assign POS tags based on word
frequencies and context.

o Machine Learning Taggers: Employ machine learning algorithms (e.g., Hidden

Markov Models, Conditional Random Fields) to learn from labeled data and assign
POS tags.

Applications of POS Tagging:

 Syntax Analysis: Understanding the grammatical structure of sentences.

 Information Extraction: Identifying named entities and relationships between them.

 Machine Translation: Translating text from one language to another.

 Text Summarization: Creating concise summaries of longer texts.

 Sentiment Analysis: Determining the sentiment (positive, negative, or neutral) of a piece of

text.
• Different NLP tasks
Different NLP Tasks

Natural Language Processing (NLP) encompasses a wide range of tasks that involve the interaction
between computers and human language. Here are some of the most common NLP tasks:

Text Analysis and Understanding:

 Tokenization: Breaking text into individual words or tokens.

 Part-of-Speech Tagging (POS): Identifying the grammatical category of each word.

 Named Entity Recognition (NER): Recognizing named entities like people, organizations, and
locations.

 Dependency Parsing: Analyzing the grammatical structure of sentences to understand the

relationships between words.

 Sentiment Analysis: Determining the sentiment (positive, negative, or neutral) of a piece of

text.

 Topic Modeling: Identifying the main topics or themes present in a collection of documents.

 Text Summarization: Creating a concise summary of a longer text.

Language Generation:

 Machine Translation: Translating text from one language to another.

 Text Generation: Generating human-like text, such as creating summaries, writing articles, or
generating creative content.

 Question Answering: Answering questions based on given text or knowledge.

Dialog Systems:

 Chatbots and Virtual Assistants: Creating conversational agents that can interact with
humans.

 Dialog Management: Managing the flow of a conversation and understanding user intent.

Information Retrieval:

 Search Engines: Improving search results by understanding the meaning of search queries
and relevance of documents.

 Document Classification: Categorizing documents into predefined categories.

Other Tasks:

 Machine Learning: Applying machine learning techniques to NLP tasks, such as training
models to perform sentiment analysis or text classification.

 Knowledge Graph Construction: Building knowledge graphs to represent relationships

between entities.

 Text-to-Speech (TTS): Converting text into spoken language.

  Speech-to-Text (STT): Converting spoken language into text.

• Challenges of Text Pre-processing

• What is Corpus? What is Corpus Independent?
Corpus refers to a large collection of text data. It can be a collection of books, articles, newspapers,
websites, or any other form of written material. Corpora are used to train and evaluate NLP models,
providing them with a vast amount of data to learn from.

Corpus-Independent means that a model or technique is not reliant on a specific corpus. This implies
that the model can be applied to different corpora without requiring significant retraining or
adjustments. It's a desirable property for NLP models as it makes them more versatile and adaptable
to various text data sources.

In essence, a corpus-independent NLP model is one that can effectively handle a variety of textual
data, regardless of its specific source or characteristics.

• Tokenization
Tokenization is the process of breaking down text into individual units called tokens. These tokens
can be words, punctuation marks, or other linguistic units depending on the specific task and
language.

Key aspects of tokenization:

 Unit of Segmentation: Tokens can be words, subwords (e.g., characters or n-grams), or other
linguistic units.

 Language-Specific Rules: Different languages have different rules for tokenization, such as
handling compound words or contractions.

 Noise Removal: Tokenization often involves removing noise from the text, such as
punctuation marks, stop words, or special characters.

Example:

The sentence "The quick brown fox jumps over the lazy dog." can be tokenized as:

["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]

Applications of tokenization:

 Text Analysis: Tokenization is a fundamental step in many NLP tasks, such as sentiment
analysis, information extraction, and machine translation.

 Search Engines: Tokenization is used to break down search queries and documents into
searchable units.
  Information Retrieval: Tokenization helps in identifying relevant documents based on
keyword matching.

Challenges in tokenization:

 Ambiguous Boundaries: Some words may have ambiguous boundaries, making it difficult to
determine where one token ends and another begins (e.g., "can't" vs. "can" and "t").

 Language-Specific Rules: Different languages have different rules for tokenization, making it
challenging to develop a universal tokenizer.

 Noise and Errors: Text data can contain noise, such as typos or inconsistencies, which can
affect the tokenization process.

• Lemmatization - Refer to Code uploaded on drive

• Stemming - Refer to Code uploaded on drive
• Named Entity Recognition (NER)
Named Entity Recognition (NER) is a subtask of natural language processing (NLP) that involves
identifying named entities in text. Named entities are real-world objects such as people,
organizations, locations, dates, times, and quantities.

Key aspects of NER:

 Entity Types: NER systems typically focus on identifying specific entity types, such as person,
organization, location, date, time, and quantity.

 Contextual Understanding: NER systems must consider the context of the text to accurately
identify named entities.

 Boundary Detection: NER systems must accurately determine the boundaries of named
entities within the text.

Examples of named entities:

 People: Barack Obama, Elon Musk, Queen Elizabeth II

 Organizations: Apple, Google, NASA

 Locations: New York City, Paris, Mount Everest

 Dates: January 1, 2024, yesterday, next week

 Times: 3:15 PM, 10:00 AM

 Quantities: 100, 25%, 3.14

Applications of NER:

 Information Extraction: Extracting key information from text, such as identifying the people
involved in an event or the location of a business.
  Question Answering: Answering questions based on the information extracted from text.

 Sentiment Analysis: Understanding the sentiment expressed towards named entities.

 Knowledge Graph Construction: Building knowledge graphs to represent relationships

between entities.

 Machine Translation: Improving the accuracy of machine translation by correctly identifying

named entities.

Challenges in NER:

 Ambiguity: Many words can have multiple meanings, making it difficult to determine
whether they are named entities.

 Contextual Understanding: NER systems must consider the context of the text to accurately
identify named entities.

 Named Entity Overlap: Named entities can overlap or be nested within each other (e.g.,
"John F. Kennedy International Airport").

• How to build an NLP pipeline

Building an NLP Pipeline: A Step-by-Step Guide

An NLP pipeline is a series of steps involved in processing natural language text to extract meaningful
information. Here's a general outline of the steps involved:

1. Data Collection and Preprocessing:

 Gather data: Collect a relevant dataset of text data, considering factors like language,
domain, and task.

 Clean and preprocess data:

o Tokenization: Break text into individual words or tokens.

o Normalization: Convert text to a standard format (e.g., lowercase, remove

punctuation).

o Stemming or Lemmatization: Reduce words to their root form.

o Stop word removal: Remove common words that don't carry significant meaning.

2. Feature Engineering:

 Represent text as numerical vectors: Convert text data into a numerical representation that
can be processed by machine learning algorithms.

 Common techniques:

o Bag-of-Words

o TF-IDF (Term Frequency-Inverse Document Frequency)

 o Word Embeddings (e.g., Word2Vec, GloVe, BERT)

3. Model Selection and Training:

 Choose a suitable model: Select an appropriate NLP model based on the task and data
characteristics.

 Train the model: Feed the preprocessed data and features to the model to learn patterns and
relationships.

4. Evaluation:

 Evaluate model performance: Use appropriate metrics (e.g., accuracy, precision, recall, F1-
score) to assess the model's effectiveness on a separate test dataset.

 Iterate and refine: If necessary, adjust the model, features, or preprocessing steps to improve
performance.

5. Deployment:

 Integrate into application: Integrate the trained model into your application or system.

 Handle new data: Ensure the model can process new, unseen data effectively.

Example NLP Pipeline for Sentiment Analysis:

1. Data Collection: Gather a dataset of labeled text reviews.

2. Preprocessing: Tokenize, normalize, and remove stop words from the text.

3. Feature Engineering: Convert the preprocessed text into numerical features using techniques
like TF-IDF.

4. Model Selection: Choose a classification model like Naive Bayes, Support Vector Machine, or
a deep learning model.

5. Training: Train the model on the labeled data.

6. Evaluation: Evaluate the model's accuracy in predicting sentiment.

7. Deployment: Integrate the trained model into an application to analyze new reviews.

Remember: The specific steps and techniques used in an NLP pipeline will vary depending on the
task and the available resources. It's often a process of experimentation and iteration to find the best
approach for a given problem.