NOTES

Unit I: Introduction

Introduction to Natural Language Processing

 NLP Definition: Field that enables computers to understand, interpret, and generate human
language.
 Applications: Machine translation, chatbots, text summarization, etc.
 Key Challenges: Ambiguity, context understanding, and language variation.

Regular Expressions

 Definition: Patterns used to match character combinations in text.

 Key Symbols:
o . (any character)
o * (zero or more)
o + (one or more)
o \d (digit), \w (word character), \s (whitespace)
 Use Cases: Tokenization, pattern matching, text cleaning.

Words and Corpora

 Words: Fundamental unit of text; can include tokens like punctuation or symbols.
 Corpora: Large text collections used for training NLP models.
 Examples: Brown Corpus, Penn Treebank, etc.

Text Normalization

 Purpose: Standardizing text for consistent processing.

 Steps:
o Lowercasing
o Removing punctuation
o Expanding contractions (e.g., "can't" → "cannot")
o Removing stopwords (e.g., "the", "and")

Minimum Edit Distance

 Definition: Measures the difference between two strings by calculating the minimum number
of insertions, deletions, or substitutions required.
 Algorithm:
o Levenshtein Distance for general string comparison.
o Dynamic Programming used for efficient calculation.
Unit II: N-gram Language Models and Naïve Bayes

N-gram Language Models

 Definition: Predicts the probability of a word based on its preceding words.

 Types: Unigram (1 word), Bigram (2 words), Trigram (3 words), etc.
 Formula:

P(wn∣w1n−1)≈P(wn∣wn−1)P(w_n | w_1^{n-1}) \approx P(w_n | w_{n-1})

Evaluating Language Models

 Perplexity: Measures model uncertainty; lower values indicate better models.

Perplexity=2−1N∑i=1Nlog⁡2P(wi)\text{Perplexity} = 2^{-\frac{1}{N} \sum_{i=1}^N \log_2

P(w_i)}

Generalization and Zeros

 Challenge: Unseen words result in zero probabilities.

 Solution: Use smoothing techniques.

Smoothing Techniques

 Add-one (Laplace) Smoothing: Adds 1 to each count.

 Kneser-Ney Smoothing: Advanced method emphasizing rare but meaningful word
combinations.

Huge Language Models and Stupid Backoff

 Huge Models: Handle large-scale data for better predictions.

 Stupid Backoff: Efficient approximation method for large-scale models; ignores probability
normalization.

Perplexity’s Relation to Entropy

 Entropy: Measures the average uncertainty in a language model.

 Lower entropy implies a better, more predictable model.

Naive Bayes and Sentiment Classification

 Naive Bayes Classifier: A simple probabilistic model based on Bayes' Theorem.

 Formula:

P(y∣X)=P(X∣y)⋅P(y)P(X)P(y|X) = \frac{P(X|y) \cdot P(y)}{P(X)}

 Independence Assumption: Each feature is conditionally independent given the class.

Training the Naive Bayes Classifier

 Steps:
o Extract features (e.g., word counts)
o Calculate class probabilities
o Classify text based on maximum likelihood

Optimizing for Sentiment Analysis

 Feature Engineering: Use word frequencies, presence of negation, and emotive language.
 Common Datasets: IMDB reviews, Twitter sentiment datasets.

Naive Bayes for Other Text Classification

 Applications: Spam filtering, topic classification, etc.

 Advantage: Fast and effective for large text datasets.

Naive Bayes as a Language Model

 Use Case: Word prediction by modeling conditional probabilities.

Evaluation Metrics

 Precision: Proportion of true positives among predicted positives.

 Recall: Proportion of true positives among actual positives.
 F-measure (F1 Score): Harmonic mean of precision and recall.
 Cross-validation: Ensures robust model evaluation across different data subsets.

Statistical Significance Testing

 Purpose: Confirms model performance differences are meaningful.

 Common Tests: t-test, chi-square test.

Avoiding Harms in Classification

 Bias Mitigation: Ensure fairness in model training and decision-making.

 Privacy Protection: Safeguard user data when building NLP applications.

This summary highlights key concepts while maintaining clarity and focus on practical
insights. Would you like expanded explanations, diagrams, or sample questions for specific
topics?
SAMPLE PAPER-

SECTION A

S.No. | Question | Marks

1. A media monitoring agency leverages NLP to track online conversations about public
figures.
Evaluate the risks of using Regular Expressions for trend analysis when handling
informal language, internet slang, and cultural references.
(4 Marks)

Solution: Risks of Regular Expressions in NLP trend analysis:

 Ambiguity in Slang: Slang terms often have inconsistent spelling patterns, making
regex patterns unreliable.
 Regional Variations: Language nuances in dialects can result in regex patterns
missing important content.
 Overfitting: Creating overly complex regex patterns may fail when new slang terms
emerge.

2. A public safety organization wants to build an NLP system that identifies emergency-
related messages from social media posts during crises.
Develop a solution that applies text normalization and Regular Expressions to detect
critical alerts.
(4 Marks)

Solution:

 Text Normalization:
o Use tm or textclean in R to convert text to lowercase, remove stopwords,
and expand contractions.
 Regular Expressions:
o Create patterns for common emergency terms (e.g., "flood", "earthquake",
"help needed").
 Alert System:
o Implement keyword filtering and pattern matching to flag critical alerts for
rapid response teams.

3. A financial services firm implements Google Cloud AI for detecting fraudulent credit
card transactions.
Justify the effectiveness of a Hybrid Expert System in this scenario.
(4 Marks)

Solution: A Hybrid Expert System improves fraud detection by:

 Rule-Based Logic: Identifies known fraud patterns (e.g., high-value transactions in

unusual locations).
  Model-Based Learning: Uses ML models to adapt to evolving fraud tactics.
 Combination Advantage: Balances speed with adaptability, ensuring improved
detection accuracy.

4. A travel booking platform aims to enhance automated language identification for

multilingual customer interactions.
Develop a solution that integrates Kneser-Ney smoothing for accurate language
detection.
(4 Marks)

Solution:

 N-gram Model Integration: Use Kneser-Ney smoothing with n-grams to improve

probability estimates for rare language patterns.
 Language Corpus Preparation: Train on multilingual datasets to capture linguistic
diversity.
 Prediction Refinement: Use back-off techniques to enhance accuracy in low-
frequency phrases.

SECTION B

5. In response to rising misinformation during public health crises, media platforms are
adopting NLP-based systems to detect and highlight misleading content. A health
watchdog organization wants to build a cloud-based AI/ML solution on AWS Cloud
to analyze text posts across multiple languages.

The system must:

 Use N-gram Language Models to predict and flag misleading content by analyzing
language patterns.
 Integrate a Naive Bayes Classifier to categorize content as trustworthy, suspicious,
or false based on text features.
 Employ Kneser-Ney Smoothing to refine rare but critical phrases (e.g., "vaccine
hoax" or "cure scam").
 Ensure ethical considerations by reducing false positives that may harm legitimate
health campaigns.

Question: Design a comprehensive cloud-based NLP solution that addresses the following:

1. Architecture of your proposed system using AWS services (e.g., Amazon

Comprehend, AWS SageMaker).
2. Steps for training the N-gram Language Model and Naive Bayes Classifier with
real-world health data.
3. Ethical strategies to ensure the system avoids bias, protects sensitive health
information, and minimizes harm to credible public health authorities.
(Support your answer with practical cloud-based AI/ML strategies, data handling techniques,
and ethical considerations.)
(9 Marks)

Solution:

1. Architecture Design:
o Data Ingestion: Amazon Kinesis to stream live health content data.
o Data Storage: Amazon S3 to manage text data and labeled training sets.
o Model Training: AWS SageMaker for N-gram and Naive Bayes model
development.
o Text Analysis: Amazon Comprehend for language detection and sentiment
analysis.
2. Training Steps:
o Preprocess text data using tokenization, stemming, and stopword removal.
o Train the N-gram Language Model using Kneser-Ney smoothing for improved
probability estimates.
o Implement the Naive Bayes Classifier to predict text credibility with features
like word frequency and context.
3. Ethical Strategies:
o Minimize bias by ensuring the training data includes diverse language sources.
o Enhance data privacy by encrypting sensitive health-related text.
o Implement manual review processes to flag questionable content, reducing
unfair censorship risks.