Text Classification for Social Media Posts

College Code & Name 3135 - Panimalar Engineering College Chennai City Campus
Subject Code & Name NM1090 - Natural Language Processing (NLP) Techniques
Year and Semester III Year - VI Semester
Project Team ID

Project Created by 1.

2.

3.

4.

1
 BONAFIDE CERTIFICATE

Certified that this Naan Mudhalvan project report “Text

Classification for Social Media Posts” is the bonafide work of

__________________ who carried out the project work under my

supervision.

SIGNATURE SIGNATURE
Project Coordinator SPoC
Naan Mudhalvan Naan Mudhalvan

INTERNAL EXAMINER EXTERNAL EXAMINER

2
 ABSTRACT

The exponential growth of social media platforms has led to an overwhelming

amount of textual data being generated daily. Social media users express opinions,
share experiences, and engage in discussions that create a vast repository of textual
data. Analysing this data provides valuable insights into user sentiments, trends, and
emerging topics. Businesses, policymakers, and researchers leverage social media
analysis to understand customer behaviour, brand perception, and societal issues.

This project focuses on implementing text analysis techniques for social media
posts using Natural Language Processing (NLP). NLP enables machines to process and
analyse human language, making it possible to extract meaningful patterns from
unstructured text data.

This report details the project’s objectives, technology stack, implementation

methodology, sample outputs, and conclusions drawn from the results. The findings
demonstrate the effectiveness of NLP in analysing social media data, offering
significant applications in business intelligence, marketing, and public opinion
analysis. Future enhancements can include real-time sentiment tracking, multilingual
support, and deep learning-based improvements for better accuracy

3
 TABLE OF CONTENT

CHAPTER NO TITLE PAGE NO

ABSTRACT 3

1 INTRODUCTION 5

2 TECHNOLOGIES USED 7

3 PROJECT IMPLEMENTATION 9

4 CODING 11

5 TESTING AND OPTIMIZATION 13

6 SAMPLE OUTPUT 17

7 CONCLUSION 18

REFERENCES 19

4
 CHAPTER 1
INTRODUCTION

Here individuals and organizations express opinions, share information, and

engage in discussions. With millions of posts generated every day on platforms like
Twitter, Facebook, and Instagram, analysing this vast amount of text has become an
essential task for businesses, researchers, and policymakers. The ability to extract
insights from social media posts provides an opportunity to understand public
sentiment, detect emerging trends, and gain valuable business intelligence.

Text analysis, a branch of Natural Language Processing (NLP), enables computers

to interpret, process, and analyse textual data automatically. Through techniques such
as sentiment analysis, topic modelling, and keyword extraction, organizations can
categorize content, assess user emotions, and identify key themes in discussions. These
techniques help in diverse applications, including market research, customer feedback
analysis, brand monitoring, and even political sentiment tracking.

This project aims to develop a text analysis system that processes and analyzes
social media posts to derive meaningful insights. By leveraging machine learning and
NLP techniques, the project focuses on:

 Sentiment Analysis: Determining whether a post conveys a positive, negative, or

neutral sentiment.

 Topic Modelling: Identifying key topics discussed in social media conversations.

 Keyword Extraction: Highlighting the most relevant and frequently mentioned

words and phrases.

The system is implemented using Python-based libraries, including NLTK, SpaCy,

Scikit-learn, TensorFlow, and Gensim. Social media data is collected using APIs, pre-

5
processed to remove noise, and analysed using various machine learning models. This
report outlines the methodologies used, the technical implementation, sample results,
and conclusions drawn from the analysis. Through this project, we aim to demonstrate
the significance of automated text analysis for extracting actionable insights from
unstructured social media data.

6
 CHAPTER 2
TECHNOLOGIES USED
Programming Languages & Libraries

 Python: Python is the primary language used for implementing this project due
to its extensive support for Natural Language Processing (NLP) and machine
learning. It provides various libraries and frameworks to simplify text analysis,
making it a preferred choice for researchers and developers.

 NLTK & SpaCy: These libraries are used for preprocessing text data. NLTK
provides a suite of text processing tools such as tokenization, stemming, and
lemmatization, while SpaCy is optimized for performance and includes pre-
trained models for named entity recognition and part-of-speech tagging.

 Scikit-Learn: This machine learning library is utilized for building classification

models. It provides efficient implementations of algorithms such as Logistic
Regression and Random Forest, which are used in sentiment analysis.

 TensorFlow/Keras: These deep learning frameworks are used to develop and

train neural networks for sentiment analysis. The Long Short-Term Memory
(LSTM) model, implemented using TensorFlow/Keras, helps capture sequential
dependencies in text data.

 Gensim: This library is used for topic modeling and word embeddings. It supports
the Latent Dirichlet Allocation (LDA) algorithm, which helps identify key topics in
social media posts.

 Pandas & NumPy: These libraries assist in data handling and preprocessing.
Pandas provides data structures like DataFrames to manipulate and clean
datasets, while NumPy is used for numerical computations.

7
  Matplotlib & Seaborn: These visualization libraries help in generating insightful
graphs and plots to represent analytical results. They are used to visualize
sentiment distribution, keyword frequencies, and topic modeling results.

Platforms & Tools

 Jupyter Notebook: This interactive computing environment is used for coding

and experimenting with different text analysis techniques. It allows easy
visualization and debugging of data at each stage of processing.

 Google Colab: This cloud-based tool provides GPU acceleration, which is

beneficial for training deep learning models. It allows seamless collaboration and
eliminates the need for local hardware resources.

 Twitter API & Tweepy: The Twitter API enables real-time data collection from
Twitter, while Tweepy is a Python library that simplifies the process of accessing
and extracting tweets. These tools help gather large datasets for analysis.

 MongoDB: This NoSQL database is used for storing and retrieving collected social
media data. Its flexible schema allows efficient handling of unstructured text
data, making it suitable for large-scale text analysis.

8
 CHAPTER 3
PROJECT IMPLEMENTATION
1. Data Collection

Social media posts are collected using the Twitter API. The collected data includes
tweets, retweets, and user interactions. The text data is then preprocessed by removing
stopwords, punctuations, and performing lemmatization to ensure clean and
meaningful input for analysis.

2. Data Preprocessing

 Tokenization: Breaking text into individual words.

 Stopword Removal: Eliminating common words that do not add meaningful

information.

 Lemmatization: Converting words to their root forms to standardize text.

 Removing URLs, Mentions & Hashtags: Cleaning unnecessary elements from

tweets to improve analysis accuracy.

3. Sentiment Analysis

 Approach: Supervised machine learning technique using a pre-labeled dataset.

 Models Used:

o Logistic Regression: A simple yet effective classification algorithm.

o Random Forest Classifier: A robust ensemble learning method.

o LSTM-based Deep Learning Model: A deep learning approach to capture

sequence dependencies in text.

9
  Evaluation Metrics:

o Accuracy: Measures the correctness of the predictions.

o Precision: Assesses the quality of positive sentiment classification.

o Recall: Determines how well the model identifies relevant instances.

o F1-score: A balance between precision and recall for optimal performance.

4. Topic Modeling

 Latent Dirichlet Allocation (LDA): Extracts key discussion topics from text data,
grouping related words to identify underlying themes.

 Word Cloud Visualization: A graphical representation of frequently occurring

keywords in different topics to help interpret data intuitively.

5. Keyword Extraction

 TF-IDF (Term Frequency - Inverse Document Frequency): Identifies significant

words by measuring their importance across multiple posts.

 RAKE (Rapid Automatic Keyword Extraction Algorithm): Extracts key phrases and
relevant terms from text automatically, enhancing trend detection.

10
 CHAPTER 4
CODING
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Sample dataset
data = {'post': ["I love this product!", "This is the worst service ever!", "Great
experience overall."],
'sentiment': ['positive', 'negative', 'positive']}
df = pd.DataFrame(data)

# Text preprocessing
def preprocess_text(text):
tokens = word_tokenize(text.lower())
tokens = [word for word in tokens if word.isalnum() and word not in
stopwords.words('english')]
return ' '.join(tokens)

df['cleaned_post'] = df['post'].apply(preprocess_text)

11
# Feature extraction
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(df['cleaned_post'])
y = df['sentiment']

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Model training
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy}')

12
 CHAPTER 5
TESTING AND OPTIMIZATION

Project testing can involve various types depending on the nature of the project
(e.g., software development, product design, or research). Here are some common
types of project testing:
1. Unit Testing
What it is: Testing individual components or units of a project (typically code).
Used for: Ensuring that each unit of the project functions as expected.
Example: Testing individual functions or methods in software development.
2. Integration Testing
What it is: Testing the interaction between different components or systems to
ensure they work together.
Used for: Ensuring that when multiple components are combined, they function as
expected.
Example: Testing how the frontend and backend communicate in a web application.
3. System Testing
What it is: Testing the complete and integrated system to verify if it meets the
specified requirements.
Used for: Ensuring that the overall system works as intended.
Example: Testing the full functionality of a software application.
4. Acceptance Testing
What it is: Testing to ensure the product meets the business requirements and is
ready for deployment.
Used for: Determining if the project is complete and ready for end users.
Example: User acceptance testing (UAT) where end-users verify the product.

13
5. Regression Testing
What it is: Testing after changes (e.g., code updates) to ensure that new code hasn't
broken existing functionality.
Used for: Ensuring new features or fixes don't affect the existing parts of the project.
Example: Re-running tests after fixing bugs in software to ensure old functionality
still works.
6. Performance Testing
What it is: Testing how the system performs under load.
Used for: Identifying performance bottlenecks and ensuring the system can handle
high volumes of traffic or data.
Example: Load testing a website to see how it performs with a high number of
concurrent users.
7. Security Testing
What it is: Testing for vulnerabilities and weaknesses in the system.
Used for: Ensuring that the project is secure and that sensitive data is protected.
Example: Penetration testing to find and fix security vulnerabilities in a software
product.
8. Usability Testing
What it is: Testing the product from an end-user perspective to ensure it is easy to
use and intuitive.
Used for: Ensuring that the product is user-friendly and provides a positive user
experience.
Example: Observing users interacting with a website and identifying usability issues.
9. Alpha Testing
What it is: Internal testing of the product to find bugs and issues before it’s released
to a select group of users.
Used for: Identifying major issues before releasing the product to beta testers.

14
 Example: Testing a new app internally within the company.
10. Beta Testing
What it is: Testing by a small group of external users before the product is officially
launched.
Used for: Getting feedback from real users in real-world environments.
Example: Allowing a group of users to test a new software version before the official
public release.
11. Stress Testing
What it is: Testing the system beyond normal operating conditions to determine its
breaking point.
Used for: Identifying how the system behaves under extreme stress or failure
conditions.
Example: Stress testing a website by simulating thousands of simultaneous users.
12. Smoke Testing
What it is: A preliminary test to check if the basic features of the project are
working.
Used for: Determining if the project is stable enough for further testing.
Example: Quickly checking if a web application loads without crashing.
13. Compatibility Testing
What it is: Testing how the system works across different platforms, devices,
browsers, or environments.
Used for: Ensuring the project functions well across various conditions and
configurations.
Example: Testing a website on multiple browsers (Chrome, Firefox, Safari).
14. Exploratory Testing
What it is: Testing without predefined test cases, often used for discovery or
uncovering unexpected issues.

15
 Used for: Investigating unknown areas of the project or testing edge cases.
Example: A tester exploring the app's interface to see if anything breaks.
15. A/B Testing
What it is: Comparing two versions of a product to determine which one performs
better with users.
Used for: Testing different versions to identify which one drives better results.
Example: Testing two variations of a website's landing page to see which version
increases user sign-ups.

16
 CHAPTER 6
SAMPLE OUTPUT

17
 CHAPTER 7
CONCLUSION
The analysis of social media posts using Natural Language Processing provides
valuable insights into sentiment trends, emerging topics, and significant keywords. This
project successfully implemented various text analysis techniques, including sentiment
classification, topic modelling, and keyword extraction, to process and interprets large
volumes of unstructured social media data.The results demonstrate the effectiveness of
machine learning and deep learning models in identifying sentiments with high
accuracy. The integration of Logistic Regression, Random Forest, and LSTM models
provided a comparative analysis, allowing for a more comprehensive evaluation.
Additionally, topic modelling using LDA highlighted prevalent themes in social media
discussions, while keyword extraction techniques such as TF-IDF and RAKE helped
identify significant terms used in posts.
This project has broad applications in business intelligence, marketing, customer
feedback analysis, and public opinion monitoring. Future enhancements could include
multilingual support, real-time sentiment tracking, and the incorporation of
transformer-based models like BERT for improved accuracy. By advancing these
capabilities, social media text analysis can become even more powerful in
understanding and predicting trends in public discourse.

18
REFERENCES
1. Bird, S., Klein, E., & Loper, E. (2009). Natural Language Processing with Python.
O'Reilly Media.
2. Jurafsky, D., & Martin, J. H. (2021). Speech and Language Processing. Pearson.
3. Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR.
4. Chollet, F. (2018). Deep Learning with Python. Manning Publications.
5. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. JMLR.

19