SPAM EMAIL

DETECTION
METHODS

By: Amran Qasim

CATHOLIC UNIVERSITY IN ERBIL

INTRODUCTION
SPAM emails are unsolicited and unwanted messages sent in bulk to a large
number of recipients. These messages often contain advertisements, phishing
links, and other types of malicious content. In recent years, the volume of
SPAM email has increased significantly, and various techniques have been
developed to detect and filter out these messages. In this report, we will
discuss the various methods used for SPAM email detection.

1- PREPROCESSING

The first step in SPAM email detection is preprocessing. In this step, the emails
are preprocessed to extract relevant features. This includes removing stop words,
stemming, and converting the text to lowercase. The preprocessed data is then
used for further analysis.

SPAM EMAIL DETECTION METHODS PAGE 1

1.1 Removing stop words
Stop words are common words such as "the", "and", "or", "in", and "of" that appear
frequently in natural language and do not carry much meaning on their own.
When processing text data for SPAM email detection, it is common to remove
stop words from the message content to reduce noise and improve the
performance of the detection system.
Removing stop words involves creating a list of common stop words and then
removing them from the text data before further processing. The list of stop
words can be language-specific, as different languages have different sets of
common words that may need to be removed.
There are various libraries and tools available for removing stop words, such as
NLTK (Natural Language Toolkit) and spaCy in Python. These libraries provide a
pre-defined set of stop words for different languages, as well as the ability to
add custom stop words to the list.
By removing stop words, the remaining words in the text data carry more weight
and are more useful for determining the nature of the message, such as whether
it is spam or not

2 RULE-BASED METHODS
Rule-based methods are the most commonly used methods for SPAM email
detection. These methods use a set of rules to identify SPAM emails. For
example, a rule may be defined to flag emails that contain certain keywords,
such as "buy now" or "click here." These rules can be based on content, header
information, or a combination of both.
2.1 Examples of rule-based methods
Stemming is the process of reducing inflected (or derived) words to their base or
root form. For example, the words "running", "runner", and "runs" are all
variations of the base word "run", and stemming would reduce all three words to
"run".
Stemming is a common preprocessing step used in SPAM email detection, as it
can help reduce the number of unique words in the text data and improve the
efficiency of the detection system. By reducing different forms of the same word
to a single base form, stemming can also improve the accuracy of the detection
system by treating them as the same word.

SPAM EMAIL DETECTION METHODS PAGE 2

2.2 Advantages and disadvantages of rule-based methods

Advantages:
Rule-based methods can be very accurate if the rules are well-designed and
comprehensive.
They can be customized for specific domains or types of messages, such as
messages in a particular language or messages related to a specific topic.
Rule-based methods are often transparent, as the rules can be easily
examined and understood by humans.
They can be efficient in terms of processing time, as they do not require
training on large datasets.

Disadvantages:
Rule-based methods may not be able to handle novel or unknown types of
SPAM, as the rules are predefined and may not be able to adapt to new
patterns or techniques used by spammers.
They may also generate false positives or false negatives if the rules are too
strict or too lenient, respectively.
Creating and maintaining a comprehensive set of rules can be time-
consuming and require expertise in the domain of SPAM email detection.
Rule-based methods may not generalize well across different types of
messages or languages, as the rules are designed for specific scenarios and
may not be applicable to other contexts.

3 MACHINE LEARNING METHODS

Machine learning methods are also used for SPAM email detection. These
methods use algorithms to learn patterns in the data and identify SPAM emails.
These algorithms include Naive Bayes, Decision Trees, Random Forests, and
Support Vector Machines (SVM). Machine learning methods can also be
combined with rule-based methods to improve accuracy.

SPAM EMAIL DETECTION METHODS PAGE 3

3.1 Examples of machine learning methods

There are several machine learning methods that can be used for SPAM email
detection. Here are some examples:
1. Naive Bayes Classifier: This is a probabilistic algorithm that uses Bayes'
theorem to predict the probability of a message being SPAM or not based on
the occurrence of certain words or features in the message. Naive Bayes
classifiers are relatively simple to implement and can be trained on large
datasets, making them a popular choice for SPAM email detection.
2. Support Vector Machines (SVMs): SVMs are a type of supervised learning
algorithm that can be used for classification tasks. SVMs aim to find the
optimal hyperplane that separates the SPAM and non-SPAM messages in the
feature space. SVMs have been shown to perform well on a variety of
classification tasks, including SPAM email detection.

4 CONTENT-BASED FILTERING

Content-based filtering is another method used for SPAM email detection. This
method uses the content of the email to identify whether it is a SPAM email or
not. It involves analyzing the text, HTML, images, and other features of the email
to identify SPAM characteristics.

4.1 Examples of content-based filteringing

1. Keyword filtering: This method involves checking the message for specific
keywords or phrases that are commonly associated with SPAM, such as "buy
now", "free offer", or "limited time only". If the message contains a high
number of these keywords, it is likely to be classified as SPAM.
2. Bayesian filtering: This is a statistical method that uses Bayes' theorem to
calculate the probability of a message being SPAM based on the occurrence
of certain words or phrases. The algorithm is trained on a dataset of known
SPAM and non-SPAM messages to learn which words and phrases are most
indicative of SPAM.

SPAM EMAIL DETECTION METHODS PAGE 4

5 HEADER-BASED FILTERING

Header-based filtering is a method that uses the header information of an email

to identify SPAM emails. This includes analyzing the sender's address, IP address,
and domain name to determine if they are trustworthy. This method is less
accurate than content-based filtering but can be effective when combined with
other methods.

5.1 Examples of Header-based filteringing

1. Sender policy framework (SPF): SPF is a technique that checks the domain
name of the sender against a list of authorized IP addresses for that domain.
If the domain name does not match any of the authorized IP addresses, the
message is likely to be classified as SPAM.
2. DomainKeys Identified Mail (DKIM): DKIM is a method of email
authentication that adds a digital signature to the message header. This
signature can be verified by the recipient's mail server to ensure that the
message was sent by an authorized sender and has not been tampered with.
3. Reverse DNS lookup: This method involves checking the domain name of the
sender's IP address against a list of known SPAM domains. If the domain
name matches a known SPAM domain, the message is likely to be classified
as SPAM.
4. Blacklist filtering: Blacklists are lists of known SPAM senders or domains
that have been identified by previous filtering methods. Messages from these
senders or domains can be automatically blocked or flagged as SPAM.
5. Whitelist filtering: Whitelists are lists of approved senders or domains that
are allowed to bypass filtering methods. Messages from these senders or
domains are automatically accepted and not flagged as SPAM.

SPAM EMAIL DETECTION METHODS PAGE 5

5.2 Advantages and disadvantages of header-based filtering

Advantages:
1. Efficiency: Header-based filtering is relatively fast and can be performed
without accessing the content of the email message. This makes it more
efficient than content-based filtering methods, which require more
computational resources.
2. Accuracy: Header-based filtering can be very accurate if the right methods
are used. For example, SPF and DKIM are highly effective methods of
detecting SPAM.
3. Easy to implement: Header-based filtering is relatively easy to implement
and does not require significant changes to the existing email server
infrastructure.
4. Scalability: Header-based filtering can be easily scaled to handle large
volumes of email traffic without significant impact on system performance.

Disadvantages:
1. Limited effectiveness: Header-based filtering methods may not be effective
against certain types of SPAM, such as those that use sophisticated social
engineering techniques or those that use compromised accounts to send
messages.
2. False positives: Header-based filtering methods may sometimes incorrectly
classify legitimate email messages as SPAM, leading to false positives.
3. Reliance on external factors: Header-based filtering methods such as SPF
and DKIM rely on external factors such as the sender's domain name and the
availability of cryptographic keys. Any issues with these external factors can
lead to false positives or false negatives.
4. Difficulty in configuration: Configuring header-based filtering methods can
sometimes be complex and require a good understanding of email server
protocols and configurations.

SPAM EMAIL DETECTION METHODS PAGE 6

6 BAYESIAN FILTERING

Bayesian filtering is a statistical method used for SPAM email detection. It

involves analyzing the probability of an email being SPAM based on its content.
Bayesian filtering is an adaptive method that learns from the user's behavior and
can improve over time.

6.1 Examples of Bayesian filtering:

1. SpamAssassin: SpamAssassin is an open-source software that uses Bayesian
filtering to detect SPAM. It examines the content of the email message and
assigns a score to each message based on a set of predefined rules. If the
score exceeds a certain threshold, the message is classified as SPAM.
2. POPFile: POPFile is a personal email classifier that uses Bayesian filtering to
classify emails into different categories such as SPAM, work, personal, etc. It
uses an algorithm called the Naive Bayes classifier to calculate the
probability of an email being SPAM based on its content.
3. CRM114: CRM114 is a filtering software that uses a number of machine
learning algorithms, including Bayesian filtering, to classify emails into
different categories. It uses a technique called Regular Expression Matching
and Markovian discrimination to classify email messages.

SPAM EMAIL DETECTION METHODS PAGE 7

 CONCLUSION

In conclusion, spam email detection is an important issue in the digital age.

Various methods exist for detecting spam emails, including rule-based
methods, machine learning methods, content-based filtering, header-based
filtering, and Bayesian filtering. Each method has its advantages and
disadvantages, and the choice of method depends on the specific needs of
the user. Overall, Bayesian filtering is a popular choice for spam email
detection due to its adaptability, accuracy, and efficiency. However, it can
still result in false positives and requires initial training. With the continued
advancement of technology and machine learning, we can expect to see
even more advanced methods for spam email detection in the future.

SPAM EMAIL DETECTION METHODS PAGE 8

 REFRENCES

1. Kumar, A., & Sharma, A. (2012). A review of spam filtering techniques. International
Journal of Computer Applications, 55(1), 25-32.

2. Yang, X., Zhang, Y., & Song, X. (2018). An improved spam email detection method
based on Naive Bayes. Journal of Ambient Intelligence and Humanized Computing, 9(6),
2151-2158.

3. Androutsopoulos, I., Koutsias, J., Chandrinos, K. V., Paliouras, G., & Spyropoulos, C. D.
(2000). An evaluation of naive Bayesian anti-spam filtering. In Proceedings of the
workshop on machine learning in the new information age (pp. 9-17).

4. Raza, A., & Kastrati, Z. (2019). Analysis of spam filtering techniques using machine
learning algorithms. International Journal of Computer Science and Information
Security, 17(10), 36-43.

5. Al-Jarrah, O. Y., Abu-Ain, A. A., & Al-Hourani, M. (2019). A review of spam email
filtering techniques. International Journal of Advanced Computer Science and
Applications, 10(9), 357-365.

6. Kaur, P., & Goyal, M. (2016). Spam email detection using machine learning
techniques: A review. International Journal of Computer Science and Mobile
Computing, 5(6), 37-44.

9. Karim, A., Hossain, M. S., & Akter, F. (2019). A machine learning-based spam email
filtering approach using feature selection. International Journal of Computational
Intelligence and Applications, 18(02), 1950009.

10. Sahami, M., Dumais, S., Heckerman, D., & Horvitz, E. (1998). A bayesian approach to
filtering junk e-mail. AAAI/IAAI, 98, 55-62.

11. Díaz-Uriarte, R., & Alvarez de Andrés, S. (2006). Gene selection and classification of
microarray data using random forest. BMC bioinformatics, 7(1), 1-13.

SPAM EMAIL DETECTION METHODS PAGE 9