Malware Detection using

Machine Learning Models

1st Chaitanya Nerkar, 2nd Tejas varpe 3rd Pratik Patil, 4th Karan Vadnere
Computer Engineering Computer Engineering
Pimpri Chinchwad College of Engineering and Research Pimpri Chinchwad College of Engineering and Research
Pune, India Pune, India
[email protected], pratik.patilc [email protected],
tejas.varpec [email protected] Karan.Vadnerec [email protected]

Abstract—The detection of malware in any kind of file is very computing resources centered around the examination of new
important in today’s day to day life. New kind of malware which and possibly dangerous files. Thus, this machine learning-
are highly powerful and efficient in exploiting the contents of based algorithms’ combination with feature extraction and
the files or affecting the system with the help of residing in
files and data..Malware is threat, highly lethal to the users if system optimization results in a robust solution to secure
it goes undetected or unnoticed. These kinds of malware which systems from emerging malware threats
are under development are different from the tradi- tional one,
and more dynamic in design and usually inherit the properties II. L ITERATURE SURVEY
from two or more than two malware types, and they are known A. A survey of malware detection using deep learning:
as polymorphic malware.. By using various model, the system
not only enhances performance and scalability but also provides Ahmed Bensaoud, Jugal Kalita, Mahmoud Bensaoud [2]
comprehensive security coverage that adapts to emerging threats. There is a comprehensive review of the lat- est developments
Hybrid approach ultimately aims to fortify the security of data in the application of deep learning approaches for malware
and system against increasingly sophisticated cyber threats. detection on various operating systems including Windows,
Index Terms—Malware detection, Machine learning, medium
sized dataset, decision tree, random forest. MacOS, Linux, Android, and iOS. It is talking about the classi-
fying accuracy capability of the CNNs and GANs models over
I. I NTRODUCTION the malware samples from text as well as image data. However,
One of the most promising approaches for file automatic the authors further discuss the application of multi-task learn-
classification to be either benign or malicious would be mal- ing and transfer learning models for high precision in the
ware detection by machine learning. This is done by training detection of malware. The authors, however identify several
the models on a large corpus of labeled files which include issues such as: lack of standard bench- marks; inability of deep
clean files and many other types of malware. Some of these learning decisions to be intuitively explained thus the need for
include key elements such as meta- data from the file, system Explainable Artificial Intelligence or XAI, and susceptibility
calls, entropy, byte sequences, and behaviors extracted from of these models to adversarial attacks, which compromise their
these files to train models. These models-think decision trees, generalization capa- bility. They stress that future research
Random For- est, Support Vector Machines, SVM, and deep should emphasize training and testing existing models on
learning algorithms such as neural networks-learn to recognize various malware datasets. The work in the future should focus
patterns and anomalies related to malicious files. Once trained, more on improving models for robustness, interpretability, and
the model can classify new, unseen files in real-time, detecting perfor- mance over unseen data. The present study is all-
previously unknown malware based on learned behaviors and inclusive information on the strengths and drawbacks in deep
features. Finally for increasing the accuracy of the model, learning-based malware detection. [1]
some other techniques like feature selection, dimension- ality B. Improving the Machine Learning Models for Malware
reduction, and ensemble learning are used, which completely Detection Using the Embedded Feature Selection Method:
refine the predic- tion. Furthermore, in optimization methods,
Mohammed CHEMMAKHA, Omar HABIBI, Mohamed
the database also arranges similar files so that storage capacity
LAZAAR[4] This paper discusses the issue of improving
gets increased and response time of the model increases. This
machine learning models in malware detection using the
way, the system identifies malware faster and with higher
embedded feature selection method, at a time when proper pre-
accuracy since re- dundancy in the system is minimized and
processing in machine learning applied to malware detection is
of crucial importance. The authors consider that data cleaning
and cleansing with feature extraction would be the crucial locate malware with high accuracy and low false positive
steps to- ward improving performance for machine learning rates. Additionally, it encourages continuous study on machine
models in malware detec- tion applications. The paper focuses learning-based solutions regarding cybersecu- rity. [3] [4] [5]
on an embedded technique for the se- lection of interesting
features supporting the improvement of efficiency and accu- D. ”Personalized Course Recommendation Based on Con-
racy in the malware detection application, as it optimizes the tent” by N. Zheng et al. (2016) :
interesting features’ choice, reducing the complexity of the III. METHODOLOGY
model without impacting its performance. This work provides
A. Data Collection and Preprocessing
insights into how feature selection might re- duce the curse of
dimensionality as well as improve machine learning systems The initial step in any machine learning project is collecting
used in cybersecurity capabilities in detection. a large, diverse dataset. For malware detection, the dataset
must consist of both malicious (malware) and benign (non-
C. Malware Detection with Machine Learning: malware) files, which can come from public repositories like
Kunal Bhat, Tejas Khairnar, Sharayu Phatangare3, Tanmay VirusTotal, Kaggle, or other malware databases. It is crucial
Narkhedkar4[8] proposed the method of de- PCCOER Ravet, that the dataset is diverse and covers a range of malware types
Department of Computer Engineering 2024 6 tecting malware to ensure the model generalizes well to various threats. After
using machine learning with hardware-assisted techniques as data collection, the preprocessing phase involves preparing the
an effective method. Since traditional techniques for malware data for the model.
detection, in- cluding static signatures, are vulnerable to obfus-
cation, as referred by the au- thors, hardware-assisted solutions B. Feature Extraction
in the form of monitoring and categorizing memory access Feature extraction plays a critical role in malware detection
patterns to reduce dependent malware signatures have been as it helps identify the key characteristics that differentiate
proposed. The work also covers feature extraction from PE malicious files from benign ones. This process can be divided
files and applica- tion of machine learning algorithms such as into multiple types of analysis, such as static analysis, dynamic
Random Forest, AdaBoost, Gra- dient Descent, and Decision analysis, and behavioral analysis. Static analysis involves ex-
Trees that can be utilized to increase the accuracy levels in tracting features from the file itself, such as byte sequences, file
detection. It includes a two-level classification architecture in- size, file extensions, and metadata, without executing the file.
side the framework for improved malware detection for known Dynamic analysis, on the other hand, involves executing the
and unknown malware. This research is behavior-based, with file in a controlled environment (sandbox) and monitoring its
an aim to have resilience against mal- ware variants and behavior, such as system calls, network activity, and resource
obfuscation; thus, it focused on the hybrid approaches of consumption.
detection being a mixture of static and dynamic analysis.
Authors proposed a design and architecture for the system in C. Model Selection and Training
efforts to make malware detection more automated, flexible, In the model selection phase, several machine learning
and robust by capturing both static features and dy- namic algorithms are evaluated to find the best model for the
features of malware. [2] given malware detection task. Commonly used models for
, M. Dubovitskaya, and G. Neven, [10] suggest oblivious this task include Support Vector Machines (SVM), Random
transfer with access control. The authors design the scheme Forest, LightGBM, and Neural Networks. SVM is a powerful
for anonymous access to a database where the various entries algorithm for classification tasks, where the goal is to find
have different access control permissions. Access control will the hyperplane that best separates the classes of data (mal-
comprise of attributes, roles, or other rights that the user has ware vs. benign). It is effective for high-dimensional spaces,
to possess to access the entry. Our protocol realizes maximal making it suitable for malware detection, where there are
security guarantees for the database and for the user: (1) only numerous features. Random Forest is an ensemble learning
authorized users can access the record; the database provider algorithm that builds multiple decision trees and combines
does not learn which record the user accesses; the database their outputs to produce more accurate predictions. It handles
provider does not learn which attributes or roles the user has large datasets well and is resistant to overfitting. LightGBM,
when she accesses the database. They show that this protocol a gradient boosting framework, is another ensemble method
is secure in the standard model (i.e., without random oracles) that is optimized for speed and accuracy. It works by building
under the bilinear Diffie- Hellman exponent and the strong a series of trees in a sequential manner where each tree
Diffie-Hellman assumptions. tries to correct the errors made by the previous one. Neural
Shoaib Akhtar andTao Feng [1] posed a protective mecha- Networks, particularly deep learning models, can be used for
nism, which evaluated three ML algorithms towards malware more complex patterns in data, and CNNs can be applied to
detection, and selected the best one among the three. The recognize specific patterns in the sequence of bytes or system
model compares this to select the most appropriate model. calls within malware. Choosing the right algorithm depends
It may be noted from the paper that machine learning al- on factors like accuracy, computational resources, and how
gorithms, especially DT, CNN, and SVM, can very efficiently the model needs to generalize to unseen malware.
D. Model Evaluation payload is non-executable, it continues with other phases
TOnce the models are trained, they need to be evaluated on of analysis.
unseen data to assess their performance. Common evaluation • Static Code Analysis: This involves examining the
metrics for malware detection include accuracy, precision, code without executing it to look for known malware
recall, F1-score, and the AUC-ROC curve. Accuracy measures signatures, malicious patterns, or suspicious constructs in
the proportion of correct predictions, but in an imbalanced the code.
dataset like malware detection, other metrics become more • Behavioral Pattern Matching: If static analysis does
important. Precision is the percentage of predicted malware not clearly provide the findings, then behavioral pattern
files that are actually malware, and recall is the percentage of matching is applied. Behavioral pattern matching involves
actual malware files that are correctly identified. F1-score is the analysis of behaviors when run against one another
the harmonic mean of precision and recall, providing a single in order to detect abnormal activities or deviations from
measure that balances the two. The AUC-ROC curve helps expected ones.
in evaluating how well the model distinguishes between the • Dynamic Behavior Analysis: Isolate the executable
classes (malware and benign). A higher AUC indicates better payload and then test it in a controlled sandbox. The
performance. Evaluating the model on these metrics helps dynamic behavior might include changing a file, making
ensure that the model not only performs well but also avoids network connections, or even trying to elevate privileges.
common pitfalls like false positives (benign files labeled as • Support Vector Classifier Training:Machine Learning
malware) and false negatives (malware files labeled as benign). applies here using SVC or Support Vector Classifier by
training a perfect classifier on labeled samples that makes
E. Model Optimization it differentiate file between benign and malicious by
After training, the next step is to fine-tune the model extracting salient features from data. This involves the
by adjusting its hyperparameters. These parameters, such as use of efficient feature extraction algorithms that help
the learning rate, batch size, and number of trees in Ran- extract critical features that promote the performance of
dom Forest or LightGBM, significantly influence the model’s the classifier.
performance. Hyperparameter tuning is typically done using • Model Evaluation: K-fold Cross-validation: We validate
techniques like Grid Search or Random Search, where the the classifier by using K-hold cross validation for accu-
model is trained with different sets of hyperparameters to racy. FNR and FPR: The model evaluation will be based
find the optimal configuration. The goal is to improve the on a balance between low FPRs (with a minimal number
model’s performance on the test set and prevent overfitting. of false positives, i.e., benign files wrongly classed as
For example, in the case of SVM, the choice of kernel and malware) and a minimal number of FN classes, where
regularization parameters can drastically affect its ability to actual malwares were not detected.
separate malware from benign files. Similarly, tuning the depth
of trees in Random Forest or adjusting the learning rate in B. URLs Phishing Detection
LightGBM can help improve accuracy and generalization.
The detection of malware of URLs (Uniform resources
IV. P ROPOSED W ORK Locator) will be perform in this stage.
Extract URL Features: Features of the URLs pertinent to the
A. Malware in Data and Files
analysis, such as the length of the URL and the characteristics
It mentions a process of identifying, analyzing, and miti- of the domain.
gating malware by combining techniques of both static and
dynamic analysis
• Threat Analysis - Malware Detection: The process
begins with an initial threat analysis, where malware
detection is attempted using various techniques to deter-
mine if the analyzed data contains a potentially malicious
payload.
• Classify Attacks Based on Header and Footer In-
formation: To classify attacks one classifies based on
header and footer information. In terms of static analysis,
analysis of the header and footer fields in files will
categorize attacks. It doesn’t execute the file but only
checks for known malicious signatures or patterns.
• Payload Analysis: If the file is suspected to carry mali-
cious content, the flowchart checks whether the payload
is executable: Yes: Once it’s executable, isolate and copy Fig. 1. System Architecture Diagram
the payload into a sandbox for further analysis. Not: If the
 VII. F UTURE S COPE
A. Potential Extensions
• Integration with Real-Time Systems: Further devel-
opment can focus on integrating the malware detection
model into real-time security systems to monitor and
detect threats in live environments.
• Exploring New Algorithms:Researching new machine
learning or deep learning algorithms to improve detection
accuracy, speed, and resource efficiency.
• Multi-Platform Expansion: Develop web and iOS ver-
sions to increase accessibility.
• Collaboration with Cloud Services: Implementing the
malware detection system into cloud platforms to provide
scalable security solutions for enterprise-level applica-
tions.
ACKNOWLEDGMENT
I would like to express my sincere gratitude to all those who
have contributed to the success of this project. My heartfelt
thanks to my mentors and colleagues for their invaluable
guidance, support, and expertise throughout the development
of this malware detection model. I also extend my appreciation
to the creators of the datasets and the developers of the
Fig. 2. Flowchart
machine learning tools, whose contributions were crucial to
the success of this project. Finally, I would like to thank my
V. C ONCLUSION family and friends for their continuous encouragement and
understanding during the course of this work.
A much greater step in cybersecurity, this malware detection
model built on machine learning is designed to detect and R EFERENCES
prevent malware attacks automatically with scalable solutions.
[1] A. Kolosnjaji, T. G. D. Y., and P. Y. M, “Deep learning for classifying
Unlike traditional signature-based methods, which depend on malware,” Computers Security, vol. 81, 2016.
known malware patterns, this model incorporates machine [2] Z. Zhang, Y. Chen, and M. Zhao, “Real-time malware detection using
learning algorithms to draw insights from system behaviors, cnns,” Journal of Computer Virology and Hacking Techniques, vol. 16,
2020.
file attributes, and network activities. Thus, it can detect both [3] U. P. Rao, M. Kumar, and R. Singh, “Deep learning for malware
known and previously unseen malware, making it all the detection,” IEEE Transactions on Information Forensics and Security,
more effective against emerging threats. Such advancements vol. 13, 2018.
[4] M. S. Akhtar and T. Feng, “Malware analysis and detection using machine
in techniques like static and dynamic analysis by the model learning algorithms,” International Journal of Computer Applications,
identify not just existing threats but predict risk cases based vol. 113, 2022.
on suspicious patterns observed in data. [5] R. Vinayakumar, K. S. R. Ananthapadmanabha, and R. K. K, “Compara-
tive study of machine learning algorithms for malware detection,” Journal
VI. C HALLENGES AND L IMITATIONS of Computer Virology and Hacking Techniques, vol. 15, 2019.

A. Challenges
• Scalability: Managing real-time updates with a growing
user base is a technical challenge that needs future
optimization.
• Uarge Dataset Requirement: Collecting and maintain-
ing large, diverse datasets for training the malware detec-
tion model is time-consuming and costly.
B. Limitations
• Evolving Malware Techniques: Malware evolves con-
stantly, making it difficult to detect new, previously
unseen threats with traditional methods.
• Real-Time Detection Challenges: Achieving real-time
malware detection with high accuracy while minimizing
computational overhead can be difficult.