Frontiers of Knowledge Journal Series | International Journal of Pure and

Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

Detection of Phishing WebsitesUsing Random Forest and XGBOOST

Algorithms
Ali Ahmad Aminu1 Abdulrahman Amatullah Yahaya Muhammad Abdulkadir Maigari
Abdulkarim2 Aliyu3 Aliyu4 Turaki5
Department of Computer Computer Science Department,
Science, Gombe State Federal Polytechnich, Bauchi, Bauchi state
University, (Nigeria) (Nigeria)
[email protected] yaros.ikara@gmail aleeyuamatullah70@gma maliyudeba@gmail. abdul2rakimaigari@gmail
.com il.com com .com

Abstract
Mitigating the risk pose by phishers and other cybercriminal in the cyber space requires a
robust and automatic means of detecting phishing websites and phishing emails since the
culprits are constantly coming up with new techniques of achieving their goals. Many
approaches have been proposed in an attempt to curb the problems caused by phishers. In
this study, we tried to extend and improve on the existing methods by proposing a hybrid
technique (Random forest and Xgboost) algorithms. Random forest (RF) was used to rank
and select the most relevant features of our datasets while xgboost was used to build the
model using the selected dataset. The model was evaluated and tested with 11055 phishing
dataset from UCI repository consisting of 4898 legitimate and 6157 phishing websites
using Accuracy, Recall, Mathew Correlation Coefficient (MCC), Precision and Fscore as
performance metrics. The proposed method was compared with some state of the art
methods from the literature and results showed that the proposed method turned out to be
the most robust method in terms of the aforementioned evaluation metrics.
Keywords: Phishing websites, Random Forest, Xgboost, Algorithm, cyberspace

1. INTRODUCTION
Phishing is a cyber crime in which cyber criminals attempt to obtain sensitive
information of cyber users such as username, passwords credit card details often for
malicious intent by disguising as a trustworthy entity in an electronic communication.
(Toolan and Carthy, 2019).The information gained by phishers are often used to access
users important accounts (facebook, twitter, email and bank) which may result in identity
theft and financial losses. (Gupta, Tewari, Jain and Agrawal, 2017). The word phishing was
first coined in 1996 as a form of online identity theft after an attack by hackers on
Open Access Journal www.smrpi.com 1
 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

AmericaOnline account. (Khonji, Iraqi and Jones, 2013). and the first phishing lawsuit was
filed in 2004 against a California teenager who created an imitation of the website
“AmericaOnline” to gain access to user sensitive information including credit card details
causing them huge financial lost. Phishers operate by sending fake emails to their victims
pretending to be from legitimate and well known organizations such as banks, university,
communication network etc. The email contents will insist/deceive the victims to follow a
link/URL to fake website where they will require to update some personal information
including their passwords and usernames to avoid losing access right to some of the
services provided by that organization. Phishers use this avenue to obtained users sensitive
information which they in turn use it to access their important accounts resulting in identity
theft and financial loss. (Adelhamid, Ayesh and Thatah, 2014).Mitigating the risk pose by
phishers and other cybercriminal in the cyber space requires a robust and automatic means
of detecting phishing websites and phishing emails since the culprits are constantly coming
up with new techniques of achieving their goals.Many approaches have been proposed in
an attempt to curb the problems caused by phishers (Abu-Nimeh, Nappa, Wang and Nair,
2017)-(El-Alfy, 2017). However, due to the dynamic nature of attackers and the
challenging nature of the problem, it still lacks a complete solution. Recently, machine
learning approaches have been found to be very successful in the automated detection of
phishing wed sites. This paper builds/extends on this by using Xgboost, an optimized
implementation of gradient boosted decision tree algorithm and Random Forest (RF)
algorithm to improve the performance that a predictive model can achieve in the detection
of a phishing website from a legitimate website.

2. PROBLEM STATEMENT
Advancement in technology has made the cyberspace an avenue for banking,
shopping, education, and entertainment. However, as most of human activities are being
moved to the cyberspace, phishers and other cybercriminals are making the cyberspace
unsafe by posing serious risks to users and businesses as well as threating global security
and the economy. (Gupta, Tewari,Jain and Agrawal, 2017). A cybercrime in which an
attacker attempts to obtain or learn sensitive information such as usernames, passwords and
credit card information often for malicious intent by masquerading as a trustworthy entity
in an electronic communication in known as phishing. Today, phishers are constantly

Open Access Journal www.smrpi.com 2

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

evolving the techniques they used for luring user to revealing their sensitive information.
They use these information to access important accounts of their victims resulting in
identity theft, denial of services, financial losses and sabotage of reputations (Adelhamid,
Ayesh and Thatah, 2014). Many techniques have been proposed in the past for phishing
website detection, however, due to the dynamic and challenging nature of the problem, the
problem still lack complete solution. Consequently, this work tries to improve the
performance that a predictive model can achieve in the task of phishing website detection
by integrating RF and Xgboost algorithms.

3. RELATED STUDIES
Many studies have been proposed to mitigate the risk caused by phishers and other
cybercriminal in the cyber space. Few of these studies are presented below:
Davut and Mustapha (Zouina and Outtaj, 2017) proposed an intelligent phishing websites
detection model based on extreme learning machine. They tested the proposed model using
a dataset having 30 input features and 1 output feature. 10 fold cross validation was used
for splitting the datasets into training and testing sets. The proposed model obtained an
average classification accuracy of 95.05%.
In (Abu-Nimeh, Nappa, Wang and Nair, 2017),Saed et al evaluated the performance
of several Machine Learning algorithm in the detection of phishing emails including
logistic regression, Classification and Regression Tree (CART), Support Vector Machine
(SVM), neural network and Random Forest using a datasets consisting of 2889 legitimate
and phishing emails. They also used 10 – fold cross validation in splitting the datasets for
training and testing. Results of their experiments revealed that Random forest turned out to
have the best performance when legitimate and phishing emails are equal with an error rate
of 07.07%.
Mouad and Banceur in (Khaytan and Handay, 2017) proposed a light weight
phishing detection system using SVM and similarity index. They tested the performance of
the proposed method using 2000 phishing datasets consisting of 1000 legitimate and
phishing websites using only six features. The six features included a similarity index
which is a new feature proposed by the authors. Their results revealed that the new feature
introduced (similarity index) improves the overall detection rate by 21%.

Open Access Journal www.smrpi.com 3

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

Although, many approaches have been proposed in an attempt to curb the problems
caused by phishers, however, due to the dynamic and challenging nature of the problem, the
problem still lack complete solution. Consequently, this work tries to improve the
performance that a predictive model can achieve in the task of phishing website detection
by integrating RF and Xgboost algorithms.

4. METHODOLOGY
4.1 Data Gathering and Description
Most researchers in phishing detection make use of datasets constructed by
themselves. However, with such type of datasets, it is difficult to evaluate and compare the
performance of a model with other models from the literature since the datasets they are
using are not publicly available for other to use and confirm their results, therefore such
results cannot be generalized (El-Alfy, 2017).
In order to assess and compare the predictive performance of the proposed model,
we adopted a recently created phishing dataset from UCI machine learning repository. This
dataset was created by Mohammmed, Thabtah and McChushy at the university of
Huddesfied, united Kingdom. (Mohammad, Thabtah abd McChuskey, 2014). The dataset
has a total of 11055 websites instances preclassified as legitimate (non phishing) and
phishing websites with 30 features. 4898 of the dataset are legitimate while the remaining
6157 are phishing. The description of the adopted features of the dataset is presented in the
table below.
Table 1:Adopted Features of Dataset
s/n Features Feature Notation Value range
1 Having IP Address Has_ip {-1, 1}
2 URL Length url_length {-1,0, 1}
3 Using URL Shortening Short_service {-1, 1}
Service
4 URL having the @ Has_@_symbol {-1,1}
symbol
5 URL has redirect symbol Double_slash_redirect {-1,1}
6 Prefix or suffix to domain Pref_suf {-1,1}
7 Having subdomains Has_subdomain {-1,1}

Open Access Journal www.smrpi.com 4

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

8 Using HTTPS with Ssl_state {-1,1}

trusted certificate
9 Domain registration Long_domain {-1,1}
length
10 Favicon Favicon {-1, 1}
11 Use of non standard port Nonst_port {-1, 1}
12 HTTPS is Domain part https_token {-1, 1}
13 External object URL External_request {-1, 1}
14 Anchor URL refer to Anchor_url {-1, 0, 1}
another domain
15 Links in meta, script, link Links_tag {-1, 0, 1}
tags
16 Server from handler SFH {-1, 1}
17 Submiting to email Submit_email {-1, 1}
18 Abnormal URL Abnormal_url {-1, 1}
19 Website forwarding Redirect {0, 1}
20 Status bar customization Mouseover {-1, 1}
21 Disabling right click Right_click {-1, 1}
22 Use of pop up window Popup {-1, 1}
23 Iframe redirect Iframe {-1, 1}
24 Domain age Domain_age {-1, 1}
25 DNS record Dns_record {-1, 1}
26 Website traffic Website_traffic {-1, 0, 1}
27 Page rank value Page_rank {-1, 1}
28 Google indexed Google index {-1, 1}
29 Links pointing to Links_to_page {-1, 1}
websites
30 Statistical Report Stats_report {-1, 1}
Class Result Result {-1, 1}

Open Access Journal www.smrpi.com 5

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

4.2 Feature Ranking and Selection using Random Forest Algorithm

Data features used to train machine learning models have great influence on the
performance of the models. Noisy features or features irelavant to the underlying
relationship may adversely affect the perfomance of a model. Feature selection is the
process of automatically selecting those features in your data that can contribute most to the
performance of your model. Many techniuqes (such as principal component analysis,
recursive feature elimination and univariate selection ) can be used for feature selection.
This work employed RF algorithms for feature ranking and selection. Random forests is an
ensemble machine learning method, which works by constructing a multitude of decision
trees at training time and outputting the class that is averaged or voted by every individual
tree.(Zhang, Qian, Mao, Huang, Huang and Si, 2018). RF was proposed by Breiman in
2001, who added an additional layer of randomness to bagging method. RF can be applied
to both classification and regression problems as well as in feature selection. (Genuer,
Poggi and Tukean-Malot, 2010).

Figure 4.2 Feature ranking of datasets features using RF algorithm.

Open Access Journal www.smrpi.com 6

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

The y axis represents the importance value for each feature in the dataset while the x
axis represents the different features in the dataset. Based on their importance values, 24
feature were selected and used for the purpose of this work.
4.3 Design of Xgboost Classifier
XGBOOST (Extreme Gredient Boosted Tree) is an optimized implementation of
gradient boosted trees first introduced by Chen and Guestrin in (Chen, Guestrin, 2016) . It
is mostly employed in classification task where it is used as a classifier for mapping input
pattern into a specific class. Xgboost implements a process known as boosting to improve
the performance of gradient boosted trees. Boosting is an essemble technique that attempts
to create a stronger classifier from a number of weak classifiers (James,Witten, Hastie and
Tibshirani, 2014). XGBOOST has many strengths when compared to the traditional
gradient boosting implementations. Among its strengths are better regularization ability
which helps to reduce overfitting, high speed and performance owing to the parrallel nature
in which trees are built, flexibility due to it costume optimization objectives and evaluation
criteria, and inbuilt routines for handling missing values. These and many other advantages
of XGBOOST have made it an excellent tool of choice for many researchers in data science
and machine learning as can be seen in the following articles. (Zimmermann, Djurken,
Mayer, Janke, Boisseir and Scholesser, 2017)-(Zhang and Zhan, 2017).
As an optimization to the gradient boosted trees, Xgboost adds a regularization term to the
loss function to establish its objective function for measuring performance given by:
( ) ( ) ( ) …..Eqn(1)
Where L is the training loss function, and  is the regularization term. The training loss
measures the performance of the model on training data. The regularization term controls
the complexity of the model, which usually controls over-fitting.
Since the base model is decision tree, the output of the model yiis voted or averaged by a
collection F of k trees denoted as follows:

∑ ( ) …..eqn(2)

The objection function at time t can be computed as follows:

( ) ∑ ∑
( ) ( ) …….eqn(3)

Where n is the number of predictions and Ωis the regularization term defined as:

( ) …….eqn(4)

Open Access Journal www.smrpi.com 7

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

Where the γis the complexity of each leaf.T is the number of leaves in a decision tree,λis a

parameter to scale the penalty andw is the vector of scores on leaves.

4.4 Evaluation Criteria

To evaluate and compare the performance of the proposed model with other models
from the literature, the following evaluation metrics were employed; accuracy (ACC) ,
precision (Prec), recall (Rec), mathew correlation coefficient (MCC), and f-score. ACC
measures the ratio of websites which are correctly predicted. Prec measures the fraction of
websites correctly predicted as phishing. Rec metric measures the fraction of phishing
websites identifield by the model. MCC measures the correlation coefficient between the
predicted and actual class. F-score measures the weighted hormonic mean of precision and
recall. All metrics employed are functions of the confusion matrix as can be seen in the
mathematical formulatons. The confusion matrix shown in Table 4.2 is a table use to
describe the performance of a classification model on a set of test data for which the true
values are given.
Table 4.2: Confusion Matrix
Predicted positive class Predicted negative class
Actual positive class TP FN
Actual negative class FP TN

The abbreviations TP, FN, FP and TN are explained as follows respectively,

TP(True Positive) is a case where a model correctly predicts a website as phishing, TN
(True Negative) is a case where a website is wrongly classified as benign. FP (False
Positive) is a case where a website is wrongly classified as phishing and lastly FN (False
negative) is when the model wrongly classified a website as benign while it is actually
phishing. The mathematical equations of the performance metrics are given below
respectively.

…..eqn(5)
( )

…..eqn(6)
( )

Open Access Journal www.smrpi.com 8

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

….eqn(7)
( )
( )
….eqn(8)
( )
( ) ( )
…eqn(9)
( )( )( )( )

5. EXPERIMENTAL SETUP
The proposed methodology was implemented in python programming language and
all experiments have been carried out on Lenevo machine running with 64 bit windows
operating system, AMD E1 essential CPU at 1.00 GHz and 4.00 RAM. After pre-
processing of our datasets, it was divided into 70% for training and 30% for testing using
hold-out stratified cross validation. Finally, based on the relevance of features to this
problem, only the first 24 features were selected and used for this work.
5.1 Result of Experiments

Chart Title
1.2
Perfomance Measure

1
0.8
0.6
0.4
0.2 Rec
0
RF and Prc
Xgboost RF PNN
Xgboost
Rec 0.9735 0.972 0.7022 0.9626 Fscore

Prc 0.9701 0.9707 0.6998 0.9666 MCC

Fscore 0.97029 0.9707 0.701 0.9646 Acc
MCC 0.9405 0.9423 0.4287 0.9203
Acc 0.9708 0.9716 0.715 0.9607
Axis Title

Figure 5.1: Performance of Different Classifier Using 4000 Instances of Datasets

Open Access Journal www.smrpi.com 9

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

1
Performance Measure

0.98
0.96
0.94
0.92 Rec
0.9
Prc
0.88
Fscore
0.86
RF and MCC
Xgboost RF PNN
Xgboost
Rec Acc
0.9773 0.9789 0.9785 0.9789
Prc 0.9713 0.9726 0.9442 0.964
Fscore 0.9709 0.9721 0.9611 0.9714
MCC 0.9418 0.9443 0.9127 0.935
Acc 0.9713 0.9726 0.9565 0.9679

Figure 5.2: Performance of Different Classifiers Using the Entire Datasets

5.2 Results Analysis

Two sets of experiments were conducted in order to evaluate and compare the
performance of the proposed method. In one set of the experiment, the performance of the
proposed model was evaluated using 4000 randomly selected instances of the datasets. In
the other set, the model was evaluated using the entire datasets. The experimental results
for both cases are presented figure 5.1 and 5.2 respectively. As can be seen from above, the
proposed method has the highest accuracy, MCC and fscore in the first set of experiments
conducted. This is followed by Xgboost, Probabilistic Neural Network (PNN) and lastly by
RF in the performance hierarchy. The proposed method outperformed the other methods as
it integrates the parallelism and regularization ability of xgboost as well as the strengths of
random forest. For the second experiment, the proposed technique also outperformed the
other methods in terms of the performance metrics employed. However, when compared
with first set of experiment, a slight improvement was observed in the performance of the
entire methods under consideration. Consequently, we can conclude that the size of the
datasets have some significance in the performance of the models. The larger the size of the
datasets, the better the performance.

Open Access Journal www.smrpi.com 10

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

6. CONCLUSIONS
In this work, we have proposed a hybrid technique (RF and XGBoost) for the
detection of phishing websites by integrating RF and XGboost algorithms. The RF is used
for evaluating the most relevant features for the experiments there by reducing
computational time, while the XGboost is used for the detection. The robustness of the
proposed method was evaluated in comparisons with the individual algorithm and a
recently proposed phishing detection technique (PNN) using MCC, Fscore and ACC as
performance metrics. From the experiments conducted, the proposed technique turned out
to be the most robust among the other algorithms.

7. REFERENCES

Toolan, F. and Carthy. J. (2019). “Phishing detection using classifier ensembles,” IneCrime
Researchers Summit, eCRIME’09.(pp. 1-9), IEEE, 2009.

Gupta, B.B., Tewari, A., Jain, A. K. and Agrawal, D. P.(2017). “Fighting against phishing
attacks: state of the art and future challenges,” Neural Computing and
Applications, 28(12), pp.3629-3654, 2017.

Khonji, M., Iraqi, Y. and Jones, A. (2013). “Phishing detection: a literature survey,” IEEE
Communications Surveys & Tutorials, 15(4), pp.2091-2121, 2013.

Abdelhamid, N., Ayesh, A. and Thabtah, F. (2014).“Phishing detection based Associative

Classification data mining,” Expert Systems with Applications, 41(13), pp.5948-
5959, 2014.

Abu-Nimeh, S., Nappa, D., Wang, X. and Nair, S.(2007). “A comparison of machine
learning techniques for phishing detection,” In Proceedings of the anti-phishing
working groups 2nd annual eCrime researchers summit (pp. 60-69), ACM, 2007.

El-Alfy, E.S.M. (2017). “Detection of phishing websites based on probabilistic neural

networks and k-medoids clustering,” The Computer Journal, 60(12), pp.1745-1759,
2017.

Zouina, M. and Outtaj, B. (2017). “A novel lightweight URL phishing detection system
using SVM and similarity index,” Human-centric Computing and Information
Sciences, 7(1), p.17, 2017.

Kaytan, M. and Hanbay, D. (2017). “Effective classification of phishing web pages based
on new rules by using extreme learning machines,” Anatolian Journal of Computer
Sciences, 2(1), pp.15-36, 2017.

Open Access Journal www.smrpi.com 11

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

Mohammad, R. M., Thabtah, F. and McCluskey, L.(2014). “Predicting phishing websites

based on self-structuring neural network,” Neural Computing and
Applications, 25(2), pp.443-458, 2014.

Zhang, D., Qian, L., Mao, B., Huang, C., Huang, B. and Si, Y. (2018). “A Data-Driven
Design for Fault Detection of Wind Turbines Using Random Forests and XGboost,”
IEEE Access, 6, pp.21020-21031, 2018.

Genuer, R. Poggi, J. M. and Tuleau-Malot, C. (2010).„„Variable selection using random

forests,‟‟ Pattern Recognit.Lett, vol.31, no. (14), pp. 2225–2236, 2010.

Chen, T. and Guestrin, C.(2016). “Xgboost: A scalable tree boosting system”

In Proceedings of the 22nd acmsigkdd international conference on knowledge
discovery and data mining, (pp. 785-794), ACM, 2016.

James, G., Witten, D., Hastie, T. and Tibshirani, R.(2014). “An Introduction to Statistical
Learning with Applications in R”. Springer 2014.

Zimmermann, T., Djürken, T., Mayer, A., Janke, M. Boissier, M., Schwarz, C., Schlosser,
R. and Uflacker, M.(2017). “Detecting Fraudulent Advertisements on a Large E-
Commerce Platform,” In EDBT/ICDT Workshops, 2017.

Zhang, L. and Zhan, C. (May 2017). “Machine Learning in Rock Facies Classification: An
Application of XGBoost,” In International Geophysical Conference, Qingdao,
China, 17-20 April 2017 (pp. 1371-1374).Society of Exploration Geophysicists and
Chinese Petroleum Society.

Open Access Journal www.smrpi.com 12

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

Author(s)
1 Ali Ahmad Aminu

He is currently a staff of Gombe State University,

(Nigeria), in Department of Computer Science.

2 Abdulrahman Abdulkarim

He is currently a staff of the Federal Polytechnic Bauchi in

Computer Science Department. He is a post graduate
student studying M.Sc. Computer Science from Abubakar
Tafawa Balewa University Bauchi, he obtained his
bachelor of technology degree (B.Tech) from the same
Institution. His research focus is on Networking and
Theoretical Computer Science.

3 Amatullah Yahaya Aliyu

She is currently a staff of The Federal Polytechnic,

Bauchi in Department of Computer Science. Bauchi State
Nigeria.

4 Muhammad Aliyu

He is currently a staff of The Federal Polytechnic, Bauchi

in Department of Computer Science. Bauchi State Nigeria.

Open Access Journal www.smrpi.com 13

 Frontiers of Knowledge Journal Series | International Journal of Pure and
Applied Sciences ISSN: 2635-3393 | Vol. 2 Issue 3 (September, 2019)

5 Abdulkadir Maigari Turaki

He is currently a staff of The Federal Polytechnic, Bauchi

in Department of Computer Science. Bauchi State Nigeria.

Open Access Journal www.smrpi.com 14