Detecting Malicious URLs

using Machine Learning Techniques

Frank Vanhoenshoven∗ , Gonzalo Nápoles∗ , Rafael Falcon† , Koen Vanhoof∗ and Mario Köppen‡
∗ Universiteit
Hasselt Campus Diepenbeek
Agoralaan Gebouw D, BE3590 Diepenbeek, Belgium
Email: [email protected]; [email protected]; [email protected]
† Electrical Engineering and Computer Science, University of Ottawa

800 King Edward Ave., Ottawa, ON K1N 6N5, Canada

Email: [email protected]
‡ Network Design and Research Center, Kyushu Institute of Technology

680-4, Kawazu, Iizuka, Fukuoka 820-8502, Japan

Email: [email protected]

Abstract—The World Wide Web supports a wide range of In order to identify these malicious sites, the web security
criminal activities such as spam-advertised e-commerce, financial community has developed blacklisting services. These black-
fraud and malware dissemination. Although the precise motiva- lists are in turn developed by a myriad of techniques including
tions behind these schemes may differ, the common denominator
lies in the fact that unsuspecting users visit their sites. These visits manual reporting, honeypots, and web crawlers combined with
can be driven by email, web search results or links from other site analysis heuristics [2]. While URL blacklisting has been
web pages. In all cases, however, the user is required to take some effective to some extent, it is rather easy for an attacker
action, such as clicking on a desired Uniform Resource Locator to deceive the system by slightly modifying one or more
(URL). In order to identify these malicious sites, the web security components of the URL string. Inevitably, many malicious
community has developed blacklisting services. These blacklists
are in turn constructed by an array of techniques including sites are not blacklisted either because they are too recent or
manual reporting, honeypots, and web crawlers combined with were never or incorrectly evaluated.
site analysis heuristics. Inevitably, many malicious sites are not Several studies in the literature tackle this problem from a
blacklisted either because they are too recent or were never or Machine Learning standpoint. That is, they compile a list of
incorrectly evaluated. URLs that have been classified as either malicious or benign
In this paper, we address the detection of malicious URLs
as a binary classification problem and study the performance and characterize each URL via a set of attributes. Classification
of several well-known classifiers, namely Naı̈ve Bayes, Support algorithms are then expected to learn the boundary between
Vector Machines, Multi-Layer Perceptron, Decision Trees, Ran- the decision classes.
dom Forest and k-Nearest Neighbors. Furthermore, we adopted De las Cuevas et. al. [3] reported classification rates about
a public dataset comprising 2.4 million URLs (instances) and 96% that climbed up to 97% with a rough-set-based feature
3.2 million features. The numerical simulations have shown that
most classification methods achieve acceptable prediction rates selection preprocessing step that reduced the original 12
without requiring either advanced feature selection techniques features to 9. The labeling of each URL was done after a set
or the involvement of a domain expert. In particular, Random of security rules dictated by the Chief Security Officer (CSO)
Forest and Multi-Layer Perceptron attain the highest accuracy. in a company. This resulted in an imbalanced classification
problem that was dealt with via undersampling. In total, 57,000
I. I NTRODUCTION
URL instances were considered after removing duplicates. The
With the undeniable prominence of the World Wide Web as authors noticed an improvement over the results attained in
the paramount platform supporting knowledge dissemination their previous work [4].
and increased economic activity, the security aspect continues Kan and Thi [5] classified web pages not by their content
to be at the forefront of many companies and governments’ but using their URLs, which is much faster as no delays
research efforts. are incurred in fetching the page content or parsing the text.
Symantec’s 2016 Internet Security Report [1] elaborates on The URL was segmented into multiple tokens from which
an ample array of global threats that includes corporate data classification features were extracted. The features modeled
breaches, attacks on browsers and websites, spear phishing sequential dependencies between tokens. The authors pointed
attempts, ransonmware and other types of fraudulent cyber to the fact that the combination of high-quality URL segmen-
activities. The report also unveils several cyber tricks used by tation and feature extraction improved the classification rate
the scammers. One well-known and surprisingly quite effective over several baseline techniques. Baykan et. al. [6] pursue a
strategy is baiting the users to click on a malicious Uniform similar objective: topic classification from URLs. They trained
Resource Locator (URL), which leads to the system being separate binary classifiers for each topic (student, faculty,
somehow compromised. course and project) and were able to improve over the best

978-1-5090-4240-1/16/$31.00 ©2016 IEEE

reported F-measure.
Ma et. al. [7] brought forth an approach to detect malicious 15

websites from the lexical and host-based features of their

number of datasets
URLs in light of the wealth of information they carry about 10

the website’s nature. Their system is able to sift through

tens of thousands of features and identify the important URL
components and metadata without requiring heavy domain 5

expertise. The approach was evaluated with up to 30,000

instances and yielded promising results, namely a very high 0

classification rate (95%-99%) and a low false positive rate. 0.1 0.2 0.3
percentage of malicious urls
0.4

The same authors in [8] resorted to online algorithms in order

to handle millions of URLs whose features evolve over time. Fig. 1. Frequency of datasets having a given percentage of malicious URLs
A system was developed to gather real-time URL features.
The authors paired it with a real-time feed of labeled URLs
from a large web mail provider. A classification rate of 99% 64 features that are non-binary1 contain both numerical and
is reported on a balanced dataset using confidence-weighted discrete, ordinal values.
learning. Classification accuracy has to be achieved by relying mainly
Zhao and Hoi [9] avoid the (typical) class imbalance in the on the power of the methods used. There is no involvement
malicious URL detection problem and the need for a large of domain experts to comment on feature importance nor
amount of training data through their Cost-Sensitive Online correlations.
Active Learning (CSOAL) framework. CSOAL queries only For the entire dataset, roughly 33% of all URLs are classi-
a small fraction of the available data (about 0.5% out of 1 fied as malicious. In Fig 1, a histogram shows the frequency
million instances) and directly optimizes two cost-sensitive (y-axis) of datasets having a given percentage of malicious
measures to address the class-imbalance issue. The empiri- URLs (x-axis). It can be seen that for the vast majority of
cal evidence indicated that their scheme achieved better or datasets, between 30% and 40% of all URLs are malicious.
highly comparable classification performance when compared The outliers to the left, containing less than 10% of malicious
to the state-of-the-art cost-insensitive and cost-sensitive online URLs, belong to Day34, 36, and 37. Day95 reports the
classification algorithms using a huge amount of labeled data. highest proportion at almost 45%. Given the proportions of the
classifications in each dataset, predicting benign for each URL
This paper addresses the detection of malicious URLs as a
in a randomly chosen test set would likely result in an accuracy
binary classification problem and studies the performance of
of 66%. Similar predictions for test sets sampled from Days
several well-known classifiers, namely Naı̈ve Bayes, Support
34, 36, and 37 could even achieve an accuracy of 90% easily.
Vector Machines, Multi-Layer Perceptron, Decision Trees,
As we expect classification algorithms to accurately predict
Random Forest and k-Nearest Neighbors. Furthermore, we
the labels, we can use these proportions to assess the results.
examine and compare the results for three different feature
sets, selected a set totaling 3.2 million features for 2.4 million III. C LASSIFICATION T ECHNIQUES
URLs (instances).
Pattern classification [11] is a common scenario in real-
The rest of this paper is organized as follows. Section II
world problems and most available models are inspired by
describes the problem under consideration and Section III
human reasoning and behavior when facing such scenarios.
outlines several popular classification techniques for solving it.
This problem consist in assigning the correct category to
The empirical results are put forth in Section IV. Concluding
a new observation (pattern) of the problem, given a set of
remarks are enunciated in Section V.
alternative reponses. Patterns are normally described by a set
of attributes which could be numerical, categorical or both.
Formally speaking, the pattern classification problem [11] is
II. P ROBLEM D ESCRIPTION about building a mapping 𝑓 : 𝒰 → 𝒟 that assigns to each
pattern/object 𝑥 ∈ 𝒰 described by the attribute set Ψ =
The data for this research, presented by Ma et al. [10], {𝜓1 . . . , 𝜓𝑀 } a decision class 𝑑 from the 𝑘 possible ones in
consists of 121 sets of URLs that have been collected for 121 𝒟 = {𝐷1 , . . . , 𝐷𝑘 }. Generally, the mapping is learned using
different days. For Day0, 16,000 URLS have been collected, a supervised approach, i.e., using an available set of historical
Day45 contains only 130 entries; For all other days, 20,000 data about the classification problem for training the model.
URLs are registered. All URLs are characterized by multiple 1 X4, X5, X6, X8, X11, X16, X17, X18, X19, X21, X22, X23, X30, X33,
attributes and each is classified as either malicious or benign. X35, X39, X41, X43, X55, X57, X59, X61, X63, X65, X67, X69, X71,
The entire dataset consists of over 2.3 million URLs, each X73, X75, X77, X79, X81, X83, X85, X87, X89, X91, X93, X95, X97, X99,
X101, X103, X105, X107, X109, X111, X113, X120, X126, X132, X134,
having over 3.2 million features. The overwhelming majority X136, X138, X140, X142, X144, X146, X148, X150, X161, X194, X270,
of these features can be identified as binary attributes. The X7801
The learning process is often driven by the minimization theory: they may be observable quantities, latent variables,
of a cost/error metric. More complex classification problems unknown parameters or hypotheses. Edges in this model rep-
include learning in presence of class imbalance [12] [13], class resent conditional dependencies; unconnected nodes represent
noise [14], partially labeled data [15] [16] or multiple classes variables that have no dependency relation to the other nodes.
per object [17], among other scenarios. Each node computes its value using a probability function that
The wide literature on classification models (henceforth outputs the probability (or probability distribution) of the node,
simply called “classifiers”) offers a range of possible solu- from a particular set of values of the preceding nodes.
tions and approaches for facing classification problems. Deci- The inference process in Bayesian networks is based on
sion trees, 𝑘-nearest neighbors, Bayesian networks, Random Bayes’ rule and it computes the posterior probability of each
Forests, Support Vector Machines or artificial neural networks explanation. The learning process needs to determine the graph
stand among the most popular classifiers, each having its own structure and the parameters of the model (the conditional
advantages and drawbacks. The reader may refer to the studies probability distribution at each node). If the variables are
conducted in [18] and [19] for further information about the discrete, this can be represented as a table, which lists the
performance of these traditional classifiers. probability that the child node takes on each of its different
A. Decision trees values for each combination of values of its parents. However
all relations in the structure have to be learned from data or
In the context of machine learning, a decision tree is a can be designed in prior, having sufficient knowledge of the
tree-like graph structure, where each node represents a test dependencies in the application domain. An alternative is to
on an attribute. Each branch represents the outcome of the use Naı̈ve Bayes classifiers [22]. A family of algorithms have
test and the leaf nodes represent the class label obtained after been proposed for learning the structure and parameters of
all decisions made through that branch. The paths from root to these models, all of them based in a common assumption:
leaf represent classification rules. The goal in this scheme then the value of a particular feature is independent of the value
is to represent the data while minimizing the complexity of of any other feature given the class variable, thus avoiding
the model. Several algorithms for constructing such optimized the modeling of such dependencies. Despite their apparently
trees have been proposed. C4.5 [20] for example, derives from oversimplistic design, Naı̈ve Bayes classifiers work quite well
the well-known divide-and-conquer technique and have been on diverse application problems, although other more recent
widely used in several application fields, being one of the techniques like Random Forest or Support Vector Machine
most popular machine learning algorithms. This scheme builds frequently outperform this classifier.
decision trees from a set of training data using the concept of
information entropy. At each node of the tree, C4.5 chooses the D. Random forest
attribute of the data that most effectively splits its set according Random Forest (RF) [23] is a well-known ensemble learn-
to the normalized information gain (difference in entropy). It ing method for supervised classification or regression. This
is important to note that this simple, yet efficient technique, is machine learning technique operates by building an ensemble
capable of handling missing values in the data sets and both of random decision trees at training time and outputting the
numerical and categorical attributes. class that is the mode of the classes (classification) or mean
B. K-nearest neighbor prediction (regression) of the individual trees. Therefore a
The 𝑘-Nearest Neighbors algorithm (𝑘NN) is a method used RF is a classifier consisting in a collection of tree structured
for classification and regression [21]. For the inference pro- classifiers which uses random selection in two moments. In
cess, this technique determine the 𝑘 closest training examples a first step, the algorithm selects several (e.g. 500) bootstrap
to the testing instance. The output in a classification context samples from the historical data. For each bootstrap selection
is the mode of the class values of the selected neighbors. In 𝑘, the size of the selected data is roughly 2/3rd of the total
case of regression is the mean of the values of its 𝑘 nearest training data (exactly 63.2%). Cases are selected randomly
neighbors. Therefore 𝑘NN is a type of instance-based learning, with replacement from the original data and observations in
or lazy learning, where the function is only approximated the original data set that do not occur in a bootstrap sample
locally and all computation is deferred until classification. The are called out-of-bag (OOB) observation. In a second step,
use of weights for the neighbors is a common technique and a classification tree is trained using each bootstrap sample,
can be based on the distance to the instance in question. A but only a small number of randomly selected variables
commonly used distance metric for continuous variables is (commonly the square root of the number of variables) are
Euclidean distance, or Hamming distance for discrete features. used for partitioning the tree. The OOB error rate is computed
The neighbors are chosen from historical data, but no learning for each tree, using the rest (36.8%) of historical data. The
algorithm is performed. A shortcoming of the 𝑘NN algorithm overall OOB error rate is then aggregated, observe that RF
is that it is sensitive to the local structure of the data. does not require a split sampling method to assess accuracy of
the model. The final output of the model is the mode (or mean)
C. Bayesian networks of the predictions from each individual tree. Random Forest
Bayesian networks [22] are directed acyclic graphs where comes at the expense of a some loss of interpretability, but
nodes represent random variables following the Bayesian generally greatly boosts the performance of the final model,
becoming one of the most likely to be the best performing TABLE I
classifier in real-world classification problems [18] [19]. T RANSFORMING NOMINAL FEATURES TO BINARY VALUES

country usa ghana russia australia

E. Support vector machine 1 usa =⇒ 1 0 0 0
2 ghana =⇒ 0 1 0 0
Support Vector Machines (SVM) [24] are another pow- 3 russia =⇒ 0 0 1 0
erful supervised learning technique that can be used for 4 australia =⇒ 0 0 0 1
classification and regression analysis. The base idea of SVM 5 ghana =⇒ 0 1 0 0
training algorithm starts from a given set of training examples
(support vectors), where each one is marked for belonging
techniques like RF or SVM are reported to be very competitive
to one of two categories, as a non-probabilistic binary linear
[18] [19].
classifier. Then, the SVM can be seen as a representation of
the examples as points in space, with the goal of separating IV. E XPERIMENTAL R ESULTS
examples in categories divided by a clear gap. More formally, A. Feature Selection
SVM tries to find a hyperplane or set of hyperplanes in a
high- or infinite-dimensional space, such as a good separation Feature selection is a delicate and difficult issue in a data
is achieved by the hyperplane that has the largest distance set consisting of over 2 million entries, each having over 3
million attributes, hence making pattern detection and feature
to the nearest training-data point of any class (functional
margin), minimizing the generalization error of the classifier. correlation two computationally heavy tasks.
In the dataset, nominal features of an URL are coded as a set
Unclassified instances then are mapped into the same space
of binary attributes, each corresponding to one of the possible
for classification.
values, see Table I for an unrelated example. When spreading
The mapping of the original finite-dimensional space into
out such a categorical value across multiple binary attributes,
a much higher-dimensional space, presumably makes the sep-
none of the individual attributes contains full information
aration easier to identify in that space. The hyperplanes in
about the feature, unless its value happens to be 1. This implies
the higher-dimensional space are defined as the set of points
that simply identifying a set of binary attributes as related to
whose dot product with a vector in that space is constant.
the same nominal feature, is not sufficient for adequate feature
The vectors defining the hyperplanes can be chosen to be
selection. One has to be able to detect those attributes for
linear combinations of images of feature vectors. Then, SVM
whom a value of 1 is the most significant in classifying an
schemes are defined in terms of a kernel function 𝑘(𝑥, 𝑦)
URL.
selected to suit the problem, ensuring that dot products may
Having over 3 million features, the calculation of inter-
be computed in terms of the variables in the original space.
dependencies between those features is a computationally
Several extensions of SVM such as support vector clustering
demanding task. Given the amount of binary attributes, only
[25] or transductive Support Vector Machines [26] are worth-
67 out of 3 million have real-values, pattern detection does
noting. SVM remains as a competitive performing machine
not guarantee a ready-to-use solution. Instead, we opted for
learning technique for supervised classification [18] [19].
different feature selection methods in which individual features
that have the highest absolute Pearson correlation coefficient
F. Multi-layer perceptron
with the actual class of the URL.
A multilayer perceptron (MLP) [27] is a feed-forward Feature set A contains binary and real-value attributes. The
artificial neural network model that attempts to map a set of first data set, Day0, is used to calculate the Pearson correlation
input data onto a set of corresponding outputs. An MLP can of each attribute with the class of the URL. Attributes that
be described as a direct graph where nodes are called neurons show an absolute correlation coefficient that is higher than a
and they are organized in three types of layers: input nodes, arbitrary value of 0.2, are selected.
output nodes and hidden layers. Each layer is fully connected Feature set B contains only binary value attributes. The first
to the next one and each neuron (except for inputs) has a data set, Day0, is used to calculate the Pearson correlation of
nonlinear activation function (e.g. hyperbolic tangent, logistic each attribute with the class of the URL. To correct for the
function). Each neuron updates its value taking into account absence of the real-value numbers, a lower threshold value
the values of the connected neurons and the weights of these of 0.1 is used to select all attributes with a higher absolute
connections. Pearson coefficient.
Several supervised learning algorithms have been proposed Feature set C contains all and only real-value attributes,
in literature for changing connection weights after each in- regardless of their correlation with the prediction class. These
stance of historical data is presented, based on the amount features are more likely than the binary to possess a non-zero
of error in the output compared to the expected result. Back- value, thus having a better chance of containing descriptive
propagation is perhaps the most extended one, consisting in information about the URL. Moreover, the real-value features
a generalization of the least means squares algorithm in the can be retrieved without interference from a domain expert,
linear perceptron. MLP have been widely used in literature nor is it required to do descriptive data analysis to detect
for classification problems of diverse origin, however, other patterns for feature selection.
 As a final remark in this section, we would like to mention TABLE II
that independent classifications with each of the methods, for AVERAGE ACCURACY PER CLASSIFICATION METHOD AND PER FEATURE
SET
5 different randomly chosen attributes yield very bad results.
All predictors classified every URL as benign, which is true A B C
for the majority of the URLs, hereby achieving an overall RF 98.26% (1) 96.91% (1) 97.91% (1) 97.69%
classification accuracy of 66.5%. MLP 97.97% (2) 96.57% (4) 97.31% (2) 97.28%
C4.5 97.33% (5) 96.78% (2) 96.33% (3) 96.82%
B. Method Comparison 𝑘NN 97.54% (3) 95.23% (6) 95.98% (4) 96.25%
SVM 97.11% (6) 96.01% (5) 95.17% (5) 96.10%
Given the large amount of data available, a non-stratified C5.0 97.40% (4) 96.72% (3) 93.65% (7) 95.92%
independent random sample with equal probability is taken NB 95.98% (7) 91.36% (7) 94.25% (6) 93.86%
from a set of row numbers between 1 and 20,000. For each of 97.37% 95.65% 95.80%
the 121 data sets, the URLs corresponding to the row numbers
in the sample, would be added to the training set. This results TABLE III
P OST- HOC PROCEDURES : PAIRED COMPARISONS OF ACCURACY WITH
in 2.5% of all URLs used for learning the models. REGARD TO ACCURACY OBTAINED BY RF
The test set consists of 121 different data sets, one for each
day, containing all URLs that have not been used for training. RF Hochberg Holm Hommel
All numbers from the experimental results have been achieved MLP 0.00104 0.00104 0.00104
C4.5 4.6E-14 4.6E-14 4.6E-14
by calculating the prediction accuracy for each of the 121 𝑘NN <2E-16 <2E-16 <2E-16
different test sets. SVM <2E-16 <2E-16 <2E-16
In order to evaluate the models, we use three different C5.0 <2E-16 <2E-16 <2E-16
NB <2E-16 <2E-16 <2E-16
metrics. Accuracy, Eq. 1, can be seen as the overall success
rate of the method in terms of predictions. Precision, Eq. 2, is
the ratio of positive predictions that are correctly classified. The results for 𝑘NN are obtained with 𝑘 = 2. Random
Low precision suggests the method is inclined to classify Forest uses 500 trees. In SVM, the radial basis function is
URLs as malicious, even when they are not. Recall, Eq. 3, used, with 𝛾 = 𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓1𝑓 𝑒𝑎𝑡𝑢𝑟𝑒𝑠 . MLP has 10 neurons in the
is the ratio of actually positive cases that are also identified as first hidden layer and 30 in the second.
such. Table II display the average accuracy rate per classification
𝑇𝑃 + 𝑇𝑁 method and per feature set. The classification methods are
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (1) sorted based on mean accuracy across the different feature sets.
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
The table includes summaries per method and per feature set,
𝑇𝑃 as well as a ranking of the performance of each classification
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (2)
𝑇𝑃 + 𝐹𝑃 method per feature set.
A first observation is that all methods and all feature sets
𝑇𝑃
𝑟𝑒𝑐𝑎𝑙𝑙 = (3) achieve reasonably high accuracy scores, with very small
𝑇𝑃 + 𝐹𝑁 differences in between. On average, only Näive Bayes achieves
The formula’s used in Eq. 1, 2, and 3 use the following an overall accuracy below 95%. In total, only three times an
symbols: average accuracy below 95% has been observed. This is the
∙ TP : number of true positives, malicious URLs that are case for Näive Bayes with feature sets B and C and C5.0 with
classified as such feature set C.
∙ TN : number of true negatives, benign URLs that are The ranking of the methods is fairly similar across all
classified as such feature sets. The lower ranking for 𝑘NN using feature sety
∙ FP : number of false positives, benign URLs that are B might be explained by the fact that a random, minor, noise
classified as malicious factor had to be added to the set of binary features. This is
∙ FN : number of false negatives, malicious URLs that are needed to diminish the number of ties that would occur after
classified as benign all euclidean distances between URLs have been calculated.
Precision and recall are included to counter the unbalanced The highest accuracy score has been obtained by RF,
data sets, containing more benign than malicious URLs. using feature set A. Even though the differences between
As low recall rates indicate that a method is failing to the methods are generally small, pair-wise statistical tests
identify malicious URLs, it could be argued that this measure using Hochberg, Holm and Hommel post-hoc procedures,
is most important in a context where the cost of an undetected see Table III seem to support the claim that, overall, RF
malicious URL outweighs the cost of rejecting a benign URL. significantly outperforms the other classification methods in
When there is high recall and low precision, the method terms of prediction accuracy.
is classifying too many URLs as malicious. Ideally, both Despite the fact that all accuracy scores ressemble ea-
measures are as high as possible, but, more importantly, they chother, the Bergmann post-hoc procedure in Table IV con-
should have roughly the same value, indicating a balanced and siders most differences significant at a level .01. The test does
unbiased prediction. not lend proof to conclude that there is a significant difference
 TABLE IV TABLE VI
B ERGMANN POST- HOC PROCEDURE : ACCURACY ACROSS ALL FEATURE P OST- HOC PROCEDURES FOR PAIRED COMPARISONS USING DIFFERENT
SETS FEATURE SETS

RF MLP C4.5 𝑘NN SVM C5.0 A Hochberg Holm Hommel

MLP 0.00E+00 B <2E-16 <2E-16 <2E-16
C4.5 0.00E+00 3.23E-05 C <2E-16 <2E-16 <2E-16
𝑘NN 0.00E+00 0.00E+00 2.22E-10 B
SVM 0.00E+00 0.00E+00 0.00E+00 4.03E-08 C 0.082 0.082 0.082
C5.0 0.00E+00 0.00E+00 1.42E-14 1.56E-01 3.23E-05
NB 0.00E+00 0.00E+00 0.00E+00 0.00E+00 1.51E-12 0.00E+00

between feature set B and C, see Table VI.

TABLE V Fig. 2 depicts boxplots for the different performance mea-
B ERGMANN POST- HOC PROCEDURE : ACCURACY FOR FEATURE SET A
ONLY
sures per classification method and per feature set. The meth-
ods are sorted based on mean accuracy per method, across
RF MLP 𝑘NN c50 c45 SVM feature sets.
MLP 1.12E-03 As with accuracy, precision and recall seem to be generally
𝑘NN 0.00E+00 3.41E-08
C5.0 0.00E+00 3.23E-13 8.71E-02 high, but display a much wider range across the data sets.
C4.5 0.00E+00 0.00E+00 6.86E-03 1.81E-01 Especially the precision measure seems to have an occasional
SVM 0.00E+00 0.00E+00 2.41E-12 1.05E-07 7.15E-05 outlier with a relatively low value compared to recall and
NB 0.00E+00 0.00E+00 0.00E+00 0.00E+00 1.55E-13 2.77E-03
accuracy. According to the definition, this suggests that occa-
sionally, a high number of URLs may be incorrectly classified
as malicious.
in accuracy between the results obtained from C5.0 and thos
By definition, classifications with a recall measure that is
obtained using 𝑘NN. Nevertheless; these results suggest that,
relatively low compared to its accuracy and precision, have a
for the largest part, the ranking of the methods as depicted in
tendency to classify URLs as benign, resulting in a relatively
the table can be considered significant, electing RF as the best
bigger number of malicious URLs to remain undetected. Fig. 2
classifier across all feature sets, followed by MLP and C4.5.
suggests that Näive Bayes has a tendency towards low recall
As the results suggest that feature set A has the best
scores. In general, there do not seem to be big differences
performance in terms of accuracy, Table V compares the
between recall and precision for feature set A and B. It should
results of Bergmann’s post-hoc procedure using the accuracy
be noted though, that feature set B has a tendency to reveal
for feature set A only. The methods are sorted based on
slightly lower recall. This tendency is even more pronounced
highest mean accuracy obtained for that feature set, resulting
for feature set C, which is especially the case for Support
in a slightly different sequence than can be seen in Table
Vector Machines, Näive Bayes, and C4.5.
IV. Showing significance at level .01, the procedure again
lends credibility to the ranking of most methods. There is V. C ONCLUSION
no sufficient proof to conclude that theaccuracy obained by The numerical simulations and the corresponding statistical
𝑘NN is superior to the accuracy achieved via C5.0. In turn, tests strongly suggest that the differences between the methods
C5.0 does not significantly outperfrom C4.5 at the .01 level. It are significant. Therefore, the ranking of the methods as shown
should be noted that the difference in accuracy between 𝑘NN in Table II can be accepted as a correct and significant ranking
and C4.5 is nevertheless to be considered significant. in terms of prediction accuracy. Although all methods achieve
Having RF as the highest ranked method, the ranking of the fairly high prediction accuracy, Random Forest appears to be
other classification methods is slightly different depending on the most appropriate classification algorithm for this problem,
the feature set. This ranking per feature set has been included followed by MLP. Moreover, Random Forest achieves high
between brackets next to each accuracy score in Table II. scores for both precision and recall, which not only indi-
Mention can be made of C5.0 which has the poorest ranking cates well-balanced and unbiased prediction results, but also
for feature set C, but ends up in third place for the other two provides credibility to the method’s ability to maximize the
sets. detection of malicious URLs within reasonable boundaries.
Feature set B, containing only binary parameters apparently Regarding feature selection, the feature set containing the
offers little variation to the classification methods as all features that have the highest absolute Pearson coefficient
classifications end up with an accuracy score of around 96.5%, with the prediction class, outperforms the two other sets at a
with the exception of Näive Bayes. significant level. No significant differences in accuracy can be
Overall, feature set A seems to obtain the highest perfor- found when comparing feature set B, highly correlated binary
mance results overall for every classification method. Feature features, with feature set C, only the real-value features.
sets B and C alternate as second ranked, depending on the Despite the significance of the difference in accuracy it
method used, and end up on virtually the same overall average should be noted that the differences between the feature sets
accuracy score. This claim is supported by the pair-wise tests, remain relatively small. Depending on the level of accuracy
where no significant difference in accuracy can be found that is considered to be acceptable, one might pick either
 A B C

RF

MLP

C4.5
method

kNN

SVM

C5.0

NB

0.6 0.7 0.8 0.9 1.0 0.6 0.7 0.8 0.9 1.0 0.6 0.7 0.8 0.9 1.0
value

metric_name accuracy precision recall

Fig. 2. Accuracy, precision and recall per classification method and per feature set

feature set B or feature set C as proxies for set A. The benefit [2] P. Prakash, M. Kumar, R. R. Kompella, and M. Gupta, “Phishnet:
of using feature set C over the other sets would be that it does predictive blacklisting to detect phishing attacks,” in INFOCOM, 2010
Proceedings IEEE. IEEE, 2010, pp. 1–5.
not require a preliminary exploration analysis to determine the [3] P. de las Cuevas, Z. Chelly, A. Mora, J. Merelo, and A. Esparcia-
correlations, relying solely on the descriptive power of a real- Alcázar, “An improved decision system for URL accesses based on a
value feature over a binary feature. rough feature selection technique,” in Recent Advances in Computational
Intelligence in Defense and Security. Springer, 2016, pp. 139–167.
It is worth highlighting that these accuracy rates have been [4] A. Mora, P. De las Cuevas, and J. Merelo, “Going a step beyond the
achieved without requiring either advanced feature selection black and white lists for URL accesses in the enterprise by means of
techniques or the involvement of a domain expert. Never- categorical classifiers,” ECTA, pp. 125–134, 2014.
[5] M.-Y. Kan and H. O. N. Thi, “Fast webpage classification using url
theless, the methods seem to achieve competitive results. We features,” in Proceedings of the 14th ACM international conference on
believe that classification problems for over 2 million entries Information and knowledge management. ACM, 2005, pp. 325–326.
with over 3 million features is as much about feature selection [6] E. Baykan, M. Henzinger, L. Marian, and I. Weber, “Purely URL-based
as it is about the identification of an adequate classifier. The topic classification,” in Proceedings of the 18th international conference
on World wide web. ACM, 2009, pp. 1109–1110.
fact that the feature set is sparsely populated and contains [7] J. Ma, L. K. Saul, S. Savage, and G. M. Voelker, “Beyond blacklists:
mainly binary attributes makes feature selection a more chal- learning to detect malicious web sites from suspicious urls,” in Proceed-
lenging task. Categorical features are spread out over multiple ings of the 15th ACM SIGKDD international conference on Knowledge
discovery and data mining. ACM, 2009, pp. 1245–1254.
binary attributes, causing none of the attributes to contain full [8] ——, “Learning to detect malicious URLs,” ACM Transactions on
knowledge about the feature. Given that numerical features are Intelligent Systems and Technology (TIST), vol. 2, no. 3, p. 30, 2011.
not coded this way and, as a consequence, do not suffer from [9] P. Zhao and S. C. Hoi, “Cost-sensitive online active learning with
the aforementioned drawbacks, they are interesting candidates application to malicious URL detection,” in Proceedings of the 19th
ACM SIGKDD international conference on Knowledge discovery and
to use during training. The results of this paper suggest that the data mining. ACM, 2013, pp. 919–927.
classification methods achieve competitive prediction accuracy [10] J. Ma, L. K. Saul, S. Savage, and G. M. Voelker, “Identifying Suspicious
rates for URL classification when only the numerical features URLs: An Application of Large-scale Online Learning,” in Proceedings
of the 26th Annual International Conference on Machine Learning.
are used for training. ACM, 2009, pp. 681–688.
[11] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern classification, 2nd ed.
John Wiley & Sons, 2012.
R EFERENCES [12] Y. Sun, A. K. Wong, and M. S. Kamel, “Classification of imbalanced
data: a review,” International Journal of Pattern Recognition and Arti-
[1] Symantec, “2016 Internet security threat report,” https://www.symantec. ficial Intelligence, vol. 23, no. 04, pp. 687–719, 2009.
com/security-center/threat-report, 2016, [Online; accessed 11-Aug- [13] V. López, A. Fernández, S. Garcı́a, V. Palade, and F. Herrera, “An insight
2016]. into classification with imbalanced data: Empirical results and current
 trends on using data intrinsic characteristics,” Information Sciences, vol.
250, pp. 113–141, 2013.
[14] B. Frénay and M. Verleysen, “Classification in the presence of label
noise: a survey,” Neural Networks and Learning Systems, IEEE Trans-
actions on, vol. 25, no. 5, pp. 845–869, 2014.
[15] I. Cohen, F. G. Cozman, N. Sebe, M. C. Cirelo, and T. S. Huang,
“Semisupervised learning of classifiers: Theory, algorithms, and their
application to human-computer interaction,” Pattern Analysis and Ma-
chine Intelligence, IEEE Transactions on, vol. 26, no. 12, pp. 1553–
1566, 2004.
[16] I. Triguero, S. Garcı́a, and F. Herrera, “Self-labeled techniques for
semi-supervised learning: taxonomy, software and empirical study,”
Knowledge and Information Systems, vol. 42, no. 2, pp. 245–284, 2015.
[17] G. Tsoumakas and I. Katakis, “Multi-label classification: an overview,”
International Journal of Data Warehousing and Mining, vol. 3, no. 3,
pp. 1–13, 2007.
[18] M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do
we Need Hundreds of Classifiers to Solve Real World Classification
Problems?” Journal of Machine Learning Research, vol. 15, pp. 3133–
3181, 2014.
[19] M. Wainberg, B. Alipanahi, and B. J. Frey, “Are random forests truly
the best classifiers?” Journal of Machine Learning Research, vol. 17,
no. 110, pp. 1–5, 2016.
[20] J. R. Quinlan, C4.5: programs for machine learning. Morgan Kauffman
Publishers, 1993.
[21] T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE
Transactions on Information Theory, vol. 13, no. 1, pp. 21–27, 1967.
[22] N. Friedman, D. Geiger, and M. Goldszmidt, “Bayesian network clas-
sifiers,” Machine learning, vol. 29, no. 2-3, pp. 131–163, 1997.
[23] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp.
5–32, 2001.
[24] M. A. Hearst, S. T. Dumais, E. Osman, J. Platt, and B. Scholkopf,
“Support vector machines,” Intelligent Systems and their Applications,
IEEE, vol. 13, no. 4, pp. 18–28, 1998.
[25] A. Ben-Hur, D. Horn, H. T. Siegelmann, and V. Vapnik, “Support vector
clustering,” Journal of machine learning research, vol. 2, no. Dec, pp.
125–137, 2001.
[26] T. Joachims, “Transductive inference for text classification using support
vector machines,” in ICML, vol. 99, 1999, pp. 200–209.
[27] F. Rosenblatt, “Principles of neurodynamics. perceptrons and the theory
of brain mechanisms,” DTIC Document, Tech. Rep., 1961.