J. Inst. Eng. India Ser.

B (June 2021) 102(3):605–616

https://doi.org/10.1007/s40031-021-00563-z

REVIEW PAPER

Evaluation of Machine Learning Algorithms Used on Attacks

Detection in Industrial Control Systems
Pallavi Arora1,2 • Baljeet Kaur1,2 • Marcio Andrey Teixeira3

Received: 16 October 2019 / Accepted: 22 February 2021 / Published online: 23 March 2021
 The Institution of Engineers (India) 2021

Abstract The Industrial Internet of Things corresponds to Introduction

several industrial devices that are equipped with sensors
connected to networks gathering and sharing data. These Industrial Internet of Things (IIoT) transpires owing to IoT
devices are being used by the industry, providing a new technologies to upgrade building processes and commer-
global industrial system on a scale never seen before, cial procedures. It is a network of devices connected
called Industry 4.0. The conjunction of industrial IoT and through communication technologies to form a system that
intelligent automation has been an asset for many enter- collects, exchanges and analyzes data, which helps in
prises, allowing the machines to take on tasks that previous making a smart decision faster. For example, it is possible
generations of automation could not handle. On the other to predict issues in the machinery of an industrial process
hand, the number of cyber attacks is increasing since the and address these issues in a proactive way before a part of
industrial devices have become connected to the Internet. it fails or the whole machinery goes down. IIoT helps in
In this paper, separate machine learning (ML) algorithms developing smart environment by saving time, energy and
for instance Random Forest, Support Vector Machine money, and these devices are being used by the traditional
(SVM), Decision Tree, Artificial Neural Networks (ANN), Industrial Control System (ICS).
K-Nearest Neighbors (KNN) and Naı̈ve Bayes are evalu- According to various IoT Analytics, it is expected a
ated for attacks detection against ICS and their perfor- number of 10 billion of IoT devices connected before 2021,
mance metrics are recorded. The outcome shows great and 23 billion of IoT devices by 2025. These numbers
execution of machine learning algorithms in identifying cover the customer side gadgets (e.g., Smart Home, Smart
assaults, furthermore, shows a meager erroneous alarm rate Watch) and the ventures side (such as attached apparatus),
thus implies, it identifies ordinary traffic very well. and does not include phone, tables or laptop. However, the
systems that use the IoT devices become vulnerable to
Keywords ICS  Cybersecurity  Machine learning  cyber attacks, and considering the industrial control sys-
Cyber attacks tems, these attacks are very critical since they may cause
physical damage and even threaten human lives.
Historically, the industrial devices used on ICS did not
have connections to the Internet. That is, the devices were
connected between them, but they did not have outside
connections. With that in mind, the ICS communication
& Pallavi Arora
[email protected]
protocols were developed without security mechanisms,
like cryptography, security layer, etc., which currently
1
IK Gujral Punjab Technical University, Kapurthala, India increase the risks of attacks against the ICS due to the
2
Guru Nanak Dev Engineering College, Ludhiana, India internet connectivity. For example, one of the most popular
3 protocol used in ICS is the Modbus protocol [1]. Modbus
Department of Informatics, Federal Institute of Education,
Science, and Technology of Sao Paulo, Catanduva, SP was developed in 1979 by Modicon [2], and it is used until
15808-305, Brazil now in ICS system [2].

123
606 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

Studies have been conducted to avoid attacks in ICS manufacturing processes reside at this level and com-
networks, which incorporates the utilization of ML algo- municates with level 0.
rithms to identify the attacks in the ICS network [3]. One of • Level 2: Control system monitor, supervise and regu-
the main ideas is the development of new Intrusion lates physical processes during their runtime. Report-
Detection Systems (IDS) which can detect attacks ing, asset management, production, scheduling and
according to the network traffic behavior. However, to middleware are at this level.
achieve this goal, the performance of the ML algorithms in • Level 3: It includes business logistics and enterprise
detecting attacks must be accessed. systems.
There are kinds of attacks, like DoS and DDoS that can
Earlier these industrial environments were isolated and
be detected by traditional IDS. However, reconnaissance
there was no communication with devices outside of the
and zero-day attacks are hard to be recognized by these
industrial environments. However, today the ICS is moving
IDSs, being necessary the development of new strategies to
towards more open standards providing compatibility with
identify these classes of attacks. This paper presents the
Ethernet and other communication protocols. Insertion of
evaluation of five ML algorithms that were used during the
IT capabilities into existing physical systems requires
process of identification of reconnaissance assault in ICS
security implications. It concerns safety, security and pri-
networks. The ML algorithms were trained and tested using
vacy of critical infrastructure systems.
the dataset developed by [4]. Sophisticated reconnaissance
In traditional IT, we think about information security as
attacks were conducted against an ICS testbed, where the
achieving confidentiality, integrity and availability of
detection performance of the ML algorithms was analyzed.
information with confidentiality being the primary concern.
We want to keep data secret, e.g., personal identifiable
information such as credit card numbers. In ICS first and
Background
foremost consideration is safety, but after that availability
so making sure that the system is always up and running.
IIoT refers to a variety of IoT applications used in smart
Then comes integrity: making sure that what is shown on
manufacturing. From smartphones to smart appliances in
control room screen also matches with reality. Lastly,
our home, this technology is moving towards transforming
confidentiality comes into play, but the data here do not
the world of industry. Machines will be self-optimized,
require the same level of security as in IT networks.
self-configured and will even use artificial intelligence to
do complex tasks in order to deliver better quality services.
Supervisory Control and Data Acquisition (SCADA)
It includes technologies such as Internet of things (IoT),
robotics, artificial intelligence, blockchain, etc., which are
SCADA framework is a kind of software application pro-
giving a new shape to the work culture in different
gram for observing and controlling industrial process. It
industries.
collects data from remote locations in real time and takes
supervisory decisions to adjust controls. The main func-
Industrial Control Systems
tions of SCADA include data acquisition, data communi-
cation and presentation. Field devices, for example,
The IIoT devices vary from tiny environmental sensors to a
programmable logic controllers (PLCs), remote terminal
complex industrial robot. These gadgets are utilized in ICS
units (RTUs) and intelligent electronic devices (IEDs)
conditions. ICS is a general term involving process control
collect industrial data from remote sites. Data are sent to
system (PCS), distributed control system (DCS) and
the control center via wired or wireless links. The data are
supervisory control and data acquisition system (SCADA)
accessed by the clients with the help of control server.
frameworks. These are regularly applied to the frameworks
SCADA substructures are swiftly transforming from
that control, supervise and administer gigantic production
conventional constrictive protocols to Internet Protocol
frameworks, for example, electric power generators,
(IP)—build substructures. Embracing to IP-based sub-
petrochemical activities, dams, chemical enterprises and so
structures gives monetary advantages in a stage of
forth. Figure 1 shows the general architecture of an ICS.
extraordinary emulation. Thus, an accelerating number of
As can be seen in Fig. 1, the ICS architecture is orga-
substructures are trusted upon to propel toward IP-based
nized in layers as follows:
substructures. It is preposterous to totally superimpose an
• Level 0: This level consists of sensors and actuators that IT security structure for SCADA frameworks. The main
are involved in the manufacturing process. component of SCADA architecture is communication
• Level 1: Programmable logic controller (PLC) and between center of operations, i.e., control room and all the
distributed control system (DCS) which manipulate devices linked to the control room. These control devices

123
J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616 607

Fig. 1 General architecture of

an ICS

such as PLCs and RTUs also communicate with each other gain control of system behavior, access sensitive data.
via communication protocols. Earlier ICS depends on security through quality of being
The most commonly used protocols for these commu- difficult to understand. However, use of open standard
nications are DNP3 and Modbus. Modbus is an application network protocols has left ICS exposed to nasty attacks and
layer protocol, which acts as a communication channel penetration by internet malware. The blind deployment of
between controller and field devices as well as inter-con- firewalls, cryptography and antivirus software leaves sys-
troller between field devices. It was initially started by tem operators with a false sense of security.
Modicon in 1979 but now managed by Modbus foundation. The attacks on ICS can be categorized as reconnais-
It was designed for real-time communication between sance, response injection attack, command injection attacks
controller and field devices and operates in both point-to- and denial of service (DOS) attacks [6]. A brief description
point network and multidrop network. of each attack is made below:
Modbus devices utilize master–slave design where
Modbus master can trigger intercommunications called Reconnaissance
queries and Modbus slaves reply by providing the solicited
data back to master or by carrying out the execution It is the first stage of any attack on networking system.
mentioned in the query. The Modbus slave can be any Hackers use scan tools to inspect the topology of victim
external equipment such as an I/o transducer, valve, net- network and identify devices in the network as well as their
work drive or some measuring instrument. vulnerabilities. In the ICS context, reconnaissance attacks
There is a dire need to raise attention to the problem that can be categorized as follows: Address scans, function
ICS installations were no longer stand alone and obscure code scans, device identification attack, points scan.
systems. If we apply standard IT solutions at central control
stations, they only guard and strengthen the inner core of Response Injection Attack
ICS installation and leave field devices vulnerable to
attacks. This kind of attack is common and successful on the
Internet due to their numerous types, large variety of
Attack Vectors Against ICS/SCADA Systems attacks and the complexity sometimes required to protect
them. The applications need data to work, and ICS appli-
There are three general forms of attacks on ICS: (1) Tra- cations is not different. The packets are sent and received
ditional attacks; (2) Attacks on PLC (3); and Attacks on through ICS protocols without any authentication method
Sensors [5]. Traditional attacks include memory exploits to to validate the packets. So, the packets can be captured by

123
608 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

attackers that in their turn can modify the packets of the technique encompasses ten phases. Risk elements calculate
ICS network. For example, the attackers can get packets the probability of threat agent to explore vulnerabilities that
which have values read from sensors, modified these values turns threat into accident and causes an impact on different
and the forward fake response packets. For example, a actors. Use case used by the authors was a doctor per-
response injection attack can originate from a PLC, and the forming a remote surgery controlling a robot.
sensor value sent to the supervisory software can be Mahdavinejad et al. [9] present in a detailed survey of
interpreted in a wrong way. ML algorithms that demonstrates how these techniques can
be applied to data in order to extract patterns, knowledge or
Command Injection Attacks high-level data. It also highlights the challenges faced by
ML algorithm when applied to IoT data. The main case
Command injection attack is a kind of attack very common used by the authors is that of smart cities. They applied
in web environments. Considering that the ICS is con- Support Vector Machine (SVM) to the traffic data of the
nected to the Internet, the hacker injects false commands Aarhus smart city. The following deductions are made by
into a control system. The main goal is the execution of the authors: (1) It is significant to create algorithms that can
arbitrary commands on the operating system of the host via deal with data produced from various sources with specific
vulnerable applications. So, it is possible to overwrite the data types. (2) Resources that generate data in real time are
remote terminal register settings, as well as the ladder logic with the problem of scale and velocity. (3) The most crit-
used in the PLC. ical job is to identify a good data model that matches the
data for pattern recognition and do better analysis of IoT
Denial of Service Attacks data. The main contribution of the paper can be summed up
as how ML mechanisms are practiced in IoT smart data,
The fundamental objective of the DOS attack is to disrupt classification of ML mechanisms that are employed in IoT,
the activities of a service or system, attempting to make a characteristics of IoT data in the real world and usage of
system’s resource unavailable to its users. In the ICS smart city as a case study of IoT applications.
environment, this attack attempts to halt the functionalities Hassanzadeh et al. describe the security challenges in IT
of the ICS to efficiently incapacitate the entire system. The and IIoT domain in [10]. They clearly define that solutions
systems are overloaded by receiving a lot of packets so efficient in one domain may not always apply to the other.
faster than they can be processed, generating exceptions The concentration of authors was on analyzing security
which crash the network stack. solutions for IIoT architecture. They analyze various
industrial security incidents such as Night Dragon, Stuxnet,
Shamoon. They concluded that no single security control
Literature Survey matrix can be universally suggested for ICS networks.
In Ref. [11], Jiang focuses on using deep learning
Sajid et al. [7] illustrate security problems faced while methods for instance convolutional neural networks
using IoT Cloud world by industrial SCADA systems. (CNNs) and recurrent neural networks (RNNs) for
SCADA systems are hitherto susceptible to cyber threats; infringement detection. The authors propose a multichan-
however, if we combine it with concepts of IoT, cloud nel infringement detection methodology established on
computing, etc., the security of these systems will be fur- long short-term memory recurrent neural networks (LSTM-
ther challenged. The authors also present attacks related to RNNs). The intrusion detection technique consists of data
the IoT cloud environment. Some of them are advance pre-processing, feature abstraction and multichannel
persistent threats, lack of data integrity, man-in-the-middle training and detection. A voting algorithm is used to pre-
attacks, replay attacks, etc. It also discusses about efforts dict whether the traffic is an attack or not. The algorithm
that have been made in order to secure SCADA systems. achieves accuracy of voting, which is majority result of
Some of the best practices as reviewed by authors are multichannel classifiers.
network segregation, constant monitoring and evaluation, Al-Qatf proposes in [12] an efficient deep learning
log review, network traffic analysis, updating and patching intrusion detection mechanism which is based on self-
repeatedly, proxy solutions, etc. In the end, the authors taught learning (STL). It improves accuracy of Support
conclude that there is a dire need to secure these systems Vector Machines (SVM) in terms of attacks. The algorithm
because attacks can have disastrous effect to both SCADA uses sparse autoencoder learning techniques which are very
systems and individuals associated with them. effective in new feature representation. New features are
In Nakamura et al. [8] present a risk assessment fed into the SVM algorithm to improve its detection
methodology that lays emphasis on confidentiality, pro- capability and classification accuracy. The authors test the
tection, safety, flexibility and consistency. Risk assessment algorithm for binary and multiclass classification and

123
J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616 609

compares the result with native classification algorithms Meneghello et al. [17] presents outline of security
viz. J48, Naı̈ve Bayesian, Random Forest and SVM. The threats in IoT sector and discusses its counteractions. Dif-
experiment result shows that SVM has fast-tracked training ferent standard security mechanisms such as encryption,
and testing time and performed better than previous clas- random number generator,secure hardware and IDS are
sification algorithms. discussed to protect IoT services.
In Ref. [13], Yin et al. design an intrusion detection Goel et al. [18] presents various vulnerabilities related to
system relying on deep learning approach employing IoT. Several features of IoT such as large scale, sensing,
recurrent neural networks (RNN-IDS). The authors claim intelligence, connectivity are discussed. After specifying
that deep learning implementation has led to extract better features, it also discusses security requirements in IoT
representations of the data and thus creating better models. systems. In order to secure IoT system, different security
In comparison to related work, authors have used RNN paradigms such as bluetooth low energy security, protec-
based model for intrusion detection rather than pre-train- tion against jamming attacks, intrusion detection and pre-
ing. For teaching and trials of data, the NSL—KDD dataset vention systems, TLS security are followed.
has been used. The algorithm achieves greater accuracy In Pliatsios et al. [19], all starts the paper with the
rate and detection rate with a minimal false-positive rate. general introduction to SCADA systems and its architec-
In Ahmad et al. [14] consider intrusion detection and ture. Further, it explains SCADA communication protocols
prevention systems (IDS/IPS) to be an important part of such as bitbus, modbus, dcbus, hart, profinet, DNP3, etc. It
current and future networks. The efficiency of an intrusion presents real life security incidents from year 2000 to year
detection system is measured through accuracy which can 2018 covering all important incidents such as Stuxnet,
be enhanced by decreasing false alarm rates and increasing Night Dragon, Dragonfly 2.0, etc. Attack landscape for
detection rates. Techniques used in previous literature such critical infrastructure is divided into four classes such as
as multilayer perceptron Support Vector Machine are not traditional IT based attacks, protocol-specific attacks,
capable of processing large datasets. To mitigate this configuration-based attacks and process control attacks. In
problem, techniques such as Random Forest (RF) and the last, it specifies SCADA attack detection solution. In
Extreme Learning Machine (ELM) are practiced. The this approach, first an ongoing attack is detected and then
authors conclude that ELM outstripped than all other an alert is raised. Traffic classification approach process
strategies in terms of precision, recall and accuracy on network flows and classifies it into categories of attack and
whole data specimens that comprises of both normal and non-attack. Traffic encryption approach uses cryptographic
anomalous traffic. Further authors want to explore the algorithms to encrypt the data. In the end, future
results of ELM in attribute selection and attribute trans- advancements and trends in SCADA fields are presented. It
formation methods. includes the design of secure SCADA protocols that meet
Abeshu et al. [15] expose some doubts the capabilities the requirements of industry 4.0 applications. It also
of classical ML models for attack detection in the scenario encourages the use of virtualization technologies to reduce
of IoT. So, the authors propose a deep learning-based the deployment cost and provide high reliability.
attack detection model in fog-to-things computing. A pre-
training stacked auto encoder has been used for feature
engineering, while softmax is used for categorization. The The Machine Learning Algorithms
metrics used for evaluation are accuracy, false alarm rate,
detection rate (DR) and ROC curve. The experiments A brief description of the ML algorithm has been discussed
demonstrate an increase in accuracy and efficiency using in this work. However, more details about each algorithm
deep learning technique. have been briefly described in this paper as per the earlier
In Beaver et al. [16], use ML methods to detect attacks researches [9, 20, 21].
while analyzing remote terminal unit serial communica-
tions in a gas pipeline system. The dataset was developed Decision Tree
by Mississippi State University’s critical Infrastructure
Protection Centre. The authors carefully analyzed the fea- It is a tree like arrangement which consists of branches and
tures and evaluated the performance with the aid of WEKA leaves. The leaves represent classifications and branches
ML software. The methods used are as follows: Naı̈ve represents collection of characteristics which lead to those
Bayes, Random Forests, One R, J48, NNge, SVM. A ten- classifications. Information Gain and the Gini index are
fold cross-validation is applied for both binary and multi- often used methods for identifying the optimal features that
class classification. Nearest neighbor and Random Forest best split the training samples. When we want to categorize
performed best while classification and also in terms of an anonymous specimen, it’s characteristic measures are
precision and recall. tried in opposition to the Decision Tree. A track is drawn

123
610 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

originating from root node and stopping at leaf node to classification needs to be done. The algorithm can be used
determine the class prediction of the specimen. Decision when the data is labeled, noise-free and small. More details
Tree induction is a greedy algorithm which builds a about K-Nearest Neighbor can be found in [16, 24]
Decision Tree in a top-down recursive divide and conquer
method. The key merits of Decision Tree include instinc- Naı̈ve Bayes
tive information presentation, precision in categorization
and easy execution. The Decision Tree’ biggest demerit is Naı̈ve Bayes classifier predicts the probability of an inci-
that the data, comprising categorical variables with dif- dent based on previous information related to that incident.
ferent number of strata, the information gain is skewed For example, DoS attack detection is related to network
towards features with higher levels. More details about traffic information. It is a quite simple probabilistic clas-
Decision Tree can be found in [22, 23] sifier relying on the theorem of Bayes with strong inde-
pendence assumptions. This means the presence or absence
Support Vector Machine of some characteristic is irrelevant to the existence or
nonexistence of some other characteristic. Bayesian clas-
Support Vector Machine is an extreme type of algorithm sifiers perform best in terms of speed and accuracy when
that classifies by searching a line that best splits the data applied to large datasets. More details about Naı̈ve Bayes
attributes between two or more classes. The line is sear- can be found in [19, 23, 26]
ched through maximum margins, and the distance between
the lines is equidistant. The algorithm is given labeled Logistic Regression
training data and generates an optimal hyperplane which
classifies new data. More details about Support Vector It is a statistical approach for evaluating dataset in which
Machine can be found in [16, 23, 24]. one or more independent variables decides the output. The
objective of logistic regression is to discover the best-fit-
Random Forest ting line between the dependent variables and independent
variables. Regression models (both linear and nonlinear)
Random Forest algorithm consists of several trees that are are used for predicting a real value, like salary for example.
constructed randomly and then trained to vote for a class. If your independent variable is time, then you are fore-
The maximum voted class is chosen to be final classifica- casting future values; otherwise, your model is predicting
tion output. Although Random Forest algorithms are cre- present but unknown values. More details about Logistical
ated from Decision Trees, but these algorithms differ from Regression can be found in [26].
each other significantly. Decision Trees articulate a set of
rules when the training data are put into the network, and
these rules are then used to classify new input. However, SCADA Attack Dataset and Performance Metrics
Random Forest uses Decision Trees to construct a subset of
rules and thus vote for a class. It overcomes the problem of In this section, we describe the SCADA attacks dataset
overfitting. More details about Random Forest can be used in our experiments as well as the performance metrics
found in [16, 22, 25] used to evaluate the ML models.
Despite the popularity of the ML techniques, research
K-Nearest Neighbor groups in ICS/SCADA security have reported a lack of
datasets that provides samples of attacks on ICS/SCADA
KNN classifiers classify the new input sample on the basis system [4]. This happens mainly due to security questions,
of k-nearest neighbors of that sample. Euclidean distance is once most industries do not agree to share the data of their
used to find closest neighbors. The teaching specimens are SCADA system. To address this issue, some ICS/SCADA
characterized by a n-dimensional numeral characteristic. testbed has been built to be used in cybersecurity research
Each specimen is denoted by the point within a n-dimen- [19, 24, 27]. Different attacks are made against the testbed,
sional space. In order to classify an anonymous specimen, where the network traffic is labeled between normal traffic
the algorithm searches for k training specimen that are and attack traffic, and this information is used to build the
proximate to unknown specimen in the pattern space. dataset. Then, the dataset is provided to the community for
These k training samples are referred to as k neighbors of cybersecurity research purpose. In our work, we are using
the anonymous specimen. The unidentified specimen is the dataset provided by [4, 27]. The configuration and the
allocated the most common class amid its k nearest details about the SCADA system testbed used to create the
neighbors. KNN are lazy learners because all the teaching dataset is presented in [4, 27].
specimens are kept and classifier is not built until

123
J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616 611

The Attacked Dataset Description The Performance Metrics

The dataset used in this paper was built considering the The performance of the ML and DL algorithms is mea-
attack vectors described in subsection 2.3: reconnaissance, sured by metrics which descend from the confusion matrix.
response injection and command injection attacks. Table 1 The confusion matrix is illustrated in Table 3. According to
describes the attacks carried out against the SCADA sys- the confusion matrix, the metrics used in this work to
tem testbed developed by [4, 27]. evaluate the implementation of the DL algorithms are as
The attacks were carried out during the normal operation follows:
of the SCADA system, and all network traffic was stored in
• Accuracy: It is the percentage (%) of correctly
a dataset, where each network flow was labeled into normal
projected samples considering the total number of
traffic or attack traffic. Figure 2 shows some traffic flow
projections.
samples of the used dataset.
As can be seen in Fig. 2, the used dataset is composed of Accuracy ¼ ðTP þ TNÞ=ðTP þ TN þ FP þ FNÞ  100
nine columns. The first seven columns of the dataset are the ð1Þ
network features extracted by Argus software, and the last
• False alarm rate (FAR): Represents the percentage of
two columns indicate if the network flow is normal or
regular traffic misclassified as an anomaly (attack) by
attack. The features shown in Fig. 2 have been chosen due
the model.
to their variability during the attacks. Table 2 shows the
information about the dataset. FAR ¼ FP=ðFP þ TNÞ  100 ð2Þ
As can be seen in Table 2, the number of the attack
• UN-detection rate (UND): It is the section of the
traffic is smaller than the normal traffic. This reflects what
anomaly traffic (attack) which the model misclassifies
happen in real world. Generally, the number of the attack
as normal.
traffic in the real world is smaller than the normal traffic.
This is a typical problem of unbalanced dataset. To resolve UND ¼ FN=ðFN þ TPÞ  100 ð3Þ
this problem, we have used the k-fold cross-validation • True-positive rate (TPR): It is the fraction of the traffic
technique [27]. that model rightly predicted as an attack.
In order to training and test the ML and DL algorithms
TPR ¼ TP=ðTP þ FNÞ  100 ð4Þ
used in our work, the observations of the dataset are spitted
in two parts: training set and test set, (it was used the k-fold • Receiver operating characteristic (ROC) Curve: It is a
cross-validation procedure). The training set is composed graphical study of the TPR (Eq. 4) plotted against the
of 80% of the dataset (565,284 samples) and is used to train UND (Eq. 3). The ROC curve is used to visualize the
the ML and DL models. The test set is composed of 20% of performance of the binary classifier.
the dataset (141,322 samples), and it is used to test the ML
and DL model.

Table 1 Attacks carried out against the testbed

Category of Attack name Attack description
attacks

Reconnaissance Port scanner Attack used to identify common SCADA protocols on the network. Tool used: Nmap
Reconnaissance Address scan attack This attack is used to scan network addresses and identify the Modbus server address. Nmap and
Exploit
Reconnaissance Device identification Attack used to enumerate the SCADA Modbus slave IDs on the network
attack
Response HTTP response header Metasploit. Inject new HTTP headers into the SCADA system, and write arbitrary content into the
injection injection application’s response
Command Metasploit Modbus Used to read the coils values of the SCADA devices. The coils represent the ON/OFF status of the
injection client devices controlled by the PLC, such as motors, valves, and sensors
DoS DoS SYN Flooding Initial connection request (SYN) packets are sent to PLC and HMI ports
attack

123
612 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

Fig. 2 Traffic flow samples of

the used dataset

Table 2 Information of the used dataset As shown in Fig. 3, the Decision Tree, Random Forest,
KNN and ANN algorithms have the best accuracy in
Measurement Value
comparison to other ML algorithms. There is a slight dif-
Total number of observations 706,606 ference in accuracy among these algorithms. The perfor-
Percentage of normal traffic 96% mance of the SVM, Logistical Regression and Naı̈ve Bayes
Percentage of attack traffic 4% have lower performance. However, these performances can
Average data rate 460 kbits/s be classified as a good performance. It is important to
Average packet size 76.75 bytes notice that, different from the regular network traffic, the
Average packet size 76.75 bytes SCADA system has a deterministic network traffic. So, in
this case, all ML and DL algorithms presents good accu-
racy. However, the accuracy is not a supreme parameter for
evaluating performance, hence we will use other metrics
such as FAR. Figure 4 represents the false alarm rate
Table 3 Confusion matrix
(FAR) results.
Predict class Figure 4 shows that the Logistical Regression and
Data class Classified as normal Classified as abnormal Random Forest performed best in this metric, trailed by
Normal True negative (TN) False positive (FP) SVM and Naı̈ve Bayes. The algorithms KNN, Decision
Abnormal False negative (FN) True positive (TP) Tree, and ANN have highest false alarm percentages which
means that they detected normal traffic well; however, the
number of false alarm rate is higher than the Logistical
Regression and Random Forest, which have less accuracy
performance as shown in Fig. 3. Figure 5 presents the
Numerical Results
results of the UND metric.
Figure 5 shows the results of un-detection rate metrics.
Here, we exhibit numerical outcomes of the attacks illus-
Results of UND rate are more paramount than the false
trated in the previous section. As already described in
alarm rate as in this case attack can transpire without being
subsection 5.1, 80% of the dataset is used to train the ML
noticed. The percentage of UND rate is low for ANN,
and DL Algorithms, and 20% is used to test the ML and
KNN, Decision Tree and Random Forest, where the last
DL already trained. The results of the experiments are
one shows the abysmal functioning (0,004%). The algo-
shown in Table 4, where the confusion matrix is calculated
rithms Logistic Regression, Naı̈ve Bayes and SVM present
for each algorithm used in this work.
the worst performance in this important metric. Despite
Using the results shown in Table 4, it was calculated the
these three algorithms present good accuracy performance,
performance of the ML and DL algorithms. Figure 2
it is possible to see in Fig. 5 that considering this important
depicts performance of the ML and DL algorithms in terms
metric, these algorithms did not perform well.
of accuracy.

123
J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616 613

Table 4 Confusing matrix’s results

123
614 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

Fig. 3 Accuracy results

Fig. 4 False alarm rate results

Fig. 5 Un-detection rate results

Figure 6 shows the ROC curves for Naı̈ve Bayes, ANN, Conclusion and Future Scope
Logistic Regression, Decision Tree, Random Forest, KNN
and SVM. In ROC curve top left corner shows 0% FPR and This paper exhibits the buildup of SCADA systems testbed
100% TPR and area under ROC curve demonstrates to be used for attack detection. The testbed used for con-
detection accuracy. SVM, KNN, Random Forest and ducting attacks is that of water storage tank which involves
Decision Tree have 99.9% TPR whereas ANN, Logistic water treatment and dissemination. Networking tools such
Regression and Naı̈ve Bayes have 99.5% TPR which as Argus and Wireshark are used to scrutinize different
clearly shows that all algorithms correctly identify attacks. features of network traffic and a dataset is designed for

123
J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616 615

Fig. 6 ROC curve

teaching and evaluation of all the ML models. The per- Proceedings of the 2018 Global Internet of Things Summit
formance of used ML models was evaluated using different (GIoTS) (Bilbao, 2018), pp. 1–6.
9. M.S. Mahdavinejad, M. Rezvan, M. Barekatain, P. Adibi, P.
metrics such as accuracy, FAR, UND, TPR and ROC. The Barnaghi, A.P. Sheth, Machine learning for internet of things data
outcomes show that all algorithms correctly identify analysis: a survey. Digit. Commun. Netw. 4(3), 161–175 (2018)
attacks. Our future plans focus on including more types of 10. A. Hassanzadeh, S. Modi, S. Mulchandani, Towards effective
attacks and checking the performance of intrusion detec- security control assignment in the Industrial Internet of Things, in
Proceedings of the 2015 IEEE 2nd World Forum on Internet of
tion system for every type of attack. We will consider new Things (WF-IoT) (Milan, Italy, 2015), pp. 795–800.
experiments using different type of industrial protocols like 11. F. Jiang, et al., Deep learning based multi-channel intelligent
DNP3, CAN, etc. attack detection for data security. IEEE Trans. Sustain. Comput.
(2018).
12. M. Al-Qatf, Y. Lasheng, M. Al-Habib, K. Al-Sabahi, Deep
learning approach combining sparse autoencoder with SVM for
network intrusion detection. IEEE Access 6, 52843–52856 (2018)
References 13. C. Yin, Y. Zhu, J. Fei, X. He, A deep learning approach for
intrusion detection using recurrent neural networks. IEEE Access
1. Modbus Technical Resources. [Online]. 5, 21954–21961 (2017)
http://www.modbus.org/tech.php. Accessed on 25 Jul 2019. 14. I. Ahmad, M. Basheri, M.J. Iqbal, A. Rahim, Performance
2. Modicon. [Online]. http://www.modicon.net.in/. Accessed on 26 comparison of support vector machine, random forest, and
Jul 2019. extreme learning machine for intrusion detection. IEEE Access 6,
3. M. Barreno, B. Nelson, A.D. Joseph, J.D. Tygar, The security of 33789–33795 (2018)
machine learning. Mach. Learn. 81(2), 121–148 (2010) 15. A. Abeshu, N. Chilamkurti, Deep learning: the frontier for dis-
4. M. Teixeira, T. Salman, M. Zolanvari, R. Jain, N. Meskin, M. tributed attack detection in fog-to-things computing. IEEE
Samaka, SCADA system testbed for cybersecurity research using Commun. Mag. 56(2), 169–175 (2018)
machine learning approach. Future Internet 10(8), 76 (2018) 16. J. M. Beaver, R. C. Borges-Hink, M. A. Buckner, An evaluation
5. S. McLaughlin et al., The cybersecurity landscape in industrial of machine learning methods to detect malicious SCADA com-
control systems. Proc. IEEE 104(5), 1039–1057 (2016) munications, in Proceedings of the 2013 12th International
6. C. Lin, S. Wu, M. Lee, in 2017 IEEE Conference on Dependable Conference on Machine Learning and Applications (Miami, FL,
and Secure Computing, (IEEE, Taipei, 2017), pp. 524–526. USA, 2013), pp. 54–59.
https://doi.org/10.1109/DESEC.2017.8073874 17. F. Meneghello, M. Calore, D. Zucchetto, M. Polese, A. Zanella,
7. A. Sajid, H. Abbas, K. Saleem, Cloud-assisted IoT-based IoT: internet of threats? A survey of practical security vulnera-
SCADA systems security: a review of the state of the art and bilities in real IoT devices. IEEE Intern Things J. 6(5),
future challenges. IEEE Access 4, 1375–1384 (2016) 8182–8201 (2019). https://doi.org/10.1109/JIOT.2019.2935189
8. E. T. Nakamura, S. L. Ribeiro, A privacy, security, safety, resi- 18. A. K. Goel, A. Rose, J. Gaur, B. Bhushan, Attacks, counter-
lience and reliability focused risk assessment methodology for measures and security paradigms in IoT, in Proceedings of the
IIoT systems steps to build and use secure IIoT systems, in 2019 2nd International Conference on Intelligent Computing,
Instrumentation and Control Technologies (ICICICT), (Kannur,

123
616 J. Inst. Eng. India Ser. B (June 2021) 102(3):605–616

Kerala, India, 2019), pp. 875–880, doi: 24. A. Patrascu, V.-V. Patriciu, Cyber protection of critical infras-
https://doi.org/10.1109/ICICICT46008.2019.8993338. tructures using supervised learning, in Proceedings of the 2015
19. D. Pliatsios, P. Sarigiannidis, T. Lagkas, A.G. Sarigiannidis, A 20th International Conference on Control Systems and Computer
survey on SCADA systems: secure protocols, incidents, threats Science (Bucharest, Romania, 2015), pp. 461–468.
and tactics. IEEE Commun. Surv. Tutor. 22(3), 1942–1976 25. C. Meda, F. Bisio, P. Gastaldo, R. Zunino, A machine learning
(2020). https://doi.org/10.1109/COMST.2020.2987688 approach for Twitter spammers detection, in Proceedings of the
20. Y. Xin et al., Machine learning and deep learning methods for 2014 International Carnahan Conference on Security Technology
cybersecurity. IEEE Access 6, 35365–35381 (2018) (ICCST) (Rome, Italy, 2014), pp. 1–6.
21. M. A. Al-Garadi, A. Mohamed, A. Al-Ali, X. Du, M. Guizani, A 26. Q. Liu, P. Li, W. Zhao, W. Cai, S. Yu, V.C.M. Leung, A survey
survey of machine and deep learning methods for internet of on security threats and defensive techniques of machine learning:
things (IoT) security, p. 42. a data driven view. IEEE Access 6, 12103–12117 (2018)
22. M. Almseidin, M. Alzubi, S. Kovacs, M. Alkasassbeh, Evaluation 27. M. Zolanvari, M. A. Teixeira, L. Gupta, K. M. Khan, R. Jain,
of machine learning algorithms for intrusion detection system, in Machine learning based network vulnerability analysis of
Proceedings of the 2017 IEEE 15th International Symposium on industrial internet of things. IEEE Internet Things J., pp. 1–1
Intelligent Systems and Informatics (SISY) (Subotica, Serbia, (2019).
2017), pp. 000277–000282.
23. A.L. Buczak, E. Guven, A survey of data mining and machine Publisher’s Note Springer Nature remains neutral with regard to
learning methods for cyber security intrusion detection. IEEE jurisdictional claims in published maps and institutional affiliations.
Commun. Surv. Tutor. 18(2), 1153–1176 (2016)

123