1

A Survey on Industrial Control System Testbeds

and Datasets for Security Research
Mauro Conti , Senior Member, IEEE Denis Donadel , and Federico Turrin

Abstract—The increasing digitization and interconnection of However, due to the so-called IT/OT Convergence [1], the two
legacy Industrial Control Systems (ICSs) open new vulnera- networks have been interconnected to facilitate the digitization
bility surfaces, exposing such systems to malicious attackers. of processes, opening new vulnerability surfaces.
Furthermore, since ICSs are often employed in critical infrastruc-
arXiv:2102.05631v3 [cs.CR] 25 Feb 2021

tures (e.g., nuclear plants) and manufacturing companies (e.g., For Cyber-Physical Systems (CPSs), which also contains
chemical industries), attacks can lead to devastating physical ICSs, the classical CIA triad (Confidentiality, Integrity, Avail-
damages. In dealing with this security requirement, the research ability) is considered reversed, in order of importance, as
community focuses on developing new security mechanisms such Availability, Integrity, and Confidentiality [2]. In this context,
as Intrusion Detection Systems (IDSs), facilitated by leveraging reliability becomes the most critical request since, differently
modern machine learning techniques. However, these algorithms
require a testing platform and a considerable amount of data from IT systems where the main concerns are about the
to be trained and tested accurately. To satisfy this prerequisite, confidentiality of the data, for an ICS instead, the availability
Academia, Industry, and Government are increasingly proposing is fundamental since it can guarantee human safety and fault
testbed (i.e., scaled-down versions of ICSs or simulations) to tolerance [3]. For instance, in a nuclear plant environment,
test the performances of the IDSs. Furthermore, to enable data availability (e.g., the temperature of the core) is more
researchers to cross-validate security systems (e.g., security-by-
design concepts or anomaly detectors), several datasets have been important than its confidentiality [4].
collected from testbeds and shared with the community. Since these systems control physical and sometimes dan-
In this paper, we provide a deep and comprehensive overview gerous processes, security is a fundamental need. However, in
of ICSs, presenting the architecture design, the employed de- recent years several viruses attempting ICSs were identified.
vices, and the security protocols implemented. We then collect, One of the first cyberattack targeting SCADA systems dates
compare, and describe testbeds and datasets in the literature,
highlighting key challenges and design guidelines to keep in back 1982 [5] when a trojan targeting the Trans-Siberian
mind in the design phases. Furthermore, we enrich our work pipeline causes a massive explosion. In successive years,
by reporting the best performing IDS algorithms tested on many incidents exposed the security weaknesses of ICSs.
every dataset to create a baseline in state of the art for this Stuxnet [6], [7] is probably the most famous malware discov-
field. Finally, driven by knowledge accumulated during this ered in this field. Stuxnet was a worm discovered in 2010
survey’s development, we report advice and good practices on the
development, the choice, and the utilization of testbeds, datasets, targeting Programmable Logic Controllers (PLCs) used in
and IDSs. gas pipeline and power plants. It was able to cause self-
destruction of 984 centrifuges in a uranium-enrichment plant
Index Terms—Cyber-Physical Systems, Industrial Control Sys-
tems, Security, Intrusion Detection Systems, Dataset, Testbed. in Iran. In 2014, the third version of a known trojan family,
BlackEnergy [8], was developed to target ICSs. In the follow-
ing years, this trojan was spread mainly inside a Microsoft
I. I NTRODUCTION Word document that, once open, request to activate macros
RITICAL infrastructures and the emerging Industry 4.0
C are increasingly using more advanced technologies such
as computers, electrical and mechanical devices to monitor the
that hide the virus. Victims of these attacks are media and
energy companies, mining industries, railways, and airports
in Ukraine. On December 23, 2015, an attack employing
physical processes. The networks resulting from smart smart BlackEnergy3 caused a three-hour disconnection of 30 sub-
computing integration for the processes monitoring are called stations in the Kyiv Power Distribution company, leading to
Industrial Control Systems (ICSs) or, sometimes, SCADA several hours of blackouts in the area. More recently, in 2017,
systems. another important malware, TRITON [9], was identified after
ICSs are composed of two macro areas. The Operational an unscheduled shutdown of a Saudi Arabian petrochemical
Technology (OT) network includes hardware and software processing plant. TRITON reprograms some special PLCs
used to monitor and manage industrial equipment, assets, used for safety purposes, causing them to enter a failed state.
processes, and events (e.g., Programmable Logic Controllers According to a report of Kaspersky Lab [10], in the second
(PLCs), sensors, actuators). On the other side, the traditional half of 2016 the 39.2% of the industrial machines secured
Information Technology (IT) network contains workstations, by Kaspersky’s products have been attacked, a clear sign
databases, and other classical machines used to manipulate in- that threats to ICS are a growing problem nowadays. The
formation. IT and OT networks were originally disconnected. vulnerabilities affecting these systems are also reported in
recent studies on real ICS traffic over the Internet [11],
This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may [12], showing a dramatic lack of security features on the
no longer be accessible. communication.
 2

A successful attack on ICS implied a huge economic impact in terms of F1-Score, Accuracy, and Precision, which are the
on the organization. These consequences include operational most common metrics. We have accurately analyzed all the
shutdowns, damage to the equipment, business waste, intel- testbeds, datasets, and IDSs to provide the readers with an
lectual property fraud, and significant health and safety risks. exhaustive overview of the current ICS state of the art. The
Nozomi Network reports that known shutdown events of an paper aims to assist interested readers: (i) to discover the
ICS [13] due to an attack cost from 225K$ up to 600M$. different testbeds and datasets which can be used for security
An increasing attack trend against ICS is Ransomware, which research in ICS with a description of the design key points,
aims to obtain economic rescue [14]. According to Cov- (ii) to have a clear baseline when developing an IDS on a
eware [15], in Q4 of 2019, the average ransom payment particular dataset, and (iii) to understand the challenges and
increased by 104% to 84,116$, up from 41,198$ in Q3 of the good practices to keep in mind when designing an ICS
2019. One of the most recent Ransomware is EKANS, which testbed or dataset.
was discovered targeting 64 ICS [16]. We summarize our main contributions as follows:
To prevent such catastrophic events, it is fundamental to im- • We provide a comprehensive background on ICS, which
plement novel security-by-design approaches, and where it is offers an overview on the reference architecture and the
impossible to apply them, prevention or mitigation techniques main components characterizing such systems;
must be integrated. However, to develop a new security-by- • We present the most employed industrial communication
design concept, it is required a complete testing infrastructure. protocols with a particular focus on the intrinsic secu-
Generally, researchers rely on scaled-down versions of a real rity features and proposed security expansions of each
ICS, created ad-hoc to reproduce real-world systems but in protocol;
a controlled environment, called testbed. Testbeds could be • We provide an exhaustive overview of the current ICS
based on physical devices to provide reliable data at the cost state of the art by analyzing different testbeds, dataset,
of being more expensive or virtual if the application does and IDS related to the ICS field available on literature to
not require exact measures. However, the development of a provide the reader with key points design concepts;
new testbed is not straightforward, instead is challenging from • We offer the reader an exhaustive survey of the different
different points of view, ranging from implementation costs, testbeds and datasets which can be used for security
sharing capability, and fidelity (Section VI). research in ICS;
To develop prevention and mitigation techniques, nowadays, • We describe the best performing IDS developed for the
researchers involve machine learning techniques that exploit presented datasets. During the development of this work,
big amounts of data to train classification algorithms to detect we noted a lack of a defined methodology to test the
misbehavior or potential attacks. The straightforward approach detection frameworks (e.g., testing the IDS on the single
to collect data is to record and provide to researchers data from attacks or the whole dataset) and a defined baseline to
real ICSs. However, since these systems are generally critical compare the developed IDS. We believe that this baseline
and fundamental for society, this strategy can be challenging in can offer a starting point for future researchers to begin
many aspects. For instance, it is difficult, if not impossible, to working on ICS security having a clear idea of the current
deploy attacks in a real environment because they can damage state of the art direction and trend;
the physical process or some devices. Moreover, privacy is • Finally, we offer a review of the challenges and the
a problem: private companies could be reluctant to share good practices to keep in mind when designing an ICS
system data from their ICSs. In fact, disclosing this data can testbed, dataset, or IDS, with some insight into the future
cause theft of intellectual property or reveal the infrastructure’s directions useful to fill the field gaps.
vulnerabilities, attracting malicious attackers’ attention. From To continue collecting the testbeds and datasets in the future
a testbed, it is possible to generate data and share them with and sharing them with the community, we also developed a
other researchers to compare and improve different detection website (Section VI) to support the resource sharing among
algorithms’ results. These captures are called datasets and can the researchers in this field.
be composed of physical measures (i.e., data from OT sensors)
and/or network traffic (i.e., data from network communica-
tions). Datasets are an excellent testing solution due to their B. Survey Organization
simplicity and availability. However, they are also challenging The remainder of this paper is organized as follows. In Sec-
from many points of view, for instance, in the generation tion II we provide an overview of the previous survey on this
process and lack modularity (Section VII). field, highlighting how we differ from them. In Section III we
provide background on Industrial Control Systems describing
the reference architecture and the common devices employed.
A. Contribution In Section IV we describe the most typical protocols for ICS
In this paper, we present a comprehensive survey specifi- communication, highlighting the main characteristics, security
cally targeting the security research platform in the ICS field. features and offering an analysis of their diffusion in the
This work aims to collect all the information related to the market. In Section V we briefly recall the concept of Intru-
testbed and dataset to support future research and studies on sion Detection System, also describing conventional attacks
this sector. Furthermore, for each dataset identified, we report implemented on ICSs. Then, in Section VI and Section VII
the best score achieved by an Intrusion Detection System (IDS) we describe and analyze respectively the different testbeds and
 3

datasets present in the literature. In Section VIII, some advice based on the research area, which motivates the development
and good practices are illustrated both for researchers that of each system. In this work, instead, we classify testbed
use these technologies both for institutions that want to create mainly based on the platform type, providing the reader an
brand new datasets or testbeds. Finally Section IX concludes overview of the most suited ICS testbeds and datasets for his
the paper. research.
Recently in 2019, Geng et al. [21] presented a survey on
ICS testbeds based on the same four requirements of [17].
II. R ELATED ICS S URVEYS
Besides analyzing different ICS datasets, the authors also
Literature includes different surveys comparing testbeds and present the different techniques that can be employed to build
datasets created for applications in the ICS field. However, a testbed, including application scenarios, the main challenges,
to the best of our knowledge, no detailed analysis gathers and future development directions. However, the authors’ only
and describes both the ICS datasets and testbeds, but also the introduced an analysis of each testbed’s structure without
main IDSs implemented on them. For every dataset, we also going into details or providing comparison tables.
report the algorithm with the best performances and the most In the same year, Choi et al. [22] gathered and analyzed
interesting and innovative detection approaches. We believe datasets for ICS security research, providing different compar-
that this could be useful for future research in this field and ison tables to understand the most suitable dataset depending
set a baseline to compare the different detection results. on the case study. The authors based the comparison on the
In 2015, Holm et al. [17] proposed a complete revision attack vector strategy. The paper includes 11 commonly used
of several papers related to ICS testbeds. The authors then datasets. Some existing datasets not widely used or without
focused on the objective and the component’s implementation attacks data are intentionally not considered. However, even
of 30 different testbeds. Furthermore, the authors provided if not suited for anomaly detection tasks, the latter could be
an analysis of each testbed’s main requirements (i.e., fidelity, useful to study the ICS environment’s behavior. Also, one of
repeatability, measurement accuracy, and safe executions). The the presented dataset (i.e., DEFCON23) is no longer available.
paper’s main scope was to provide an overview of the actual In 2020, in [23] the authors present an exhaustive survey
state of such systems’ development without detailing each with guidelines and good practices to help the building of an
single testbed composition. In fact, except for a table that ICS testbed, highlighting the main challenges and the results
indicates each testbed’s location, all the others show only of a focus group involving security experts to identify relevant
aggregated information. Moreover, since the paper is not design factors and guidelines. In the same years, the same au-
recent, some of the presented testbeds are quite old and not thors published [24], with interesting guidelines for each ICS
widely used nowadays, while others that were only designed layer and a set of characteristics to consider when outlining
have never been made (e.g., [18]). testbed objectives, architecture, and evaluation process. While
McLaughlin et al. [19], in 2016, presented a complete these works are interesting and give a comprehensive insight
survey on state of the art in ICS security. The paper briefly into the process of designing and evaluating a testbed, they
introduces the ICS operation’s key principles and the history do not consider datasets and IDSs, their requirements, and
of cyberattacks targeting ICSs. The authors then addressed the relationships.
vulnerability assessment process, outlining the cybersecurity Our survey aims to collect all the platforms (i.e., testbeds
assessment strategy advised for ICS and providing a list and datasets) useful for ICS security research. We base the
of steps to study the security and the vulnerabilities of an existing literature to provide a detailed analysis of the current
industrial system. In the end, the authors focused on new research issues, challenges, and future directions characteriz-
attacks and mitigation techniques. Moreover, the paper briefly ing this field. We also report the best performance of the IDS
presents a small list of some testbeds which can be used for on every dataset, which can be helpful for future IDS research
security research in this field. Differently, our work is less baseline.
focused on providing a complete landscape on ICS security,
while it offers a more in-depth analysis and review of all the III. I NDUSTRIAL C ONTROL S YSTEMS
most used testbeds and datasets for ICS security research.
In this section, we offer a background on ICSs, useful when
In 2017, Cintuglu et al. [20] presented a comprehensive start approaching this field and to understand the remainder
survey focused on smart grid testbeds, providing a systematic of this paper. Firstly, in Section III-A, we focus on the
study with a particular focus on their domains, research goals, architecture of such systems compared to the classical systems
test platforms, and communications infrastructures. There are architecture. Then we present a summary of the widely used
some intersections between smart grid and ICS fields, such ICS components in Section III-B.
as some used components and protocols. Nevertheless, some
different concepts require a separate ICS analysis, like the
specific applications and sensors used, combined with the A. ICS Architecture
complexity of smart grids. To classify smart grid testbeds, the Industrial Control Systems (ICSs) are composed of the in-
authors provide different possible taxonomy. Some of them terconnection of different computers, electrical and mechanical
can be applied to the entire ICS field (e.g., platform type). devices used to manage physical processes. These systems
Instead, some others are specific to Smart Grid (e.g., NIST grid are usually very complex and include heterogeneous hardware
domain). The employed testbed classification in [20] is mainly and software components such as sensors, actuators, physical
 4

systems and processes being controlled or monitored, compu- Enterprise network

tational nodes, communication protocols, Supervisory Control Enterprise level 5
And Data Acquisition (SCADA) systems, and controllers [25]. Zone Site business planning Business logistic systems
and logistic network level 4
Control can be fully automated or may include a human in
the loop that interacts via a Human Machine Interface (HMI).
ICSs are widespread in modern industries (e.g., gas pipeline, Demilitarized Zone (DMZ)

water treatments) and critical infrastructures (e.g., power plant

and railway). Engineering Manufacturing operations systems
Workstations level 3
Unlike classical IT systems, ICSs are composed of standard Control systems
network traffic over TCP/IP stack and data from physical HMI
Control level 2
processes and low-level components. This interconnected and Zone Intelligent devices
PLC IED RTU
intertwined nature can open a wide space for new generation level 1
Physical process
attacks exploiting new vulnerabilities surfaces. Several proto- Sensors Actuators
level 0
cols are used in ICS, based on the specific purpose of each
system. Industrial protocols are specifically designed to deal Safety Functional safety
with real-time constraints and legacy devices in an air-gap Zone

environment. Many protocols do not implement any encryption

Fig. 1: ICS Purdue Model architecture.
or authentication mechanism due to these constraints, opening
several vulnerabilities surfaces. Moreover, sometimes, the in-
dustrial protocols are customized from the company opening,
world, mainly due to the implementation difficulty or, more
again, many documentation and vulnerability issues.
generally, the companies’ insufficient attention to the industry
The reference architecture of the ICS is the Purdue building phase’s security aspects. This condition exposes the
Model [21], [26]. As depicted in Figure 1, the Purdue module critical part of the system (i.e., OT network) to potential
divides an ICS network into logical segments with similar attacks.
functions or similar requirements:
Compared with the classical IT environment, ICSs need a
1) Enterprise Zone, or IT network, includes the traditional different risk handling strategy. The reliability is fundamental,
IT devices and systems such as the logistic business and outrages are not tolerated due to the critical nature of
systems and the enterprise network. the processes monitored, unlike IT, where occasional failures
2) Demilitarized Zone (DMZ) controls the exchange of are acceptable. The risk impact is also different: in the IT
data between the Control Zone and the Enterprise Zone, environment, the principal risk is the compromising of privacy
managing the connection between the IT and the OT and confidentiality (e.g., loss or unauthorized alteration of
networks in a secure way; data). Instead, in the OT environment, a data compromise can
3) Control Zone, sometimes also referred to as OT net- cause a loss of production, equipment, and, in the worst case,
work, includes systems and equipment for monitoring, a loss of lives or environmental damage.
controlling, and maintaining the automated operation of Another difference with respecting traditional IT systems
the logistic and physical processes. It is divided into four relies on information handling performance: in an IT envi-
sub-levels: ronment, the throughput must be high enough, while delays
• Level 0 includes sensors and actuators that act and jitter are accepted. On the other hand, in the industrial
directly on the physical process; field, communication is defined with a regular polling time.
• Level 1 includes intelligent devices such as PLC, Generally, this polling time is in second or millisecond or-
Intelligent Electronic Device (IED), and Remote ders, but delays are serious concerns. Finally, in IT systems,
Terminal Units (RTU); recovery can be made by rebooting. In contrast, in the OT
• Level 2 includes control systems such as Human system, fault tolerance is essential since a reboot would imply
Machine Interfaces (HMI), alarms, and control room shutting down the entire industry and can lead to enormous
workstations; economic losses [18].
• Level 3 includes manufacturing operation systems For all these reasons, and considering that nowadays most of
that are often responsible for managing control plant the ICS are connected with the Enterprise zone, it is essential
operations to produce the desired end product; to protect them using new and precise technologies.
Level 2 and Level 3 devices can communicate with the
Enterprise Zone through the DMZ.
4) Safety Zone includes devices and systems for managing B. ICS components
ICS security by monitoring for anomalies and avoiding Industrial Control Systems are composed of a wide range
dangerous failures; of heterogeneous devices and components with a specif role
The role of the DMZ is to filter the internal communication in the system. In this section, we briefly introduce the most
of the network. In fact, according to the Purdue model, all common devices in the ICS fields. These devices are generally
the traffic Exchanged between OT and IT networks must pass installed or simulated in the testbed to replicate the ICS
through the DMZ. However, this is rarely respected in the real- environment.
 5

Programmable Logic Controller (PLC). PLC is a operating system hosting the various software for controller
microprocessor-controlled electronic device that reads input and applications. Engineers use this platform to manage the
signals from sensors, executes programmed instructions using controllers.
these inputs and orders from supervisory controllers, and Human Machine Interface (HMI). The HMI is a software
creates output signals that may change switch settings or installed on desktop computers, tablets, smartphones, or ded-
move actuators. PLC is generally the boundary between the icated flat panel screens that permit operators to check and
OT network and the physical process. It is often rugged monitor the automation processes. As illustrated in Figure 2,
to operate in critical environmental conditions such as very the HMI shows the state of a plant operator, such as process
high or low temperature, vibration, or in the presence of big values, alarms, and data trends. An HMI can monitor multiple
electromagnetic fields. As with most ICS components, PLCs process networks and several devices. An operator can use the
are designed to last more than 10-15 years in continuous op- HMI to send manual commands to controllers, for instance,
erations. The Real-Time Operations System (RTOS) installed to change some values in the production chain. Generally,
in each PLC makes it suited for critical operations. The time the HMI shows a diagram or plant process model with status
to read all inputs, execute logic, and write outputs must last information to facilitate such a job.
only a few milliseconds. Modern PLCs may use a UNIX- Data Historian. A Data Historian is a software application
derived micro-kernel and present a built-in web interface that used to collect real-time data from the processes and aggregate
makes the management more simple but exposing the device them into a database for analysis. Data Historian mainly
to new vulnerabilities. A PLC has a power supply, central collects the same information shown in an HMI. The database
processing unit (CPU), communications interface, and ana- and the hardware, generally a desktop workstation or a server,
log/digital input/output (I/O) modules that can be connected is designed for a very fast ingest of data without dropping data
to sensors (input) or actuators (outputs). These components are and uses industrial interface protocols.
generally connected to the local network to communicate with Front End Processor (FEP). The FEP is a dedicated
supervisory processes. Based on the manufacturing company communications processor used to poll status information from
and the user requests, these communications can happen multiple devices to give operators the possibility to monitor
through different mediums (e.g., serial, fiber optic, wireless) the system’s overall status.
using different protocols. Communications Gateways. A Communications Gateway
Remote Terminal Unit (RTU). An RTU is a microprocessor- is essential to make communications possible between de-
controlled electronic device. Like PLC, it is designed for harsh vices from different manufacturers that use different protocols.
environments and is generally located far from the control Gateways can translate packets from a sending system to the
center, for instance, in voltage switch-gear. There are two receiver protocol.
types of RTUs: station and field RTUs. Field RTU receives Master Terminal Unit (MTU). MTUs manage the communi-
input signals from field devices and sensors and then executes cation with the RTU or the PLC, gathers data from the PLCs,
programmed logic with these inputs. It gathers data by polling and process them. The communication between the MTU
the field devices/sensors at a predefined interval. It is an inter- and the PLCs is bidirectional, but only the MTU can initiate
face between field devices/sensors and the station RTU, which the communication. Therefore, the MTU uses a master-slave
receives data from field RTUs and orders from supervisory communication where the MTU is the Master and PLCs are the
controllers. Then station RTU generates outputs used to control slaves. Messages from the MTU to the PLCs can be triggered
physical devices like actuators. Both field and station RTU by an operator or be automatically triggered. These messages
has a power supply, CPU, and digital/analog I/O modules. can either read memory parts representing current values like
For communication with the control center, RTU uses WAN water flow, oil pressure, the temperature of a tank, or either
technologies such as satellite, microwave, unlicensed radio, write values in the memory and modify the configuration.
cellular backhaul, GPRS, ISDN, POTS, TETRA, or Internet- Supervisory Control And Data Acquisition (SCADA).
based links. SCADA devices are placed on the higher level of the ICS
Intelligent Electronic Device (IED). An IED is a device hierarchy and are used to monitor and control centralized data
containing one or more processors that can receive or send acquired from different field sites. Furthermore, they manage
data from an external source. Examples of IEDs are electronic the communication between the various devices and represent
multi-function meters, digital relays, and controllers. Thanks the remote connection point for the remote operators with the
to the higher complexity compared to a PLC and RTU, OT network. Over the year, SCADA systems protocols moved
IED can perform more operations. An IED can be used for from proprietary standards towards open international stan-
protection functions like detecting faults at a substation or for dards, resulting in attackers knowing precisely the protocols.
control functions such as local and remote control of switching That is why there is a gain of interest in reinforcing industrial
objects and provide a visual display and operator controls control systems security.
on the device front panel. Other functions can be related to ICS Field Devices. Field devices include all the compo-
monitoring (for instance, a circuit breaker condition), metering nents that are in direct contact with the physical process.
(e.g., tracking three-phase currents), and communications with The controllers can use them to get information regarding
supervisory components. the physical process (e.g., the measure of temperature or
Engineering Workstation. The Engineering Workstation is pressure using sensors). Instead, actuators can interact with
generally a desktop computer or server running a standard a physical process following commands from a controller
 6

Monitoring

Decision

Cyber-Physical
System

Action
Fig. 2: An example of HMI interface generated with Promotic
Open-Source Tool [27]. Fig. 3: The CPS close loop.

(e.g., control motors, pumps, valves, turbines, agitators). The – A tick if Encryption, Integrity, or Authentication
communication with the controllers is generally performed via are available;
I/O modules. Field Devices are implemented in the so-called • Information related to the Enhancement version with
CPS closed-loop to perform the three CPS main functions: security measures offered by the manufacturer:
monitoring using sensors, making decisions using PLCs, and – Name of the new Version of the protocol;
applying actions using actuators. These three functions operate – Year of release;
within a feedback loop covering, as shown in Figure 3. – A tick if Encryption, Integrity, or Authentication
To sum up, controllers such as PLC, RTU, and IED are are available.
mainly used to interact with the Field Devices that can instead
directly operate on the processes. HMI, Front End Processor,
A. Industrial Protocols
Engineering Workstation, and Data Historian are used to
control and manage the system data. Instead, SCADA and Modbus. Modbus is a serial communication protocol initially
Gateways Communications are used [28] to set up all the published by Modicon (now Schneider Electric) in 1979 for
connections between different components. use with its programmable logic controllers (PLCs). Today
Modbus [30] is one of the most used and famous protocols
in the ICSs. During the years, various versions of Modbus
IV. I NDUSTRIAL P ROTOCOLS S ECURITY have been released. The first version was thought for serial
With the growth of the ICSs, several new protocols have communications, allowing to establish asynchronous serial
been developed to support the specific requirements of the OT communications on RS-232 and RS-485 interface. Modbus is
environment, like fault tolerance and reliability. The majority also adapted to transmission means other than copper, such
of these protocols were designed to operate in an air-gapped as optical fiber and radio links. A typical communication via
environment. Therefore originally, less importance was given Modbus consists essentially of three stages: the formulation
to the security aspects with respect to the real-time constraint. of a request from one device to another, the execution of
Some of them have no security features at all (e.g., Authen- the actions necessary to satisfy the request, and the resulting
tication, Encryption). However, after the IT and OT conver- information’s return to the initial device. This approach’s
gence, they have still been used in practice [11]. This section main advantage lies in the interaction mode between the
reports the main industrial protocols focusing on the security various network nodes: being a client-server type, each server
proprieties initially and currently implemented. Standards like device can exchange data simultaneously with more than one
PowerLink Ethernet, EtherCAT, Constrained Application Pro- client. The Modbus/TCP variant is substantially identical to
tocol (CoAP), Message Queue Telemetry Transport (MQTT), the original serial version, but with the addition of a TCP/IP
ZigBee PRO, WirelessHART, or ISA100.11.a are not used in encapsulation module.
the datasets and testbeds identified in this paper. Therefore we Security. Implementations of serial Modbus use both RS232
decided not to report them. However, some of these protocols and RS485, which are physical layer communication proto-
are widely used in different fields, for instance, in Industrial cols. It makes no sense to speak of security on this layer, as
Internet of Things (IIoT) scenarios. these are functionalities developed on higher layers. Modbus
Table I summarized the main information related to the was designed to be used in environments isolated from the
protocols presented in this section. It includes: Internet regarding the application layer’s security. Therefore it
• Name of the Manufacturer;
does not include any security mechanism on this layer. These
• Standard Ports according to IANA [29];
deficiencies are magnified by the fact that Modbus is a protocol
• Information related to the Original protocol:
designed for legacy programming control elements like remote
terminal units (RTUs) or PLCs making the injection of
– Name of the protocol;
– Year of release;
 7

TABLE I: Summary of the main protocol’s characteristics and them security measures implemented in the enhancement
extensions. In particular, the table shows E: Encryption; I: Integrity; and A: Authentication. The Enhancement part refers to
the version proposed by the manufacturer.

Original Enhancement
Manufacturer Ports Name Year E I A Version Year E I A
Schneider Electric 502/802 Modbus 1979 Modbus/TCP Security 2008 X X X
GE Harris 20000 DNP3 1990 X DNP3-SA 2020 X X X
Siemens 102 S7Comm 1994 S7CommPlus 2014 X X
PROFINET Int. - PROFINET 2003 PROFINET Security Classes 2019 X X X
ODVA 44818 EtherNet/IP 2000 CIP Security 2015 X X X
OPC Foundation 4841 OPC 1996 OPC-UA 2006 X X X
IEC 2404 IEC 104 2000 IEC 62351 2007 X X X
IEC 102 IEC 61850 2003 IEC 62351 2007 X X X

TABLE II: Protocol used in the presented testbeds and datasets. • indicates that the protocol is supported/available. In some
work it is not indicated the version of Modbus (TCP, RTU, or ASCII) adopted. In such cases, a • is used to indicate a general
version of the protocol.

Name Modbus S7Comm EtherNet/IP DNP3 Logs Phy. Others

4SICS TCP • • •
Aghamolki et al. • • IEEE-C37.118
Ahmed et al. • • Profinet
Alves et al. TCP
BATADAL •
Blazek et al. IEC61850
BU-Testbed •
CockpitCI TCP
CyberCity Dataset TCP • NetBIOS
CyberCity Testbed TCP • NetBIOS
D1: Power System • •
D2: Gas Pipeline TCP
D3b: Water Storage Tank •
D4: New Gas Pipeline •
D5: Energy M. S. D. •
Davis et al. TCP Custom TCP
DVCP • Profinet
Electra Modbus •
Electra S7Comm •
EPIC (IPSC) TCP
EPIC (iTrust) TCP IEC61850
EPIC Dataset TCP •
EPS-ICS Unknown
Farooqui et al. Unknown
 8

TABLE II – continued from previous page

Name Modbus S7Comm EtherNet/IP DNP3 Logs Phy. Others
Gas Pipeline testbed TCP
Genge et al. • • Profinet
Giani et al. • •
Gillen et al. • CIP
GRFICS •
HAI Dataset •
HAI Testbed Fieldbus
Hui Nuclear • Profinet, Custom TCP
HVAC Traces • DCE/RPC, NetBIOS
HYDRA •
Jarmakiewicz et al. IEC61850, IEC104
Jin et al. •
Kaouk et al. TCP
Kim et al. Variable
Koganti et al. •
Koutsandria et al. •
KYPO4INDUSTRY • •
Lancaster’s testbed Converted to IP
Lee et al. • IEC61850
LegoSCADA • •
Lemay Covert •
Lemay SCADA •
LICSTER TCP
Maynard SCADA IEC104, OPC
Microgrid Unknown
MiniCPS TCP •
Mississipi Ethernet Ethernet
Mississipi Serial RTU, ASCII •
Modbus SCADA #1 TCP, RTU
MSICST TCP • •
NIST TCP • DeviceNet, OPC
PNNL Not specified
PowerCyber • IEC61850
Queiroz et al. TCP
QUT DNP3 • •
QUT S7 (Myers) • •
QUT S7Comm •
Reavers & Morris TCP, RTU
RICS-el IEC104
S4x15 ICS TCP BACnet
Sayegh et al. FINS
 9

TABLE II – continued from previous page

Name Modbus S7Comm EtherNet/IP DNP3 Logs Phy. Others
SCADA-SST •
SCADASim TCP •
SCADAVT TCP Custom TCP
SGTB Unknown
Singhet et al. • IEC104
SNL Testbed TCP • IEC104 (partially)
SWaT Dataset • • CIP
SWaT • CIP
T-GPP TCP •
TASSCS TCP
Teixeira et. al •
TETRIS
Turbo-Gas Power Plant TCP •
VPST •
VTET • • OPC
WADI Dataset •
WADI TCP
Wang et al. TCP •
WUSTL-IIOT-2018 •
Yang et al. IEC104
Zhang et al. Unknown

Fig. 4: Percentage distribution of different protocols in the datasets and testbeds analyzed.
Modbus (39.5%)

DNP3 (12.9%)

Unknown (2.4%)
CIP (2.4%)
NetBIOS (2.4%)
EtherNet/IP (10.5%) OPC (2.4%)
Profinet (3.2%)
IEC61850 (4%)
Other (8.9%)
IEC104 (4.8%)
S7Comm (6.5%)
 10

malicious code into these elements easier. Modbus Secu- Security. S7Comm is a closed protocol, so there is no related
rity [31] offers a Modbus/TCP version enhancement focused documentation. However, as various works underline, the base
on using the port 802. This new version enables TLS to version of S7Comm does not include security features, and
provide confidentiality, integrity, and authentication using it is vulnerable to replay attacks [39]. However, in 2010
x.509v3 certificates. Moreover, it specified certificate-based Stuxnet exploited the security vulnerabilities of S7Comm to
authorization using role information transferred via certificate compromise a Nuclear Plant in Iran. As a result of this
expansions. Researchers have also proposed different modifi- incident, Siemens has developed a new version of the protocol,
cations to introduce confidentiality [32] or authentication [33] called S7CommPlus, with replay-attack protection. It has been
via covert-channel on Modbus. proven that this version is also vulnerable to reverse debugging
DNP3. Westronic, Inc. (now GE Harris) designed DNP in attacks [39].
1990. In January 1995, the DNP Users Group Technical Com- PROFINET. Developed by PROFIBUS & PROFINET In-
mittee was formed to review enhancements and recommend ternational (PI), PROFINET [40] is an open standard for
them for approval to the general Users Group. One of the Industrial Ethernet standardized in IEC 61158 and IEC 61784.
most important tasks of this body was to publish the “DNP Introduced in 2003, it is an evolution of the PROFIBUS
Subset Definitions” document, which establishes standards standard, whose lines can be integrated into the PROFINET
for scaled-up or scaled-down implementations of DNP3 [34]. system via an IO-Proxy. This protocol follows the provider-
DNP3 is an open, intelligent, robust, and efficient SCADA consumer model for data exchange in a cascading real-time
protocol organized into four layers: physical, data link, pseudo- concept. It is compatible with Ethernet thanks to its flexible
transport, and application. In serial implementations, com- line, ring, star structures, and copper and fiber-optic cable
mands are issued broadcast. DNP3 contains significant features solutions. It is also compatible with radio communications
that make it more robust, efficient, and interoperable than older such as WLAN and Bluetooth. Thanks to Ethernet-based
protocols such as Modbus, at the cost of higher complexity. communication, it provides a direct interface to the IT level.
The protocol’s primary goal is to maximize system availability The primary functions include a cyclic exchange of I/O data
by putting less care into confidentiality and data integrity with real-time properties, acyclic data communication for
factors. DNP3 organizes data into data types such as binary reading and writing of demand-oriented data, including the
inputs/outputs, analog inputs/outputs, counters, time and date, identification and maintenance function, and a flexible alarm
file transfer objects. model for error signaling with three alarm levels.
Security. As previously mentioned, DNP3 is a protocol Security. PROFINET is a protocol operating in the application,
designed to maximize system availability by putting less care link, and physical layers. The link layer in this protocol uses
into confidentiality and data integrity factors. Data link level FDL (Fieldbus Data Link) to manage access to the medium.
includes the detection of transmission errors through Cyclical FDL operates with a hybrid access method that combines
Redundancy Check (CRC) calculation. However, CRC is not master-slave technology with the passing of a token, indicating
a proper security measure since if an attacker can modify who can initiate communication and occupy the bus. These
a packet, he/she can also change the CRC. At the applica- measures ensure that devices do not communicate simulta-
tion level, some efforts have been made to provide a safe neously. However, FDL constitutes any safety mechanism
authentication standard in DNP3. While in the beginning, and may be susceptible to attacks involving traffic injection
pre-shared keys were used to authenticate, according to the or Denial Of Service (DoS). In 2019, PI introduced three
standard IEEE 1815-2010 (deprecated), the latest versions Security Classes to offer a way to select security measures
implement Public Key Infrastructure (PKI) with remote key based on the consumer needs [41]. Class 1 improves robust-
changes (standard IEE 1815-2012). Recently, in 2020, GE ness through a digital signing of General Station Description
Harris presents DNP3 version 6, introducing DNP3-SA [35], a (GSD) files using a PKI infrastructure, an extended Simple
separate protocol layer that supports Message Authentication Network Management Protocol (SNMP) configuration, and
Codec (MACs) to provide secure communication sessions, a DCP in read-only mode. Class 2 expands the previous
including authentication and integrity. Moreover, this version class by offering integrity and authenticity via cryptographic
supports encryption to offer data confidentiality by using the functions and confidentiality only of the configuration data.
AES-256 algorithm. Some other solutions have been proposed Instead, Class 3 offers all the previous characteristics and the
in the literature to implement cryptography protections, such confidentiality of all the data. Furthermore, it is worth mention
as end-to-end encryption [36] and VPN for IP networks [37]. that PROFIBUS offers some services that can use TCP/IP
S7Comm. Introduced in 1995, S7comm (S7 Communica- as a transport protocol, but only during an initial phase for
tion) [38] is a Siemens proprietary protocol that runs between device assignment. It is possible to add some of the classical
standard PLCs of the Siemens S7-200/300/400 family and TCP/IP cryptography and authentication security elements in
new generation PLCs like S7-1200/1500. It is a proprietary these services.
and closed standard without significant literature related to it. ODVA’s networks. Founded in 1995, ODVA [42] is a global
Siemens has a proprietary HMI software for the SIMATIC association whose members comprise the world’s leading
products and an Ethernet driver that provides connectivity to automation companies with the mission of developing advance
devices via the Siemens TCP/IP Ethernet protocol. In addition open and interoperable communication technologies for indus-
to this driver, there are also 3rd-party communication suites trial automation. The primary interest is developing the Com-
for interfacing and exchanging data with Siemens S7 PLCs. mon Industrial Protocol (CIP), supporting the various network
 11

adoptions such as DeviceNet, CompoNet, ControlNet, and the difficult to apply patches to industrial control systems, many
widely used EtherNet/IP. CIP encompasses a comprehensive discovered vulnerabilities with available patches continue to
suite of messages and services to collect industrial automation be potentially exploitable industrial control networks. Instead,
applications such as control, safety, energy, synchronization, OPC-UA implements a security model and five security
motion, information, and network management. This protocol classes, bringing greater security to the architecture at the
allows users to integrate these applications with the IT Ethernet cost of slightly higher complexity [46]. It is also possible to
networks and the Internet. The protocol follows a model for implement only a fraction of the security measures by using
objects: each one is made up of attributes (data), services one of the five security classes provided. The security model
(commands), connections, and behavior (the relationships allows generating a secure channel that provides encryption,
between data and services). CIP also defines device types, signatures, and certificates at the communication layer. Fur-
with each device type having a device profile. The device thermore, a session in the application layer is used to manage
profiles indicate which CIP objects must be implemented, what user authentication and user authorization. Thanks to these
configuration options are possible, and the formats of I/O data. security measures, it is advisable to deploy OPC-UA rather
EtherNet/IP is an adaption of CIP to the Ethernet TCP/IP than the classic version of OPC and to update the already
stack, while DeviceNet provides a way to use CIP over the deployed versions wherever possible.
CAN technology. ControlNet uses CIP over a Concurrent IEC 60870-5-104 (IEC 104). Released in 2000, IEC 60870-
Time Division Multiple Access (CTDMA) data link layer, 5-104 (IEC 104) protocol [47] is an extension of the IEC
and CompoNet implements CIP on a Time Division Multiple 101 protocol with the changes in transport, network, link, and
Access (TDMA) data link layer. physical layer services to suit the complete network access.
Security. Recently, in 2015, ODVA introduced the CIP Se- The standard uses an open TCP/IP interface to network to
curity framework [43] to provide security measures to CIP connect to the LAN (Local Area Network), and routers with
protocol. Since different systems might need different security different facilities can be used to connect to the WAN (Wide
levels, CIP Security provides different security specifications Area Network). There are two different methods of trans-
profiles to help users configuring inter-operable devices. On porting messages. The first provides bit-serial communications
EtherNet/IP, it enables TLS and DTLS to secure the TCP over low-bandwidth communications channels. In the second,
and UDP transport layer protocols. TLS and DTLS provide introduced with IEC 104, the protocol’s lower levels have
authentication of the endpoints using X.509 certificates or pre- been completely replaced by the TCP/IP transport and network
shared keys, message integrity and authentication employing protocols. Thanks to the IEC 104 simple structure in terms of
TLS message authentication code (HMAC), and optional mes- its data types and data addressing options, it is possible to
sage encryption. quickly achieve interoperability with other protocols.
Open Platform Communications (OPC). The classic Security. IEC 104, has been proven to be vulnerable to
OPC [44], developed in 1996, was designed to provide a different types of attacks, such as man-in-the-middle and
communication protocol for personal computer-based software replay attacks [48]. A more recent and secure standard of
applications and automation hardware. It was based on Mi- the IEC family is IEC 62351. This version implements end-
crosoft’s distributed component object model, making them to-end encryption to prevent attacks such as replay, man-in-
platform-dependent and not suitable for cross-domain scenar- the-middle, and packet injection. However, due to the higher
ios and the Internet. Nowadays, the classic OPC is no anymore complexity, industries rarely upgrade IEC 104 to IEC 62351.
developed. In 2006 a new version, OPC United Architecture IEC 61850. Like IEC 104, IEC 61850 [49] was originally
(OPC-UA), was released as an operational framework for designed to enable communications inside substations automa-
communications in process control systems. It provides greater tion systems. In recent versions, an extension of IEC 61850
interoperability, eliminating MS-Windows dependency, but allows substation-to-substation communication and provides
maintaining retro compatibility with its predecessor. This spec- tools for translation with other protocols such as IEC 60870-5,
ification is built around Service-Oriented Architecture (SOA) DNP3, and Modbus. The protocol is devised using an object-
and is based on web services, making it easier to implement oriented design suited for communication between devices of
OPC connections over the Internet. The general layout of the different vendors. To provide long-term stability, IEC 61850
communication is simple: the hardware devices (e.g., PLC, divides the information model and communication protocols.
Controller) act as data sources, and the software applications For not time-critical applications, the protocol uses MMS
(e.g., SCADA, HMI) play the role of data consumers, whereas via TCP/IP as the communication protocol. Instead, GOOSE
the OPC interface acts as connectivity middleware, enabling can be employed over Ethernet if the time constraint is
the data flow. Using the OPC, the client applications access critical. In the case of voltage and current sample information
and manage the field information without knowing the physical transportation, SV over Ethernet is generally used. However,
nature of data sources. With OPC-UA improvements, the recent versions of the standard provide GOOSE/SV mapping
protocol is widely used in critical and industrial fields such over TCP/IP using UDP packets at the transport layer for inter-
as energy automation, virtualized environment, and building substation information exchange.
automation. Security. IEC 62351 standard provides various security mea-
Security. The use of Distributed Component Object Model sures, offering guidelines and developing a secure operation
(DCOM) and Remote Procedure Call (RCP) make OPC very framework. Since in time-critical application encryption is not
susceptible to different attacks [45]. Since it is inherently suitable due to the 3ms delivery overhead, the standard recom-
 12

mends using digital signature generated by SHA256 and RSA of Modbus and S7Comm devices during the two years of
public key algorithms. For MMS communications instead, recording, while the number of DNP3 devices decreased. A
TLS is recommended, with optional end-to-end encryption of similar study, presented by Barbieri et al. [11], leveraged
all the packets exchanged [50]. Furthermore, the employment Shodan and an Internet Exchange Point (IXP) in Italy to
of IEC 61850 in heterogeneous networks exposes the system measure ICS host exposure. They discover many devices using
to protocol mapping vulnerabilities. It is possible to prevent Modbus, MQTT, and Niagara Fox. Furthermore, the authors
these vulnerabilities by developing ad-hoc security by design also identified EtherNet/IP, S7Comm, and BACNet devices but
architectures [51]. with significantly lower samples.
Other protocols. In addition to previously presented proto- In addition to measurement works, we can also rely on
cols, some datasets contain few packets related to other generic market analysis. According to an HMS report [54], the overall
protocols used in a wide range of applications. Published in market share of Industrial Ethernet protocols increased in
1995, BACnet is a data communication protocol for building 2020. In particular, EtherNet/IP and Profinet obtained first
automation and control networks supported by some HVAC place with 17% of market share, while in third place there
components but not widely used [52]. Distributed Computing is EtherCat with a share of 7%. On the other side, Fieldbus
Environment/Remote Procedure Calls (DCE/RPC) is a remote protocols such as Profibus and DeviceNet showed a decrease
procedure call system that allows programmers to write dis- of 5% on the market share with respect to the previous year.
tributed software as if it were on the same computer. One of the Interestingly that Modbus protocol (TCP and RTU variants),
datasets presented in this paper includes DCE/RPC, together despite being heavily employed in testing, results in a 10% of
with NetBIOS, a networking protocol allowing applications on market share (5% RTU, 5% TCP).
separate computers to communicate over a LAN. Other generic Based on these findings, Modbus/TCP results as the most
packets are visible in some datasets like Address Resolution employed protocol in testbeds, datasets, and Internet mea-
Protocol (ARP) and Domain Name System (DNS) requests surements. Nevertheless, it obtains a low market share in the
but are generally not related to the industrial field. last year (i.e., 2020), outranked by EtherNet/IP, which is also
employed in a significant part of the testbeds and datasets
presented in this survey, and Profinet, which instead is used
B. Industrial Protocols Employment in only the 3.2% of the testing system analyzed in this work.
In Table II, we provide the complete list of the datasets Therefore, Profinet can be an interesting protocol to introduce
and testbeds analyzed in this work, together with the protocol in future testbeds and datasets to follow the market trend.
used in the specific platform. In detail, the table associates Other protocols that an increasing market share are BACnet,
to each testbed the protocols supported and to each dataset TridiumFox, and NiagaraFox, which are not present in the
the protocols available. Moreover, it indicates if data logs and testing platforms, except for one dataset that contains BACnet
physical measures are provided in the datasets. As previously packets. Finally, another protocol that could be interesting to
described, there are several different protocols employed in include in testing platforms is EtherCAT, which has a 7% of
the ICS field. In Figure 4 we reported the percentage of usage market share. However, no testing system currently supported
of each protocol in the testbeds and datasets investigated in it.
this survey. Modbus, and its different versions (i.e., TCP, RTU,
ASCII), are the most used protocols, while EtherNet/IP, DNP3, V. ICS ATTACK AND D EFENCE
and S7Comm follow with a lower but significant employments.
In this section, we offer an overview of the various attacks
Since the testbed and dataset employed should represent and defense mechanisms in ICSs. In particular, in Section V-A
an approximation of real-world scenarios, it is interesting we present an overview of the typical attacks which can
to compare if the distribution of the protocols implemented target ICSs and are implemented in the different testbeds
in the different datasets is similar to the protocol’s distri- and datasets. Instead, in Section V-B we proposed a brief
bution in the real industrial system. Verifying this claim is overview of the techniques which can be employed to detect
a challenging task due to the various privacy concerns of and mitigate cyberattacks in this field.
companies in disclosing information. Various works tried to
deal with this problem by measuring the industrial traffic
present on the Internet. Although with limitations, the traffic A. Typical Attacks
measurement can represent a reasonable estimate of the most ICSs are extremely complex systems, which connecting IT
popular industrial protocols currently used. By leveraging components with sensors, actuators, and other OT devices.
Censys search engine, Xu et al. [53] scanned the Internet for Such an interconnected scenario with a wide variety of various
about two years (from 2015 to 2017), examining for industrial components may hide attack surfaces caused, for instance, by
devices exposed. In particular, they focused on five protocols: device-specific vulnerabilities or misconfigurations.
Modbus, S7Comm, DNP3, BACnet, and Tridium Fox. Results Having a clear idea of the different attack typologies is
show a significant prevalence of Modbus and Tridium Fox essential in building and testing defenses. Based on that,
devices (with more than 20K devices found), a middle spread testbeds should be capable of simulating verisimilar attacks,
of BACnet (about 11K devices) and S7Comm (about 4.5K while datasets should include not only normal operation data
devices), and a lower number of devices using DNP3 (less than but also attack data. Reproducing attacks is a challenging task
1K). Furthermore, the authors noticed an increasing number because the simulation should precisely emulate a realistic
 13

abnormal operating condition. However, it is impossible to to unpredicted behaviors in the ICS.

replicate every type of attack due to the devices’ potential Physical Process Attacks. This class of attacks aims to alter
damages. Some attacks could also shift the testbed’s operating the physical process and the complex relations of the system to
behavior in a dangerous state and seriously damage the ma- manage it. Cyber-Physical Systems enable such attack surfaces
chines. Furthermore, the limited class of attacks implemented due to the field device (i.e., sensors and actuators), sometimes
could raise a generalization problem of the detection strategy, in remote places. To achieve these attacks, the attacker could
not transferable to novel and unknown attacks. have previously obtained access to the system with one or
In a CPS scenario, by definition, there are two possible more of the network attacks previously described. Generally,
attacks vector surfaces on the system. Network-based attacks, physical process attacks represent the final attack chain goal,
targeting the networking part of the network such as packets, which started with the network as an entry point.
protocols or routing policies, and Physical-based attacks, • Stealth Attack generates small perturbations in the sys-
aimed at corrupting the physical process of the devices. Some- tem process to create long term damages (e.g., loss in
times, these two attack categories’ goals may also converge or production terms or the devices’ degradation). The stealth
combine to reach a specific target. attack can use a static perturbation, by introducing a
Network Attacks. The most common attack models include
constant error in the physical measure (e.g., increasing
the Control Zone network access by the attacker to com-
or decreasing the production), or dynamic, by rapidly
promise an ICS. An attacker can obtain the network control
oscillating between upper and lower measurement bounds
through a phishing attack to the site operators [9] or by
(e.g., causing turbulence in the flows). This class of
exploiting the security lack of the legacy devices connected
attacks is generally difficult to detect since it maintains
to the Internet [11]. There are different actions that malicious
the process inside its limits.
actors can perform, but it is possible to categorize the main
• Device Manumission is achieved by physically tamper-
ones into five different classes [55], [56] of network attack.
ing with the field device to comprise the data recorded.
These attacks are also implemented in the testbeds to generate
This attack aims to induce wrong measurements in the
abnormal operating conditions.
system exploiting the distributed and, therefore, less
• Reconnaissance Attack aims at the identification of
monitored nature of these systems.
potential victims within a network. Usually, this class of • Direct Damage Attacks aims to disrupt and damage
attack is used to plan other moves, such as identify other the entire process or physical equipment by introducing
vulnerable devices. These attacks can be passive (e.g., significant process variations that bring the system into
port mirror) or active (e.g., nmap). an unsafe state. This attack may also have severe conse-
• Man-in-the-Middle (MitM) Attack allows an attacker to
quences on the population or the environment around the
sit in the middle of communicating parties. The attacker site.
is then able to read or modify the communications, inject
commands, or drop packets. The final aims can range
from the control of some devices to the disruption of the B. ICS Defence Techniques
ICS’s normal state to damage the system’s owner or the It is possible to enforce ICS security by implementing
system itself. security-by-design network architectures. For instance, it is
• Injection Attack aims at supplying untrusted and mali- possible to use DMZ as specified in the Purdue Model
cious inputs to a system. Typically, in an ICS, an attacker (Figure 1), enforcing network separation and segregation.
can inject data such as false measures from sensors or Furthermore, boundary protections and firewalls with ICS-
actuators (Data Injection Attack) or command (Command specific rules help protect an ICS from external attacks.
Injection Attack). Often a compromised node launch this The National Institute of Standard and Technology (NIST)
type of attack, but, in some cases, the injected data can proposed a complete guide explaining how to set up a secure
originate from other sources (e.g., a new entry point for network to protect an ICS [2]. However, security-by-design
the network). can be challenging to consider in ICSs, due to implementation
• Replay Attack is based on the retransmission of a valid constraints. Sometimes, it could also happen that companies
message that has been previously seen in the network. consider the security aspects after the construction phase. De-
This attack is difficult to be detected, and it can lead to tection mechanisms can solve this limitation and be integrated
malfunctions of the system. For example, in a nuclear into the system, even if not always easy, after the construction
plant context, the attacker could retransmit a message phase, for instance, in central nodes or with network tap.
with a low temperature of the reactor instead of rising, It is possible to deploy process-aware techniques to detect
inhibiting the activation of safety measures. attacks that cannot be identified by code execution monitoring
• Denial of Service (DoS) Attack is used to make devices or other traditional methods used in IT environments. There
unavailable by overloading the system resources to dis- are several techniques to do so, and the idea behind them is to
rupt the communication between machines in the system. exploit the massive amount of data collected from the sensors
Usually, a common technique is packet flooding and, if and predict an ICS’s operations. Moreover, they represent
packets are generated from many different sources, it is a cost-effective solution since they can be installed without
called Distributed Denial of Service (DDoS). This attack changing the system topology or substituting every network
can stop some devices, making them unavailable and lead device.
 14

In the following, we briefly report the two main categories A. Testbeds Classification
of IDS, which employ two different approaches to detect
attacks or domain drifts. There are different possible classifications of a testbed,
Knowledge-based intrusion detection (also called misuse- basing on its sector, construction methodology, or the process
based) focus on looking for runtime features, such as phys- involved. In this survey, and particularly in this section,
ical values of sensors and actuators or network traffic, that we consider the functional elements involved in the testbed,
match a specific pattern of misbehavior. This method aims to classifying them as Physical, Virtual, or Hybrid testbed. The
exploit the stationary of ICSs, which, unlike IT systems, are different testbed categories are illustrated in Figure 5, even
characterized by control loop operation regulated by a constant if sometimes the difference between can be minimal. For
polling time communication. However, these detection systems instance, many of the virtual testbeds presented can be inter-
only react to known dangerous behavior, so there is no connected with physical devices or wholly virtualized. Instead,
protection against zero-day vulnerabilities. For this reason, Hybrid systems were designed with some real components
the research community is focusing on developing a dynamic and, without them, they could not work correctly.
mechanism that can identify domain shifts without the need Physical testbeds use real hardware and software to con-
for signatures [57]. figure both the network and physical layers. They are a
The current research trend focuses on the anomaly-based suitable approach when researchers need a solution to collect
intrusion detection, which looks for runtime features that differ realistic measurement variation and latencies. Furthermore, it
from normal behavior. The normal behavior pattern can be is possible to exploit the vulnerabilities of a specific device.
defined using unsupervised approaches training the model On the other hand, physical testbeds are expensive both in
with live data or semi-supervised utilizing a set of truth data. construction and maintenance. They generally have a long
This approach is called behavior specification-based intrusion building time, and they may not provide a safe execution of
detection. It represents a suitable ICS solution since it aims dangerous physical processes (e.g., nuclear sector).
to dynamically learn the regular behavior model of network On the contrary, virtual testbeds leverage software simu-
traffic and physical models. lations and emulations with single or multiple programs to
This last method is promising thanks to modern machine reproduce the entire network and all the different components.
learning and deep learning techniques that can be used for A virtual testbed represents a low-cost solution, but it is
anomaly detection classification. A common requirement of not easy to simulate high fidelity physical processes due to
these algorithms is the need for a considerable quantity of data: the virtualized environment. Despite this lack of precision,
generally, the more data you provide to the training phase, the dangerous and risky processes (e.g., Nuclear sector) can be,
more precise your detection will be. in this way, simulated in a laboratory. Matlab, Modelica,
IDS are also classified according to the data source. Ptolemy, and PowerWorld are software used in the process
Network-based IDS uses network adapters to collect and simulation phase. Other tools are used to model control center
analyze packets in real-time. On the contrary, host-based IDS communication networks (e.g., DETER, Emulab, CORE, ns3)
monitors the documents, processes, and other information and other devices used in the system such as PLCs (e.g.,
specific to a particular device to identify. The disadvantage STEP7, RSEmulate, Modbus Rsim, Soft-PLC). Despite not
is that monitoring regard only one node in the network, while generating data with perfect fidelity, these approaches are easy
with the former approach, all the network is under control. to update and upgrade, which gives them good flexibility and
On the other hand, host-based can detect also threat coming extendibility.
from sources other than the network (e.g., USB sticks) [58]. A widely diffused approach is developing testbeds com-
A novel detection design concept that exploits the correlation posed of both physical devices and software simulations. This
of multiple ICS points was proposed by Bernieri et al. [59]. approach represents a good trade-off between physical and
In this work, the authors proposed a distributed detection virtual solutions and is called a hybrid testbed. The main
approach to consider the different information characterizing difference between the complete physical testbeds is that part
ICSs to identify more complex vulnerabilities. of the components is simulated using specialized software.
This solution can reduce the system’s fidelity, but on the other
VI. ICS T ESTBEDS hand, it permits to contain the cost and development time.
In this section, we present a comprehensive analysis of However, as stated before, the separations between Virtual
the various ICS testbeds available in the literature. Firstly, and Hybrid testbed is not always well defined. Sometimes
in Section VI-A we introduce the classification method we virtual testbeds can be modified to work as a hybrid testbed
employ in this work. Instead, in Section VI-B and Sec- by supporting physical devices. For example, VTET [60] can
tion VI-C we recall, respectively, the requirements for an be deployed using physical PLCs to replace the simulated
effective testbed and the main challenges in developing an ones. In this work, we consider as Hybrid a virtualized testbed
ICS testbed. Then, we propose a detailed description of a set composed of at least one real industrial device (e.g., PLC, IED,
of interesting testbeds we choose testbed, dividing them into actuator, sensor).
the three categories that we design. In particular, Section VI-D In Figure 7, we reported the geographic distribution of the
contains physical testbeds, Section VI-E presents virtualized Physical and Hybrid around the world. We think that this
testbed, and Section VI-F describes hybrid testbeds which are representation could help the reader see the current research
a conjunction point of the other two categories. trend in this sector in the world. In particular, Figure 6
 15

• Cost the estimated testbed implementation cost. It can

be:
– Low for a cost estimated < 500 $
– Medium for a cost estimated between 500 $ and
10k $
– High for a cost estimated > 10k $
• Reference includes a reference to a description of the
testbed.
Fig. 5: A summary of differences between testbed types. • Resource, if available, indicates a resource for the down-
load.
This information was not always available or easy to retrieve;
provide an high level view of where the testbeds are placed therefore, the degree of detail may vary according to the
in the world, while Figures 7d, 7c, and 7b show close- specific dataset.
up on the countries with more than one testbeds. In these
figures, the marker size represents the estimated cost of the
B. Testbeds Requirements
testbed. Simultaneously, the color indicates the Citations of
the associated reference according to Google Scholar at the When researchers need to work with a real-world ICS
writing time. Furthermore, we developed a website with an environment, the proper solution is to build a testbed for
interactive map to collect and also provide information about conducting rigorous, transparent, and replicable testing of new
future ICS testbeds and datasets1 . Our goal is to continue to technologies. The different testbeds vary in dimension, com-
update this collection in the future. Moreover, in Table III we plexity, or sector. According to [21], an effective testbed needs
reported a brief comparison between the testbed presented in to satisfy four main requirements: i) Fidelity, ii) Repeatability,
this paper, highlighting their main information and features. iii) Measurement Accuracy, and iv) Safe execution. Some-
In particular for every testbed we reported the following times, it could be challenging to satisfy all these requirements
information. together; therefore, it is important to determine an optimal
trade-off based on the research needs during the design phase.
• Name of the testbed (or of the authors if a name is not
A testbed should be developed to achieve a good fidelity
provided);
by accurately replicate the devices and processes from a real-
• Institution in which the testbed has been developed;
world ICS. This is an expensive and space-consuming task,
• Country The country on which is based the testbed or
making it difficult for other researchers without much funding
the institution of the first author;
to replicate the same environment to verify the results. In these
• Sector indicates the field of the represented process;
cases, mathematical models can be employed to virtualize
• Category of the testbed. It can be Physical, Virtual, or
physical processes in a cheap but less accurate way.
Hybrid;
Repeatability is an essential property for a testbed: it allows
• Physical Process indicates how is implemented the phys-
other researchers to reproduce the findings and compare other
ical level. It can be Simulated with a software or Real if
solutions on the same system. This property can be easily
consists of a physical implementation;
achievable for completely simulated testbeds, while it can be
• License of the testbed. It can be:
extremely challenging for ICSs that employ physical compo-
– Open-source if the source code is freely available; nents or processes.
– Open description if, despite the source code is not A testbed should monitor a physical process and take
provided, the description is sufficiently detailed to accurate measurements without interfering with it. Sensors
allow a reader developing a similar copy; must be placed smartly, and if different points of measures
– Education if it is maintained by an university and are available, they must be carefully synchronized to provide
open to collaborator; accurate and reliable data.
– Collaborations if it is maintained by an institution Often ICSs are used to manage critical physical processes
which can accept collaborations; (e.g., chemical reactions, nuclear plants). If under attack, these
– Not available if it is owed by a private company and kinds of processes can be dangerous and can cause physical
so not accessible or not available online. damage to the system itself. Since researchers need to study
• Scope indicates the applications of the testbed. It can be: countermeasures’ effect and effectiveness to attacks, testbeds
– Security if the main scope is related to cyber security must be provided with safe execute risky processes. This
research; design challenge can be mitigated by employing simulations
– Forensic if the target scope is to provide a way to at the cost of a loss of accuracy. In other cases, processes
perform forensics research; are instead less critical. However, they can have an expensive
– Pedagogy if the main scope is to provide education or time-consuming recovery after an attack (e.g., after an
to students; attack completely empties a container into a water treatment
– General if a precise scope is not specified. system, it will take time to refill it again). In these scenarios, a
virtual approach can be an excellent alternative to the physical
1 https://spritz.math.unipd.it/projects/ics survey/ replication [60].
 16

Fig. 6: Physical and hybrid with physical process testbeds distribution around the World.

Cost
Idaho National Laboratory
Binghamton University
High Medium Low Iowa State University

US Dep. of Comm.

Ohio State University

SANS Institute
Citations
University of Tennessee
Mississippi State University
300 250 200 150 100 50 0

University of New Orleans

(a) Legend (b) North America

Beijing University of Technology

Queen's University Belfast National Sec. Res. Inst.
American University of Beirut
Lancaster University
Affiliated Institute of ETRI
Hochschule Augsburg
Brno University of Technology
Masaryk University
Universite Paris-Saclay
Joint Research Centre

University of Roma Tre

Singapore University of Technology and Design

(c) Europe (d) Asia

Fig. 7: Physical and hybrid with physical process testbeds distribution on the continents with more than one testbed: North
America, Europe, and Asia. If there is more than one dataset in a place (e.g., Singapore SUTD), we aggregated the information.
 17

TABLE III: Summary of testbeds presented in the literature. We denote Category as H: Hybrid, P: Physical, and V: Virtual.
To specify the ICS process Physics we use S: Simulated, R: Real, M: Mixed, and No: if there is not a physical process. To
denote the License, we use OD: Open Description, E: Education, OS: Open-Source, C: Collaboration, and NA: Not Available.
To denote the Scope, we use S: Security, G: General, P: Pedagogy, and F: Forensic. To denote the Cost we use L: Low, M:
Medium, and H: High. The entries in Yellow indicates that the testbed is hybrid, the entries in blue mean that the testbed is
physical, while entries in green correspond to virtual testbeds.
Na
me
Category

Institution

Country

Sector

Physics

License

Scope

Cost

Reference

Resource
or
aut
hor
s

Aghamolki et al. H USF Florida, US Power Grid S OD S M [61] -

Alves et al. H UAH Alabama, US Gas Pipeline S OD S L [62] -
CockpitCI H University of Coimbra Portugal Power Grid S OD S M [63] -
CyberCity H SANS Institute - City M E S, P H [64] -
EPIC (IPSC) H JRC Ispra Italy General CPS S OS S L [65] [66]
EPS-ICS H BIT China Generic ICS R NA G L [67] -
Gillen et al. H ORNL Tennessee, US Cooling System S OD S M [68] -
Hui Nuclear H Queen’s University UK Nuclear Plant S OD S M [69] -
HYDRA H University of Roma Tre Italy Water Distribution R OS S L [70] [71]
Jarmakiewicz et al. H MUT Poland Power Grid S OD G M [72] -
Kaouk et al. H University of Grenoble France Generic ICS S OD S L [73] -
Kim et al. H NSRI South Korea 6 Different ICS R E S, P H [74] -
Koutsandria et al. H Sapienza University Italy Power Grid S OD S, F M [75] -
KYPO4INDUSTRY H Masaryk University Czech Republic Linear Motor R OD P M [76] -
LegoSCADA H Universite Paris-Saclay France Vehicular R OS S L [77] [78]
Microgrid H OSU Ohio, US Power Grid M OD G, P M [79] -
MSICST H - China 4 Different ICS S OD S H [80] -
NIST H USDOC US 4 Different ICS M C S M [81] -
PNNL H PNNL Washington, US Generic CPS S OD S L [82] -
Queiroz et al. H RMIT University Australia Water Distribution R OD S L [83] -
SNL Testbed H SNL New Mexico, US Generic ICS No OD S L [84] -
VPST H University of Illinois Illinois, US Power Grid S OD S, G L [85] -
Ahmed et al. P UNO Louisiana, US 3 Different ICS R OD S, F, P M [86] -
Blazek et al. P BUT Czech Republic Power Grid R OD S M [87] -
BU-Testbed P Binghamton University New York, US Power Plant R OD S M [88] -
EPIC (iTrust) P SUTD Singapore Electric Power R E S H [89] [90]
HAI Testbed P ETRI South Korea Power Plant R OD S H [91] -
Lancaster’s P Lancaster University UK Generic ICS R C G, P H [92] -
LICSTER P HS-Augsburg Germany Generic ICS R OS S, P L [93] [94]
Mississipi Ethernet P MSU Mississippi, US 2 different ICS R E S, P M [95] -
Mississipi Serial P MSU Mississippi, US Industrial op.s R E S, P M [95] -
PowerCyber P Iowa State University Iowa, US Power Grid R OD S, P L [96] -
Sayegh et al. P AUB Lebanon Generic ICS No OD S M [97] -
SGTB P INL Idaho, US Power Grid R NA S H [98] -
SWaT P SUTD Singapore Water Treatment R C S H [99] [100]
Teixeira et. al P IFET Brazil Water Distribution R OD S M [101] -
T-GPP P JRC Ispra Italy Power Plant R OD S H [102] -
WADI P SUTD Singapore Water Distribution R C S, P H [103] [104]
Yang et al. P QUB Irland Power Grid R OD S M [105] -
Zhang et al. P University of Tennessee Tennessee, US Nuclear Plant R OD S M [106] -
Davis et al. V University of Illinois Illinois, US Power Grid S OD S L [107] -
DVCP V TUHH Germany Chemical Process S OS S, F L [108] [109]
Farooqui et al. V NUST Pakistan Generic CPS S OD S L [110] -
Gas Pipeline V MSU Mississippi, US Gas Pipeline S NA S L [111] -
Genge et al. V JRC Ispra Italy Generic ICS S OD S L [112] -
Giani et al. V UC Berkeley - Generic ICS S OD S L [113] -
GRFICS V Georgia Tech Georgia, US Chemical Process S OS S, P L [114] [115]
 18

TABLE III – continued from previous page

Na
me

Category

Institution

Country

Sector

Physics

License

Scope

Cost

Reference

Resource
or
aut
hor
s

Jin et al. V UIUC Illinois, US Generic ICS S OD S L [116] -

Koganti et al. V University of Idaho Idaho, US Power Grid S OD S L [117] -
Lee et al. V Ajou University Korea Power Plant S OD S L [118] -
Maynard SCADA V Queen’s University UK Generic ICS S OS S L [119] [120]
MiniCPS V SUTD Singapore Generic CPS S OS S L [121] [122]
Reavers & Morris V Georgia Tech Georgia, US Generic ICS S OD S L [123] -
RICS-el V FOI Sweden Power Grid S OD S L [124] -
SCADA-SST V KFUPM Saudi Arabia 2 different ICS S OS S L [125] [126]
SCADASim V RMIT University Australia Generic ICS S OS S L [127] [128]
SCADAVT V RMIT University Australia Water Distribution S OD S L [129] -
Singhet et al. V C-DAC India Power Grid S OD S L [130] -
TASSCS V University of Arizona Arizona, US Power Grid S OD S L [131] -
VTET V SKL-MEAC China Chemical Process S OD S L [60] -
Wang et al. V Tsinghua University China Generic ICS S OD S L [132] -

C. Challenges in Developing a Testbed task. It is also challenging to find IT experts who have
the needed knowledge to manage and maintain a complex
The development of an industrial testbed is challenging from ICS containing several OT specifications. The mainte-
several points of view. Different works analyze the challenges nance requirements must be considered from the early
in developing a well-designed ICS testbed [133], [134]. Based design stages since the increasing complexity can become
on the existing literature, in the following, we present the main even more expensive and difficult to manage [135].
problems related to the development of such a testbed. • Cost: To build physical industrial testbeds, research
• Design Guidelines: When a research group decides to groups have to deal with building and maintenance costs.
venture into building a testbed, it is fundamental to have Expenses are one of the main reasons why there are not
a clear idea of the architecture. A clear and defined many testbeds available for research, and the ones that
architecture can be useful in the development phases and exist are generally not easily accessible by everyone. To
to project further expansions. Moreover, it can guide other overcome this problem, virtualized and emulated solu-
groups in building their own testbeds and therefore en- tions are relatively diffuse in the field, even if they cannot
abling the experiment repeatability. However, it is difficult provide the same fidelity and replication accuracy.
to identify clear guidelines that help design a testbed from • Lack of Documentation: Another challenge in ICS
the engineering perspective. research is the lack of documentation of the existing
• Real Word Representation: An industrial system must systems. Companies do not share internal information
represent a real-world industrial scenario, including all related to their system’s architecture, the devices imple-
the physical processes related to the environment. Fur- mented, or the devices’ software version. This is primarily
thermore, the Industrial testbed must include the most due to the companies’ privacy concerns, protection of
common industrial devices installed in the real world intellectual proprieties, and security reasons. In fact, if a
ICSs and supporting the most used protocols. Also, it is company discloses the presence of legacy devices with
crucial to consider different versions of devices, knowing well-known vulnerabilities, it can attract several mali-
their different security features [82], [135]. The testbed cious actors’ attention. This absence of documentation
should also include the different vulnerabilities that could, made the implementation of effective real-word testbeds
however, lead to a bias in the attack strategy vector. difficult. Furthermore, the lack of documentation can be
• Replication in Safety: The physical processes controlled problematic for the in-laboratory testbed. If poor docu-
by ICSs are wide different, ranging from manufacturing mentation is provided, new researchers who start to work
processes to critical nuclear plants. The most delicate on a testbed might spend much time understanding the
processes cannot always be replicated in a scaled-down system’s behavior and components and have a concrete
version inside a laboratory. Furthermore, during attacks idea. To provide exhaustive documentation, it is essential
targeting the process’s stability, even the less critical to write it step-by-step during the testbed building pro-
operation can express important safety issues [135]. cess, avoiding writing it after the testbed is entirely built,
• Complexity: Industrial systems devices can be hard to which can be difficult and not cost-effective [92].
configure and maintain due to their specificity and be- • Reproducibility: Due to the complexity of an ICS,
cause they are designed to perform a precise and unique it could be challenging to reproduce the experimental
 19

conditions of another research to replicate the results or makes it possible to visualize and control the systems. The
test other solutions. The differences between the original industrial protocols employed are Modbus, EtherNet/IP, and
conditions and the reproduced one can be minimal but, in PROFINET.
some cases, can be sufficient to lead to different results. Electrical Power and Intelligent Control (EPIC) [89],
To facilitate the deployment, experiment-management [90] is a high-cost 72kVA electric power testbed that mimics
systems can help researchers with the setup and the man- a real-world power system in small scale smart-grid, and it is
agement of a testbed (e.g., [136]) by using template or available for rent. The testbed is shown in Figure 8 and it is
code generation. Moreover, scripts for auto-configuration composed of four stages, namely: Generation, Transmission,
of an emulated testbed can be offered by developers Micro-grid, and Smart Home. Each stage is controlled by
(e.g., [121]) to simplify the sharing process. However, PLCs connected to a master PLC using switches and then to
suppose the testbed is composed of physical processes a SCADA gateway. The physical process is entrusted to two
and components. In that case, it could be difficult to motor-driven generators, photovoltaic panels, and a battery.
perfectly replicate them since many external variables can Communications occur using the IEC 61850 standard protocol
influence the system behavior (e.g., the temperature, the for the electrical substation and automation system that runs
pressure) [137]. over TCP/IP stack. The authors also present false data injection
• Scalability: If expanding a simulated or emulated testbed attacks, malware attacks, power supply interruption attacks,
is generally straightforward, doing it with a physical and physical damage attacks, together with possible mitigation
testbed can be challenging. Real devices are expensive, techniques. The testbed resides at the Singapore University
and researchers are not always able to afford them. of Technology and Design (SUTD), and it is used to supply
Alternatives to expand physical processes are Hardware- power to two other testbeds inside the same institution (i.e.,
In-the-Loop (HIL), i.e., mathematical representations of SWaT [99], and WADI [103]) to create also the possibility for
physical processes inserted in the chain. HILs offer research related to a cascade-connected ICSs. The authors also
great scalability of the system even if generally they shared a related dataset, which will be analyzed in Section VII.
are not advisable due to the lack of accurate mathe- HAI Tesbted (HIL-based Augmented ICS) [91], [138] is
matical models. A cheap way to add new devices is an extensive and expensive interconnection of three indepen-
to employ software simulations. Software simulations dent real ICSs coordinated by a real-time Hardware-in-The-
are cost-effective solutions with the drawback of less Loop (HIL) developed at The Affiliated Institute of ETRI,
precise and reliable physical representation. To provide Republic of Korea. Emerson’s boiler control system, GE’s
system scalability and intelligent reconfiguration of all the turbine control system, and FESTO’s water treatment control
physical devices implemented, virtualization and VLANs system are built-in small-sized by employing components used
can be an excellent solution to be implemented in ICS in industrial environments. The HIL is used to simulate the
without any substantial disadvantages [135]. power plant to combine the three control systems and form
• Data Collection: A not trivial aspect of building a an integrated power generation system. The interconnection
testbed is the data access and recording. It is generally employs Ethernet at Level 2, while different proprietary
a manual process, but it is vital to develop strategies Fieldbus versions are used to communicate with the field
to automate the collection precisely, providing reliability devices [139].The authors’ developed a tool to schedule HMI
and synchronization between the different data collection tasks for long periods without human intervention. This tool
points, for example, by introducing a central historian also helps to schedule attacks (e.g., MitM attacks) only when a
server. particular ICS state occurs. In [91] the authors present various
physical attacks targeting the pump and the pressure of the
boiler system. An expansion of the testbed [138] was built to
D. Physical Testbed make it possible to launch also network attacks using tools
Ahmed et al. [86] presented a physical testbed built at like Nessus or Acunetix.
the University of New Orleans, which models three industrial BU-Testbed [88] is a physical reproduction of two power
processes on a small scale but by employing real-world equip- generation systems developed at Binghamton University. The
ment such as transformers and PLCs. A small gas pipeline first one is composed of an AC motor directly coupled to
that transports compressed air was built using a pipe fed a permanent magnet DC motor, generating up to 400V. The
with an air compressor. A valve regulates the other end of other one instead contains an AC motor used to drive a 12-
the pipe. Instead, the second system is a power transmission volt DC blower motor used to generate electricity. The testbed
and distribution that carries electricity from power generation also includes two types of Alley Bradley PLCs and a private
sources to individual consumers. This system is composed computer with an LCD monitor used as HMI. The communi-
of a power station and four substations. Finally, the third cation uses the EtherNet/IP protocol. Furthermore, the authors
system developed is a wastewater treatment system composed explain some cyber-physical attacks which are practicable on
of sedimentation, aeration, and clarification processes. All the the testbed. These attacks regard different categories: 1) attacks
systems are installed at the top of a trolley, making the testbed on networks (i.e., MitM, DNS poisoning); 2) network conges-
easily transportable and particularly suitable for pedagogy and tions and delay (i.e., DoS); 3) attacks on controllers, sensors,
research. Each system is controlled by one PLC connected and drivers (i.e., malicious software injection and firmware
through a switch to a historian and an HMI. This last device modification); and 4) attacks on HMI and programmable
 20

stations (malware injection). In another work [140], Korkmaz to time delay attacks has been evaluated. Results show the
et al. presented a similar testbed in which the vulnerability feasibility of such attacks, which can stop the power generation
 21

power loop can be isolated and reconfigured for independent,

specialized testing. As planned in 2017, the authors obtain
more funds to expand SBTB with a SCADA testbed to be
installed in the command and control shelter to allow operators
to observe, manage, and manipulate test line configurations
and record testbed operating parameters. However, to the
best of our knowledge, the authors never release updates
about the project. This testbed is not an ordinary scaled-
down version of real systems. Instead, SGTB is a full-size
plant. Even if it represents an impressive and valuable work,
unfortunately, students and researchers have limited access to
Fig. 8: EPIC Testbed by iTrust in Singapore. such a facility [96].
SWaT [99], [143] is a six-stage water treatment plant devel-
oped by the Singapore University of Technology and Design
process and shut down the testbed. (SUTD) represented in Figure 9. One PLC (plus one for
Lancaster’s testbed by Green et al. [92] at the Lancaster backup) controls each stage, and the overall testbed leverages a
University is a big physical scaled-version of a generic in- distributed control approach. Furthermore, through a Human-
dustrial ICS (the testbed does not explicitly the physical Machine Interface (HMI), an operator can manually control
processes involved). It is composed of six Manufacturing all the system components. Communication between PLCs
Zones, a DMZ, and an Enterprise Zone. Each core zone is and sensors/actuators are based on Ethernet ring topology,
split at the network level using VLANs. The legacy serial- while PLCs communicate with each other through a separate
based communications have been upgraded to IP to reduce network based on an Ethernet star topology. The protocols
the complexity and allow communications with a vast number implemented in the systems are EtherNet/IP and Common
of ICS devices. The connections are almost all physical, apart Industrial Protocol (CIP). In the paper, the authors imple-
from two manufacturing zones connected using 3G, 4G, and mented various attacks to manipulate plant operations. The
satellite communications. To account for a changing landscape different attacks leverage different assumptions on the attack
and to add flexibility, all the desktop and server-based software model. In particular, the attacks are categorized as single-
applications run inside a VMWare vSphere server as virtual stage attacks, targeting a specific stage process and multi-
machines. The authors are continuously improving the testbed stage attacks, which combine the compromising of various
to make it more usable and more complete. Students and stages. Furthermore, each attack may target a single system
researchers of the university use the testbed, but the authors point or multiple system points. The attacks include different
also plan to make it more available for external researchers. scenarios (e.g., an attacker with access to the local plant
Morris et al. [95] at the Mississippi State University built communication network or an attacker who is on-site and
seven different small physical testbeds for security research has physical access to the device) and different types of
and pedagogy purposes. Five of them have communications attacks (e.g., MitM, eavesdrop, or packets modification). The
based on Modbus/ASCII, Modbus/RTU, and DNP3 (hence- testbed is accessible only for collaborations or by renting it.
forth called Mississipi Serial) and represent respectively: 1) Recently, a python-based software simulation of the testbed
a gas pipeline used to move petroleum products to market; 2) was developed and released with open-source code [144],
a storage tank used in the petrochemical industry; 3) a raised [145]. Also, datasets based on different data collection are
water tower used to provide pressure in the water distribution openly available upon request. These datasets contain both
system; 4) a factory conveyor belt control system, and 5) network and physical packets in normal behavior and with
an industrial blower used to force air through an exhaust the system under attacks [146]. We present the dataset in
system. These five systems are controlled by the same HMI Section VII.
but on different screens. It enables the control of all the Teixeira et al. [101] implemented an ICS testbed to model
systems from the same point and simulates a more extensive a simple water storage tank’s control system. The storage tank
system by making them operate simultaneously. The remaining is equipped with two-level sensors to control the water level.
two testbeds are connected through an Ethernet network (and When it reaches the maximum level, the upper sensor sends
then are called Mississipi Ethernet) and include: 1) a steel a signal to the PLC, which turns off the water pump used to
rolling operation; and 2) a smart grid transmission system. fill the tank. At the same time, another pump is activated to
The authors also use the testbeds to generate datasets that are draw water from the tank. When the water reaches the lower
freely available online [141]. sensors, a signal is sent to the PLC, which will reverse the two
Smart Grid Test Bed (SGTB) [98], [142] deployed by pumps’ state to fill up the tank again. The SCADA system gets
Idaho National Laboratory is the world’s first full-scale repli- data from the PLC using the Modbus protocol and displays
cation of a smart grid, and it is part of the United States them to the system operator through the HMI interface. To
National SCADA Test Bed Program. It is a 61-mile transmis- complete the study, the authors tested some attacks such as
sion massive testbed connecting twelve facilities with power scanning, device identification, and not authorized read of
distribution networks that can selectively operate at various actuators. By recording SCADA network traffic for 25 hours,
voltages (12.47kV, 24.9kV, and 34.5kV). Portions of the a dataset has been released [147] and will be presented in
 22

request is available. We will analyze the dataset in Section VII.

In 2014, Yang et al. [105] proposed a physical SCADA
power grid testbed specific designed to test their detection ap-
proach. At the control network level, the testbed is composed
of an HMI, a database to log events and data, a host used
to perform the attacks, and different networking components
(e.g., protocol gateway, switch, firewall, router). Instead, the
physical network is composed of various IED simulated, con-
nected to a real photovoltaic system. The connections between
the Gateway and the IEC devices are based on the IEC 60870-
5 series protocol, and then the Gateway translates the IEC
60870-5 to allow the communication with the HMI station.
The IDS proposed by the authors was installed between the
Fig. 9: SWaT Testbed by iTrust of Singapore. HMI and the Protocol Gateway. It monitors all the incoming
connections to the substation and the LAN network through a
port mirroring.
Section VII. In 2019, minor improvements of the testbed had Zhang et al. [106] presented a security research on a
been presented [148], such as embedding a turbidity sensor physical process ICS testbed which simulates a two-loop
and a turbidity alarm to add analog input to the system. nuclear power system. The primary loop includes a 9kW
Turbo-Gas Power Plant (T-GPP) testbed [102] is an ex- heater representing the reactor core, controlled by the SCADA
perimental platform presented by Fovino et al. at the Joint master through an open-loop controller. It also contains a
Research Centre of Ispra (Italy) to perform security research variable speed coolant pump, upper and lower delay tanks, and
on a SCADA system. It is a physical testbed that replicates other instrumentation such as a flow meter and temperature
a power plant’s dynamics process and its control systems detectors. The secondary loop is composed of valves, a mag-
providing additional mechanisms for running and analyz- netic flow meter, and two temperature detectors. The SCADA
ing the system. The testbed is composed of seven different system consists of an engineering workstation as the SCADA
functional elements: 1) Field Network, used to link PLCs master and a National Instruments chassis used to read data
with the SCADA servers, actuators, and sensors; 2) Process and control signal output modules as SCADA slave. The
Network, that interconnects the different physical subsystems; system is completed with data storage and an attacker machine
3) Intranet, the internal private network connecting PCs and with Kali Linux. LabVIEW was installed on the engineering
server of the company; 4) Demilitarized Zone, used to separate workstation to record sensor data and send control commands
IT area from OT components; 5) External Network, such to actuators. In the same paper, the authors proposed some
as the Internet; 6) Observer Network, a network of meshed attacks to the testbed (e.g., MitM, DoS). Furthermore, they
sensors to gather a massive quantity of raw data useful for the implemented some intrusion detection mechanisms based on
analysis; and 7) Horizontal Services Network, used for the Random Forest (RF), k-Nearest Neighbors (KNN), and Auto-
management of the laboratory. The paper profoundly analyzes Associative Kernel Regression (AAKR).
such systems’ vulnerabilities, highlighting those related to the
protocols implemented (i.e., Modbus/TCP and DNP3), and E. Virtual
describes various attacks deployed on the testbed: DoS, worm, Davis et al. [107] is a power grid simulated testbed based
and malware infection on the process network, phishing attack, on a client-server paradigm. The client mimicked a control
and local DNS poisoning. Finally, the authors propose different room’s graphical interface containing SCADA data and used
countermeasures to the attacks. it to control power elements. Each client can switch between
WADI [103], [149] is a scaled version of a water distribu- different servers to monitor several systems from the same
tion testbed build by the Singapore University of Technology machine. The most common operating systems support client
and Design (SUTD) to perform security researches. It consists software. On the other hand, the server is based on the
of five stages controlled by three PLC and two RTU, which PowerWorld [150] simulator and can model a complex power
can supply 10 US gallons/min of water. The communication grid. The server sends the process data to the client via a
happens using Modbus/TCP protocol at Layer 0, while at custom TCP/IP protocol, converted to Modbus/TCP using an
Level 1 network between PLCs uses TCP over Ethernet instead integrated protocol converter. Furthermore, the simulator can
of RTUs that exploit High-Speed Packet Access (HSPA) using connect and interact with real hardware devices, but it is
GPRS modem to generate a precise real-world scenario. The not mandatory. The network is emulated using RINSE [151]
authors also implemented different attacks against the testbed which allows clients to launch different commands to simulate
by manipulating data from sensors to cut off the consumer attacks (e.g., DoS attacks), defense techniques (e.g., filtering),
tank’s water supply. The system is physically connected to diagnostic tools, device controls, and simulator data. The
SWaT, and it can be used to generate a more accurate scenario authors present various attacks, such as DDoS and network
and study the cascade effects of a cyber attack on connected overload, comparing the results with and without security
ICS. Furthermore, WADI is available to organizations for joint measures. To the best of the authors’ knowledge, the testbed
research programs and usage, but a dataset generated upon is not available online.
 23

In [108] the authors present Damn Vulnerable Chemical software like Simulink [155]. Other implementation strategies
Process (DVCP), an open-source framework developed for for the system architecture are possible, like the federated
cyber-physical security experimentation based on two models simulation-based, where a different machine simulates each el-
of chemical processes. In particular, the framework includes ement. Moreover, emulation-based and implementation-based
DVCP-TE and DVCP-VAC, two simulated ICS testbed based instantiations that use actual commercial SCADA devices
respectively on Tennesse-Estman [152] and Vacuum-assisted along with simulation and emulation of software modules,
closure (VAC) [153] chemical processes simulated with Mat- network, and physical processes are depicted but not imple-
lab. The authors use these simulation models in hybrid sce- mented. Finally, the authors depict various possible attacks
narios with the simulated process and real industrial hardware (e.g., DoS, integrity attacks, phishing attacks) and suggestions
(i.e., SIMATIC S7-1200+KTP400 Starter Kit). Furthermore, about security mechanisms. To the best of our knowledge, the
the Modbus and PROFINET protocols were implemented to testbed is not publicly available online.
enable communication between the simulated process, the GRFICS [114] is a graphical and open-source [115] ICS
PLC, and the HMI. However, for this implementation, the simulation tool based on the Tennessee Eastman process
authors did not share any code or further implementation (Figure 10). Currently, the testbed is designed for educational
information. purposes and allows only the use of pre-defined functions. The
Genge et al. [112] proposed a framework based on Em- ICS devices are simulated. In particular, the OpenPLC [156]
ulab [154] for the emulation of the components and to is used for the PLCs, and the HMI Virtual Machine sim-
Simulink [155] for the physical processes simulation. The ulated an HMI using AdvancedHMI [157] software. The
architecture comprises three layers: the cyber layer containing testbed allows running many pre-defined attacks such as MitM,
the regular emulated ICT components used in SCADA sys- Command Injection, False Data Injection, Reprogramming of
tems, the physical layer providing the simulation of physical PLCs (i.e., Stuxnet), Loading Malicious Binary Payload (i.e.,
processes, and the link-layer to connect the cyber and physical TRITON), and Common IT attacks (i.e., password cracking,
layers through the use of a shared memory region. The buffer overflow). Once the attacks are launched, the interface
paper provides a qualitative comparison with other works, allows monitoring the testbed attacks’ consequences, log the
comparing the testbed with other related projects. Results show process information, and how much cost is wasted through
high performances in all the functionalities considered (e.g., the purge. Finally, the testbed allows the installation of the
repeatability, safe execution), except for the physical layer Snort detector [158] and to customize it with new rules.
fidelity, where physical testbeds perform better. Furthermore, The communications on the testbed are based on Modbus
estimating the cost to build and maintain a physical testbed protocols.
is compared with the predicted expense related to the pre- Maynard et al. [119] proposed Maynard SCADA, an open-
sented framework, showing considerable savings through the source, scalable framework for deploying a replication of a
years. A peculiarity of this framework is the possibility to SCADA network. The testbed is composed of a collection
attack the different components using specific malware. For of scripts used to build and configure virtual machines that,
instance, as a case study, the authors present Stuxnet [6] on a by default, are emulated using Oracle VirtualBox [159]. The
boiling water power plant, showing its effectiveness. Another resulting network can also support and integrate the connection
attack example targets a chemical process by deleting and with physical devices. Maynard SCADA supports IEC 60870-
delaying some packets and changing the process parameters to 5-104 (IEC104) and OPC Unified Architecture (OCP-UA) to
reach their shut-down safety limits. There are many supported support additional industrial protocols such as Modbus or IEC
protocols such as Modbus, Profinet, and DNP3. The testbed, 61850. The framework implements two types of profiles: an
implemented in C#, is not available online to the best of the operation profile, which defines the deployment of nodes, sim-
authors’ knowledge. ulators, and configuration of the network; and a configuration
Giani et al. [113] developed a virtual SCADA testbed profile to configure nodes to represent specific industrial de-
for security-related researches purposes. However, this work vices(e.g., HMI, RTU). Such profiles can be developed by the
represents a preliminary study presenting the testbed at a high- community, adding new use cases and simplifying the testbed’s
level, but without a practical implementation description. At deployment. The framework does not consider the physical
the center of the architecture, there is the SCADA master sta- process simulations, but it can be easily integrated using
tion containing the SCADA server and the HMI. The SCADA third-parties software (e.g., Simulink [155]). Furthermore, the
master station containing the SCADA server and the HMI is paper [119] shows a common metering application using seven
placed in the architecture center. SCADA master servers run virtualized nodes with detailed instructions to replicate it.
the server-side applications that communicate with the RTUs The instructions also include an accurate description system’s
using different strategies: dial-up modems, private leased line, requirements and a comparison between some other testbeds
wireless or radio channel, and LAN/WAN links. The most used in the same document. The framework is entirely open-source,
protocols for these communications are Modbus and DNP3. and it is accessible on GitHub [120], where are also available
The SCADA server is also connected to the corporate network, some datasets.
connected in turn to the Internet, exposing the system to vul- MiniCPS [121] by Antonioli and Tippenhauer is a toolkit
nerabilities, such as unauthorized remote access. The authors used to create an extensible and reproducible research envi-
planned to employ a single simulation-based instantiation to ronment for network communications, control systems, and
build all the testbed elements in the same machine using physical layer interactions in CPS. MiniCPS is an exten-
 24

Reavers & Morris [123] develop an open and complete

platform for creating virtual testbeds. The resulting system
is highly scalable, and it is possible to install plenty of
different virtual devices. The testbed’s main components are
process simulators, data loggers, and configuration files used
to configure virtual devices and connections among them. All
the simulations are implemented with Python without adopting
off-the-shelf network simulation tools. The process simulator
includes four components: 1) a simulator module, 2) a com-
munication interface, 3) an update queue, and 4) configuration
files. The simulator communicates directly with the virtual test
devices via a “backchannel” to transmit measurements and
inputs. The virtual devices supported are RTU, MTU, IED,
Fig. 10: Example of GRFICS simulator rendering. PLC, repeaters, and Programmable Automation Controllers
(PAC), which can run as standalone processes or inside virtual
machines. It is possible to connect physical devices such
sion of Mininet [160], a widespread network simulator built as wireless radios and HMI. There are two main protocols
around the Software-Defined Networking paradigm that ex- for the communications: Modbus/TCP, which can be logged
ploits lightweight system virtualization using Linux containers. using standard applications such as Wireshark or tcpdump;
Connections between simulated devices are emulated using and Modbus/RTU, which instead need a PortLogger, a class
virtual Ethernet links with an easy drag and drop interface. of proxy that reads the communication and then resends it to
These connections can be configured through Linux Traffic the channel. In the paper [123], the authors present two testbed
Control to emulate link performance such as delay, loss applications. The first represents a gas pipeline, while the
rate, and bandwidth. MiniCPS extends the classic Mininet by second models a water storage tank control system. To verify
implementing ICS components such as PLCs and allowing the simulated data’s consistency, the authors implement these
the connection with real physical devices. The testbed is two testbeds as physical ones. Furthermore, to obtain more
not focused on the physical process simulation that can be relevant results, the authors decided to compare the virtual and
implemented using third-parties process simulation engines physical testbeds’ behaviors during different attacks (e.g., data
(e.g., Simulink [155]). The reproducibility is a significant injection and DoS). Results show that the attacks are effective
advantage of this testbed: it is possible to write Python scripts in both scenarios, with some slight variations on time needed
that generate a complete ICS environment easily exportable for the attack to succeed. The paper also compares both the vir-
and shareable. On the top of the emulated Ethernet network, tual and physical systems’ normal behavior discovering many
the testbed includes different industrial protocols, in particular, similarities, but with some detectable differences. The study’s
using the CPPPO Python library, MiniCPS implement, for conclusion states that the virtual testbed is good for proof-of-
example, EtherNet/IP and Modbus/TCP. The paper [121] ac- concept, but a physical testbed is needed in some cases. To
curately describes all the design decisions and the consequent the best of our knowledge, the virtual platform is not publicly
strengths and drawbacks of the testbed. The authors present an available online, but some datasets are available [141]. Starting
attack scenario of a MitM attack on a replicated model of the from this work, Thornton and Morris in 2015 [161], deployed
SWaT testbed [99], also providing different countermeasures a similar platform that permits the usage of Simulink [155]
based on a custom SDN controller. The testbed and its instead of Python to simulate the physical processes.
documentation are open-source and available on Github [122]. RICS-el testbed [124] is a virtual testbed representing a
Morris et al. [111] presented a virtual gas pipeline system power system built on top of the Cyber Range And Training
(called Gas Pipeline testbed) that is a simulation of a testbed Environment (CRATE) infrastructure at the Swedish Defence
previously built. The testbed consists of four components Research Agency (FOI) [162]. All the hosts of the testbed are
running in different virtual machines: a virtual physical pro- run on virtual machines using VirtualBox [159]. Researches
cess, a Python-based PLC simulation, a network simulation, and vendor experts designed the OT segment. It is divided
and an HMI. The various components communicate through into the OT DMZ, the OT LAN, the substation communication
Modbus/TCP over a virtual network and may be connected WAN, and the power grid simulator, including all the RTUs.
to real devices. The virtual system allows modeling a pump, In the OT DMZ, there are the FTP server, the HMI, and
a valve, a pipeline, a fluid, and a fluid flow. The models are the historian. The WAN is used to enable communication
based on a previous physical testbed [95], allowing to compare between the 15 hosts. Three of these hosts are RTUs that
measures from the two testbeds. The virtual testbed mimics the communicate with the front-end through the IEC 60870-5-
physical device’s behavior but with some difference in pressure 104 (IEC104) protocol. The power grid simulator is the key
change frequency. Also, the startup process is similar but not component of the architecture: it can generate realistic traffic
identical. The authors present a command injection attack to and event in the whole RICS-el environment. In detail, this
the virtual testbed, but the resulting behavior is not compared testbed simulates a backbone high voltage 400kW grid with
with the physical testbed. To the best of our knowledge, this twenty substations and some medium voltage transmission.
virtual simulator is not publicly available online. Finally, to add more realist to the environment, the system is
 25

connected through another DMZ to an office IT segment. It various attacks, including spoofing, MitM, DoS, and data
contains a LAN to interconnect 17 office workstations, nine injection. Two of these attacks scenario are also implemented:
sales workstations, and some other servers. Ongoing work a DoS attack and a compromised HMI scenario to force
is focusing on adding realistic traffic to each segment by a complete network blackout. The protection system under
emulating users and different scenarios. testing was able to detect the two launched attacks.
SCADASim [127] is a simulator for SCADA systems Virtual Tennessee-Eastman Testbed (VTET) [60] is a
created on top of OMNET++ [163]. The testbed is developed simple virtual testbed that simulates a chemical ICS with
to satisfy specific requirements: 1) it allows plug-n-play to Matlab. The architecture is based on four components: a
create simulations to allow system experts to set up the physical PLC, a PC used for network communication, and
software; 2) allows connectivity to multiple external hardware two other PCs simulating the physical process and a PLC.
or software that can be used to expand the simulator; and 3) The process is the Tennessee-Eastman (TE) [152], a non-
supports multiple industry-standard protocols such as Mod- linear and continuous process widely used in the chemical
bus/TCP, DNP3, and the integration of proprietary protocols. field. VTET can work in two different modes. In the full-
The simulator contains modules for the emulation of ICS virtualization mode, the physical PLC is disconnected, and
devices (e.g., RTU, PLC, MTU, HMI) and components to the testbed is completely virtual and simulates the controller
implement different attacks (DoS, MitM, spoofing, eavesdrop- using NetToPLCSim [168] and PLCSim [169], the official
ping). SCADASim architecture includes three components: 1) simulator of Siemens PLC. Instead, the semi-virtualization
a real-time scheduler; 2) a communication port implementing mode allows replacing the simulated PLC with the real one.
protocols for communication to the external environment; and VTET supports three standard ICS protocols for network com-
3) a simulation object that models external components within munication: Open Platform Communications (OPC), Modbus,
the simulation environment. For the evaluation, the authors and S7Comm. The authors present and test five attacks, mainly
present two simulations: a smart meter and a wind power plant. using MitM and jamming techniques to disturb or disrupt the
A DoS attack and a spoofing attack are also deployed on the physical process. Unfortunately, the testbed is not available
systems and analyzed in the paper. SCADASim is an open- online to our knowledge, but the description is quite complete
source project available on Github [128]. on the paper.
SCADAVT [129] is a framework to build a virtual SCADA
model-based testbed designed for research in the security
field purposes. The framework is developed on top of the F. Hybrid
CORE emulator [164] by integrating the Modbus/TCP com- CyberCity Testbed [64], [170] is a physical representation
munication protocol between master, slave, and HMI server. of an entire city (Figure 11) developed by the SANS Institute
Simulations of I/O modules are also integrated into the CORE to test security measures on the ICS field. It includes a bank
emulator, which acts as a server, receives input data from the simulation, a hospital, a power plant, a train station, a water
external environment, and sends output data when requested town, and many other available infrastructures. Furthermore,
using a simple custom TCP-based protocol. The physical 15k “people” who have e-mail accounts, work passwords, and
process is modeled using an EPANET server [165] which bank deposits are generated to create a complete environment.
provides a graphical interface to reproduce water distribution A tabletop scale model of the town was built to visually
systems. Two attack scenarios are also presented: a DoS and show the effects of attacks on the electric train, the water
manipulation of command messages. The framework, which tower, and the traffic light. Although the lack of official
supports real devices’ connection, is described in detail, but documentation, Borges Hink et al. in [64] recover some details
the source code is not publicly available. about the components of the testbed by studying a dataset
TASSCS (Testbed for Analyzing Security of SCADA Con- generated from CyberCity, which is available online [171].
trol Systems) [131] was developed by the University of They discover a wide variety of components ranging from web
Arizona mainly to test a novel technique to protect SCADA server emulated using VMWare to physical Siemens PLCs,
systems from attacks, which the authors called Autonomic Cisco routers, and NetDuino+ controllers. The protocols used
Software Protection System (ASPS) [166]. The testbed archi- by the ICS components are mainly Modbus/TCP, EtherNet/IP,
tecture is composed of different components. The Control HQ and NetBIOS. Nowadays, the testbed is mainly used to teach
is the central command and control for all the resources and cybersecurity on ICS as part of the SANS Institute courses
services offered. It contains the HMI, the control server, the and federal agencies to perform security research.
data storage, and the engineering LAN. Through a WAN, the Experimentation Platform for Internet Contingencies
Control HQ is connected to a large scale electric grid modeled (EPIC) by Siaterlis et al. [65] is an innovative hybrid testbed
using the PowerWorld simulation tool [150]. Finally, a device to simulate CPSs based on Emulab [154]. It is developed by
is used to monitor all the ingress and egress communications the Joint Research Center at the Institute for the Protection and
that pass through the WAN to feed the ASPS. This last device Security of the Citizen in Ispra, Italy. The testbed architecture
acts as an active anomaly detector: it can identify attacks and comprises two control servers, a pool of physical resources
stop them. To enable communication, Modbus/TCP is used. used as experimental nodes (e.g., PCs, routers), and a set of
The Modbus Server is simulated using Modbus RSim (the switches employed to interconnect the nodes. Every config-
official website is not anymore available) and connected to uration step uses a web interface where a user can create a
an Opnet-based network simulator [167]. The authors present customized network. The EPIC setup phases require a detailed
 26

13MW. The cooling system cycles through over 4000 gallons

of water each minute. The developed replica is based on the
same controller, an Allen Bradley Control- Logix PLC with
34 I/O modules distributed over six chassis connected with an
Ethernet/IP ring-topology backbone. Furthermore, the HMI,
the historian, the industrial switches, and the power supplies
are perfect physical replicas. An engineering workstation is
connected to the system to configure the different components.
On the other side, the over 500 sensors and actuators employed
in the cooling system are instead emulated by using over
40 Raspberry PIs and 200 daughter boards. All the sensors
and actuators communicate with the PLC using hard-wire
electrical signals or an Ethernet-based signal line. For this
last case, raw traffic from the production environment has
been recorded. A software-based model of the protocol, traffic
Fig. 11: A figure representing CyberCity testbed.
rate, and handshakes of the real cooling system was computed
and employed by the Raspberry PIs to emulate the entire
communication. The authors collected 30 days of data from
description of the required topology using a formal language the real Summit cooling system historian to correctly emulate
(i.e., an extension of Network Simulator (NS) language). The sensors and actuators. Then they used emulation scripts to
experiment is then instantiated by using Emulab [154] which generate data from the devices. Each sensor and each actuator
can automatically configure network switches to recreate the is connected to an independent display used to verify the
desired virtual topology by connecting nodes using multiple correct measures, despite the HMI values. This is useful in the
VLANs. Finally, experiment-specific software can be launched case of attacks targeting data visualization (e.g., replay attack).
through events defined in the setup script or manually by To validate the testbed, the authors compared its behavior with
logging in to each station. Physical processes are simulated the real Summit supercomputer cooling system. Considering
using Simulink Coder [172] and managed by a software the alerts, logs, and historian data, all the data are replicated
simulation unit. For the communications, EPIC provides tools accurately. Instead, concerning the network traffic consistency,
to generate latencies for the simulation of different network results show an hour-to-hour average variance under 0.01%
types and integrate realistic background traffic datasets. It for the majority of the properties. Therefore, the fidelity is
also supports industrial protocols such as Modbus through adequately accurate to simulate the original system properly.
proxy units that translate calls between the simulation unit Hui et al. [69] introduce Hui Nuclear, a hybrid testbed
and other SCADA devices. Furthermore, after a theoretical modeling a nuclear reactor built at the Center for Secure
comparison between fidelity, repeatability, and measurement Information Technologies (CSIT) at the Queen’s Univerity of
accuracy between EPIC and other popular testbeds, these Belfast. The testbed’s scope is to generate a realistic network
characteristics are analyzed on EPIC with a deep testing phase. interaction and a simple way to collect network data to be
The software part is open-source and freely available online used in the CPS security field. The testbed implements four
[66] with complete documentation. main sub-process controlled by four PLCs. The main reactor
EPS-ICS [67] is a framework to implement a hybrid sub-process, the heat exchanger sub-process, and the heat
testbed, principally developed by the Technical Assessment exchanger sub-process controlled by physical Siemens PLCs.
Research Lab (CNITSEC) in Beijing, China. The testbed Instead, the generator sub-process is monitored by a Schneider
implements a multi-level design approach where Level 3, the PLC. The inter-communications between sub-processes are en-
corporate network, and Level 2, the supervisory control LAN, abled by physical interactions or IP network communications
are emulated. Instead, Level 1 devices, including Distributed through S7Comm protocol, Profinet, and a custom protocol
Control Systems (DCS) controllers, PLCs, and RTUs, are real based on TCP. For practical and safety reasons, the heating
physical devices. Finally, a mathematical model is used to process and the turbine are simulated by two Raspberry
simulate the physical process at Level 0, and it is implemented PIs. The network architecture exhaustive and contains all the
with Simulink [155]. This approach allows replicating the Purdue model areas, together with firewalls, IDS, and logging
interactions between the ICS components. The communication services.
interface between network testbed and physical devices is HYDRA [70] is a low-cost and open-source physical emula-
implemented through layer three switches with an IP routing. tor for critical infrastructures developed at the Università Roma
However, the industrial protocols used are not mentioned in Tre in Italy. It can be used for investigating fault diagnosis,
the relative paper. cybersecurity strategies, and testing control algorithms. The
Gillen et al. [68] presented a hybrid replication of the testbed is designed to emulate a simple water distribution
cooling system for Oak Ridge National Laboratory’s 200- system’s behavior. It employs seven tanks at the physical level
petaflop Summit supercomputer, currently declared the fastest deployed vertically. Each tank can be easily unconnected or
open-science computer in the world [173], [174]. Summit moved to another position giving the testbed high modularity
consists of over 4600 nodes and has a peak power draw of and flexibility. The communications between sensors and
 27

actuators implement the Modbus protocol on a Local Area based on open-source hardware and software, built at Masaryk
Network (LAN) to PLCs and RTUs simulated using Arduino University in the Czech Republic. This testbed consists of a
Nano and Galileo. The authors also present an attack scenario laboratory room designed to help computer science students
of a data modification attack. The code and all the testbed to learn cybersecurity in a simulated industrial environment.
technical details are open-source and available on Github [71]. The laboratory is divided into different tables to split the
Kim et al. [74] proposed a platform to perform cybersecu- students into groups and give everyone the possibility to
rity exercise for national critical infrastructure protection. The have hands-on experience on the entire system. Tables can
testbed was designed to replicate a realistic ICS environment be moved and rearranged around the room to generate a
that matches the characteristics of the Cyber Conflict Excercise flexible environment for every possible activity, ranging from
(CCE). CCE is an annual national real-time attack-defense team assignments to student presentations. A control panel
battlefield competition organized in South Korea and Locked exposes the I/O modules on each table, and the touchscreen
Shields (LS). It is the world’s largest international technical is used to interact with PLCs (simulated using Raspberry
live-fire cyber defense exercise. The platform can scale and Pi [175]), linear motor, and communication gateway. The
provide dozens of identical ICS setups to satisfy an increasing software stack includes the Linux OS, Docker ecosystem,
number of participants. With respect to standard testbeds, and on-premise OpenStack cloud environment to achieve an
this project required a visualization layer representing the automated orchestration. Thank the open-source hardware and
physical facilities and the damage caused by the attackers. software used in the system, and different industrial protocols
To make it possible, a diorama city was considered the most can be implemented, such as Modbus or DNP3. Finally, the
cost-effective and modular approach. It contains symbolic paper introduces the university’s course syllabus that employs
structures representing the critical infrastructures, surrounded the facility, showing the arguments addressed on each of the
by residential and commercial buildings, and tri-color LED 13 weeks of the course.
lights to introduce a physical representation of attacks’ effects. LegoSCADA [77], [78] is a cost-effective hybrid testbed de-
The paper [74] describes an implementation of the proposed veloped at the Universite Paris-Saclay in France. The testbed’s
platform, which includes six different critical infrastructures: a conceptual architecture is based on three block elements:
power grid, a nuclear plant, a water purification plant, railroad the controller, the system, and the sensors. The controller
control, airport control, and traffic light control. The system reads data from the sensors, computes new information, and
contains two PLCs of different vendors that control some transmits new commands to the actuators. Many RTU and
typical actuators (e.g., mechanical relay, magnetic switch, PLCs can be connected to the controller based on the system
motor). Furthermore, a platform with 255 LED lights was that we want to represent. The protocols supported are Modbus
built to illustrate the state of the critical infrastructures. The and DNP3. To test the architecture, the authors have developed
control network layer is hosted by remote cloud servers and a test scenario based on Lego Mindstorms EV3 brick [177]
contains HMI, an engineering workstation, a historian DB, a which emulates a PLC on a car, a Raspberry Pi [175] to
patch management system, and office computers. The protocol emulate an RTU connected to the vehicle, and a personal
adopted depends on the selected PLCs. computer as a controller. The controller is always correcting
In [75], Koutsandria et al. presented a hybrid testbed the car speed and polling the distance between the car and an
for testing a real-time Network IDS. To simulate the ICS obstacle. Furthermore, a single RTU and a single controller
environment, the authors employ a combination of simulated can control more PLCs, and, therefore, more cars can be
and real devices. The testbed is based on Matlab Simulink to connected to the testbed. MitM attacks are deployed on the
simulate the physical and control networks. In particular, the developed testbed, in particular replay attacks and injection
authors model the physical system with Simulink by simulat- attacks. Moreover, a watermark authentication technique has
ing IED and field devices controlled by a PLC via Modbus in a been tested to stop the attacks with interesting results.
master/slave communication model. In this setting, the authors LICSTER [93], [94] is an open-source and open-hardware
describe the implementation of the master devices both with testbed presented at the Hochschule Augsburg in Germany. Its
a SIMATIC S7-1200 PLC and a simulated PLC. The network main target is to give students and researchers an affordable
communication and information exchanged by the different system to perform security research with an expense of about
device are collocated through a network tap implemented 500 euros. The system is composed of an OpenPLC [156],
with a central hub and a Raspberry PIs [175] running a an HMI built using a web server and a SCADA system. Each
packet dissector. The authors gave particular attention also to of these components is loaded on three dedicated Raspberry
the data management and visualization part. All the traffic PIs. The physical process implemented is a representation
collected is saved in a historian server and managed with of an industrial process provided by Fischertechnik [178]. A
OSIsoft [176] PI System. Then the historian information is conveyor belt is used to move a plastic cylinder to a punching
continuously analyzed and monitored by an IDS based on machine, which is then activated. In the end, the cylinder
rules and behavior analysis. To validate this architecture and is taken back to the original position. The process can be
its capabilities, the authors also present three attacks (i.e., two easily substituted with others. Modbus/TCP is the protocol
network communication alterations and a physical behavior used to enable communication between components. Different
violation) scenario showing the effectiveness of the detection attack scenarios on LICSTER are presented and tested. The
rules. authors cover widely used threats to levels 0, 1, and 2, such
KYPO4INDUSTRY [76] is a training facility for students as passive/active sniffing, Dos, MitM, and manipulation over
 28

the network. For each attack, an evaluation is presented con- depending on the PLC used. Some vulnerability discovery
taining useful information (e.g., impact, skill level, detection experiments have been done on MSICST, ranging from discov-
difficulty). Scripts and instruction on the implementation are ering vulnerabilities on a specific type of PLC to some attacks
available on the Github repository [94]. to known vulnerabilities of S7Comm and Modbus provided
Microgrid [79] is a flexible and adaptable testbed devel- by the lack of encryption and identity authentication. Some
oped by The Ohio State University, composed of a hybrid security measures are presented as well, like a whitelist-based
setup of physical hardware and real-time simulations. The host protection software and a new IDS solution that combines
testbed contains Power Hardware-In-the-Loop (PHIL) able to traditional IT system IDS with behavior-based ICS-specific
emulate power hardware not installed in the testbed, along IDS.
with a real-time SCADA system with an OPNET [167] based NIST (National Institute of Standards and Technology)
real-time System-In-the-Loop (SITL) communication network developed a cybersecurity testbed for ICS presented in detail
simulation system. PHIL can emulate several components like in [81]. The testbed is designed to emulate three real-world
stationary battery unit, charging station, renewable energy industrial systems without replicating the entire plant or as-
resources, 9-bus or 14-bus systems. It is also possible to sembling a complete system. The first system is a Tennessee
connect physical components to PHIL. The paper presents an Eastman (TE) problem [152], a widely used process in the
implementation of a 5kVA charging system of a simulated chemical manufacturing field. The TE process is simulated
electric vehicle, a photovoltaic system, local energy storage, using an open-source code [179], and it is connected to
and different power electronic circuits. The three main compo- physical devices such as switches, PLCs, HMI, and terminals
nents of the simulated SCADA environment are the data acqui- through different protocols such as OPC, Ethernet/IP, and
sition, the real-time virtual communication network, and the DeviceNet. The second is an entirely physical cooperative
real-time control center with the HMI. The authors introduce a robotic assembly system for smart manufacturing. It contains
case study implementation by connecting local energy storage a PLC, controllers, buttons for emergency stops, HMI, and
and a second power grid PHIL simulation. Furthermore, the two robots. These devices are interconnected through Ethernet,
authors validate the case study with experimental results and EtherCAT, Serial, Modbus, and Analog/Digital signals based
analysis. The testbed is designed to study topics related to on the components’ needs. Finally, the third simulates a
smart grid and provide hands-on experience to students. pipeline network with a Wide Area Network (WAN) SCADA
MSICST (Multiple-Scenario Industrial Control System infrastructure and an intelligent transportation system, includ-
Testbed) [80] is a hybrid representation of four different ing public infrastructure components, cooperative real-time
ICS scenarios: a thermal power plant, a rail transit, a smart embedded components, and wireless components. However,
grid, and intelligent manufacturing. Physical processes are this last testbed was only introduced in the paper and was not
always simulated while the control systems are built using implemented at the publication time (i.e., 2015). The testbed is
commercial hardware and software. Furthermore, in some available upon request to academia, government, and industry
scenarios, a combination of software simulation and actual to analyze new technologies. Based on the research on these
physical equipment is used to build a more realistic scenario. testbeds, NIST published a long and complete guide to ICS
MSICST also contains an attacker model and a monitoring security in 2015 [2].
network. The thermal plant comprises four PLCs of different PNNL [82] by Edgar et al. at Pacific Northwest Na-
manufacturers used to manage the three simulated physical tional Laboratory is a remotely-configurable and community-
systems: combustion system, steam-water system, and electri- accessible hybrid testbed to support research on cyber-physical
cal system. A sand table is synchronized with the simulation equipment. This testbed combines physical, simulated, and
to visualize what is occurring using Light Emitting Diode virtual components giving considerable implementation flexi-
(LED), fans, and smoke generators. The rail transit scenario bility. In fact, the testbed allows simulating from small systems
includes three stations, two trains, and a circular rail transit like traffic lights to extremely complex scenarios such as power
line. All the components are realized in a sand table as a grids. The testbed is composed of many different back-end
scaled-down version of a real system. To achieve automatic functionalities that can be employed in user management. Each
control of trains and station components, the authors use two user can remotely deploy its own system configuration and
Siemens PLCs. Regarding the smart grid, the testbed mainly manage the operation and data gathering process. Further-
focuses on the power consumption part. It contains two smart more, users can control different areas of the architecture,
meters, a concentrator, and a station device, which can, for including the environment (used to simulate the physical
instance, display the power consumption of an area. Finally, process), devices (e.g., PLCs, RTUs), network communication
the intelligent manufacturing scenario is based on a Computer (representing the backbone communication), simulation, and
Numerical Control (CNC) that contains a controller, memory, device integration. The testbed is accessible following the
and HMI. Moreover, a Distributed Numerical Control (DNC) indications provided in the PNNL website [180] and using
system was developed to improve the manufacturing industry’s Arion [181] as modeling software.
intelligence level. A DMZ containing the data historian and SNL Testbed [84] is a complex hybrid testbed built by
the HMI server is generated to separate the OT area from the Sandia National Labs in Albuquerque, USA. It contains
enterprise zone. This latter simulates an office by using a PC simulated components (i.e., represented using a model in
with Windows 7. The protocol used for communication in the OPNET [167]), emulated nodes (i.e., using real software
OT area are mainly Modbus/TCP, S7Comm, and EtherNet/IP, running on an emulated machine), and physical (i.e., real
 29

software running in real hardware) devices. The reference a dataset must contain both network traffic, representing the
paper also includes an accurate explanation concerning the communications between the various devices, and the physical
connection between the various components. The testbed is processes’ measurements.
presented as a case study used to model a complex scenario, Datasets are generally shared as csv, arff, or pcap format
containing: the corporate network (connected to the Internet), a files, depending on the typology of data collected. An inter-
DMZ, a control system network (containing HMI, the SCADA esting solution introduced by Morris et al. [183] consists of
Server, Engineering Workstation, and Front End Processor), providing also some datasets containing only a subset of the
and the field layer (containing sensors, RTUs, and IEDs). data. They can be used, for instance, to quickly look at the
The protocols implemented are Modbus/TCP, DNP3, and IEC data without downloading huge files or training a preliminary
60870. Finally, the authors present a security assessment of algorithm during the early stages of development.
the testbed considering different threats and attacks such as There are many ways to categorize datasets. For example,
reconnaissance, resistance to standard penetration tools (e.g., Choi et al. in [22] groups datasets based on attack path. In this
Metasploit [182]), and MitM. survey, we decided to divide datasets based on the typology of
VPST (Virtual Power System Testbed) [85] of the Univer- the collected data. The capturing can contain data at physical
sity of Illinois is designed to be integrated with other testbeds level i.e., field data such as measures from sensors, actuator,
across the country to explore SCADA protocols and equip- and other physical level devices, or network level data, i.e.,
ment’s performance and security. Thanks to its easy integration packet or flow sent in the channel under control. However,
with real devices and testbeds, VPST has the advantage of datasets can contain both the typology of data, and so they
having actual HIL and a faithful communication system. The are considered both physical and network level. Sometimes,
architecture is divided into three main subsystems: the first it is possible to find other types of data, like device logs, to
handle electrical simulation using PowerWorld [150], the sec- better understand the ICS’s behavior. To perform our study
ond simulates the communication systems using RINSE [151], and provide reliable statistics, we downloaded every dataset
and the third includes all the actual devices. Furthermore, a and analyzed it reporting the main interesting properties.
framework for the Inter-Testbed Connection (ITC) is integrated
Table IV summarizes the main features and statistics of the
with VPST. This framework is based on low bandwidth and
presented datasets. We reported the following features.
reliable control plane and a high bandwidth data plane. ITC re-
quires secure connectivity, which is achieved using OpenVPN • Name of the dataset (or of the authors if a name is not
and IPSec. Moreover, the implementation of performance, re- provided);
producibility, and resource allocation properties are addressed • Sector indicates the field of the source ICS;
in the paper. Fidelity is another essential property achieved • Data type provided. Can be
by implementing real industrial protocols such as DNP3 or – Logs if logging information of the system during the
Modbus, leaving the possibility of testing new versions and process are available;
protocols (e.g., DNP3SA that provides Secure Authentication). – Network if network traffic data are provided;
The paper presents some example use cases: attack robustness – Physical if measurements of sensors and actuator
analysis, incremental deployment analysis, and Human-in-the- states are available;
loop event analysis. However, thanks to its flexibility, the
testbed is suited for many different types of research. • Time provide an approximation of the duration of the
recording;
• Entries indicates an approximate number of entries con-
VII. ICS DATASETS
tained in the dataset. In case of datasets containing
In this section, we provide a description of the ICS dataset different versions, the most used or the most recent is
available in the literature, highlighting the key design point considered;
and the most interesting and performant IDS applied to them. • Reference includes a reference to a description of the
In Section VII-A we outline the classification method that dataset;
we use in the following sections, while in Section VII-B we • Resource indicates a webpage in which the dataset is
introduce the main requirements and challenges in developing downloadable or information about how to retrieve it are
a dataset. Furthermore, we summarized in In Section VII-C we available;
briefly recall the common evaluation metric for IDS. Then, in • Attacks specified the categories of attacks contained
Section VII-D we present the datasets offering only physical in the dataset, if any. Can be Reconnaissance, Replay,
level data, while in Section VII-E we describe network level MitM, DoS, Injection , or Others which contains less
datasets. Finally, in Section VII-F we highlight datasets con- used categories. More information about the attacks are
taining both the information. presented in Section V-A.
• % indicates the percentage of data under attack on the
A. Datasets Classification total entries, if any;
Datasets are a collection of data recorded from a testbed • Format indicates the format of the files containing the
or synthetically produces, which can be used to train and capture. Can be:
test an IDS. Unlike datasets concerning IT systems, which – pcap is a widely used format containing network
are composed only of network traffic, to characterize an ICS, packets;
 30

– csv is an extension for files containing Comma breaking these relations, we would leave a trace that an IDS
Separated Values; can exploit to detect anomalies, creating detection bias. Since
– log contains textual logging of events; this property is not present in real systems, the IDS will miss
– xslx is a format for spreadsheet files; most of the attacks in physical environments, reducing the
– arff is a format used to save data for databases in a detection generalization in other systems. It is one of the main
textual format. It is generally used with Weka [184]; problems of Lemay et al. dataset [185], which use tools such
– inp contains data of emulations. In this context, it as Metasploit [182] to inject the malicious traffic. Another
is generally used with Epanet [165]. critical concern causing the lack of available datasets from
• IDS contains a reference to the best IDS available in real environments (i.e., ICS of companies) is related to the
literature applied on the testbed at the best of authors collected data’s privacy. In fact, companies may be reluctant
knowledge; to share their internal configurations, intellectual property, or
• F1-score, Accuracy, and Precision represent the eval- proprietary protocols. Moreover, giving the public access to
uation metrics of the IDS specified, according to Sec- an industrial site data may allow malicious users to identify
tion VII-C. vulnerabilities and exploit them to attack the company. As
The detection algorithms selected are implemented on the a result, many datasets are generally generated from scale-
whole dataset and not on a fraction of it. Furthermore, the down testbeds and the few real ICS environments. Since
selection does not take into consideration the rank of the many intrusion detection techniques are supervised, a complete
publication venue of the paper. For some datasets and IDSs dataset must provide labels indicating normal or abnormal
were not possible to obtain all the information since the data. Furthermore, labels are essential as ground truth for the
related paper does not provide exhaustive information. Thus, evaluation of detection performances during the test phase.
the degree of depth of analysis may not be the same for all However, the labeling process is not always straightforward.
work. For example, some attacks can move the system in abnormal
behavior after a long time the malicious packets have been
sent. In this scenario, the data labeled as malicious should start
B. Datasets Challenges & Requirements when the actual attack starts or when the system’s behavior
There are several challenges in generating a valuable starts to be compromised? An analysis of this problem can
dataset. Therefore it is fundamental to create it by following be found in [186] and [185]. In both cases, the solution could
a suitable methodology and keeping in mind the design raise a problem in the ground truth. Therefore, there is no right
requirements. Gomez et al. [55] described a framework useful or wrong answer to this question. It depends on the context
to generate reliable anomaly ICS datasets to be employed and the attack type, but it must be specified in the dataset’s
in anomaly detection tasks. Firstly, it is important to select documentation to allow researchers to act accordingly.
a priori, one or more attacks that will be implemented. To
do so, researchers must know the main protocols used in the C. Evaluation Metrics
field of interest, discover the related threat, and design attacks In this section, we briefly recall the metrics used to evaluate
according to the related vulnerabilities. Then, attacks can be the performances of the detection algorithms. According to
deployed, carefully choosing the nodes affected, each attack’s the literature, the most common metrics are Accuracy and F1-
duration, and its starting time. Finally, it is possible to capture Score. They are defined as follows.
network packets and/or data from sensors and actuators: it • Accuracy: represents the fraction of correct predictions
is essential to define the data capture duration, the sampling of the model under consideration. In the binary classifi-
frequency and smartly choose the collecting point. Generally, cation case, the accuracy is defined in terms of positives
the latter should be a central node of the system. The last and negatives samples classified as follows:
step is the final dataset generation. To generate the dataset to TP + TN
release, it is important to carefully choose the features useful Accuracy = , (1)
TP + TN + FP + FN
to describe the system under consideration. The behavior of
where TP = True Positives, TN = True Negatives, FP =
the system can be represented at packet-level, flow-level, or
False Positives, and FN = False Negatives.
physical-level data. The deployment of attacks in datasets is
• F1-Score: is a metric used to evaluate a classification,
probably the most challenging phase. In fact, if not accurately
defined as the harmonic mean between precision and
performed, the attacks generated can lead to an inaccurate
recall as follows:
system representation or bias in the detection methodology.
precision · recall
There are principally two ways to generate attacks. The first F 1 − Score = 2 · , (2)
one, and the most accurate one, is to attack the testbed in real- precision + recall
time, recording the corresponding network traffic or the ICS’s where the true negative rate, or precision is:
physical state. Another strategy is to insert synthetic malicious TP
precision = , (3)
data, a posterior, in a dataset with regular operation. However, TP + FP
this strategy could lead to inaccuracies and may not accurately while the positive and negative predictive values, or
represent the real system behavior response. In fact, if we want recall, is:
to inject packets on a dataset with normal operations, we must TP
recall = . (4)
consider all the complex cascade relations of the systems. By TP + FN
TABLE IV: Summary of datasets presented in the literature. The Data type indicated as L: Logs, N: Network; P: Physical. Times of recording are estimations and measure
units are h: hours, d: days, m: months. Entries numbers are estimations, too. We denote the Attacks launched during the recording as RC: Reconnaissance; RP: Replay;
M: MitM; I: Injection; D: DoS; O: Others. The % column indicate the percentage of data under attack with respect to the whole dataset. File Formats are indicated as
P: pcap; C: csv; L: log; A: arff; I: inp; and X: xlsx. *: the version of WADI dataset considered is the one dated November 2019; the one of SWaT dataset is instead A1
dated 2015, the most used one as the best of the authors’ knowledge.

Name Sector Data Time Entries Ref. Res. Attacks % Formats IDS F1 Acc. Prec.
D5: Energy M.S.D. Energy Manag. L 30d 6M - [141] - - C - - - -
QUT DNP3 Power Grid N, L 40d 31M [56] [187] RC, RP, M, I, O ∼0.01 P, L - - - -
QUT S7Comm Mining Refinery N, L 17.5h 2M [188] [189] M ∼10 C, L, P - - - -
4SICS Generic ICS N 46h 3M - [190] unk unk P [191] ∼1 ∼1 -
CyberCity Dataset City N 16d 170K [64] [171] I, M, D, RC, O 16.58 P - - - -
D2: Gas Pipeline Gas Pipeline N - 400K [192] [141] I 0.97 C [192] 0.75 - 0.75
D3b: Water S. T. Water Storage N - 230K [183] [141] RC, I, D 27 A [193] 0.981 0.981 0.981
D4: New Gas P. Gas Pipeline N - 270K [111] [141] M, I, O 21.86 A [193] 0.988 0.988 0.988
Electra Modbus Power System N >12h 16M [55] [194] RC, I, RP 5.2 C [55] 0.987 - 0.988
Electra S7Comm Power System N >12h 387M [55] [194] RC, I, RP 1.42 C [55] 0.996 - 1.000
HVAC Traces HVAC N 7d 40M [195] [196] - - P - - - -
Lemay Covert Breakers N 6.55h 1.6M [185] [197] Covert Channel 100 P, C - - - -
Lemay SCADA Breakers N ∼6h 900K [185] [197] RC, I, O 3.29 P, C [198] 1.000 1.000 -
Modbus SCADA #1 Liquid Pump N ∼24d 41M [199] [200] M, D 4.81 P [201] 0.775 0.812 0.964
S4x15 ICS Generic ICS N <1h 310K - [202] unk unk P [191] ∼1 ∼1 -
WUSTL-IIOT-2018 Water Control N 25h 7M [101] [147] O 6.07 C [101] 1.000 1.000 1.000
D1: Power System Power System P, L - 78K - [141] I, O 71.02 C, A [203] 0.955 0.950 0.980
EPIC Dataset Generic ICS P, N 4h 5K [89] [204] - - P, C - -
QUT S7 (Myers) Generic ICS P, N 8.5h 15M [205] [206] I, M <0.001 P, L, X [205] 0.744 - 0.727
∗
SWaT Dataset Water Treatment P, N 11d 950K [146] [204] I, O 5.76 P, C, X [207] 0.889 - 0.919
BATADAL Water Distribution P 22m 13K [208] [209] RP, M, O 1.69 C, I [210] 0.970 0.989 0.987
HAI Dataset Power Plant P 10d 1M [138] [139] RP, M 1.83 C [211] 0.780 - 0.950
WADI Dataset∗ Water Distribution P 16d 950K [103] [204] M 1.04 C [207] 0.804 - 0.908

31
 32

D. Physical Level and line maintenance and as “No Events” the normal oper-
ations. Finally, the third dataset includes 41 different labels
BATADAL (BATtle of the Attack Detection ALgo- containing more information about the attacks and various
rithms) [208], [209], [212] was a design challenge aimed at events. In particular, one label is reserved for “No Events” (i.e.,
the creation of an attack detection algorithm. Every participant Normal Operation), eighth labels contain different classes of
was provided with three datasets containing observations of the “intensity” of the “Natural” samples previously mentioned.
a simulated C-Town network [213], a real-world, medium- The remaining 32 labels identify different attacks such as Data
sized water distribution system operated through PLCs and Injection, Command Injection, and Relay Setting Change. All
SCADA systems, which allows modeling the hydraulic re- the details about the labeling process are available in the
sponse of a water distribution network under attack. The readme file at [141]. Physical measures and logs from the
dataset, provided in csv format, contains SCADA reading for control panel, relays, and Snort captures are collected from a
43 system’s variables and is designed for different purposes: physical testbed containing two power generators, four IEDs
two are thought for training (1 year in normal behavior and that can switch four breakers, all connected through switches
6 months with some partially labeled attacks) and one for and routers. Data are provided as a csv file (for the first two
testing (4 months with unlabeled attacks). The 14 cyberattacks datasets) and ARFF format (for the third dataset).
conducted on the system include malicious activation of actu- Different interesting IDSs are implemented on these
ators, change of actuators settings, replay, and MitM attacks. datasets [203], [217]–[219]. In [203] different machine
In the paper [208], the authors present the dataset and the learning-based anomaly detection algorithms are tested against
evaluation criteria for the competition (time-to-detection and the three datasets. The most performant method was JRipper
classification performance). Furthermore, it briefly explains the algorithm [220] together with Adaboost [221] to improve the
strategies employed by each participant. The dataset is free and performance. Results show an F1-score, recall, and precision
available in csv format. There is available also an inp file that almost always greater than 0.8, with a peak of F1-Score of
can be used with EPANET2 to simulate the system. A new 0.955 in the three-class dataset. Also, Accuracy was always
version of the dataset is also available at [214] contains sensor greater than 0.85. Even if the authors did not include any
readings without concealment and is discussed in [215]. numerical results based on common metrics, it is worth
The challenges’ participants developed different algorithms mention another approach presented in [217]. The authors
for intrusion detection ranging from Random Forest to Recur- presented a specification-based intrusion detection framework,
rent Neural Network. Housh and Ohar [210] achieved the best which is tested in the discussed dataset. They implemented
result by proposing a model-based fault detection approach a Bayesian network to model different threat scenarios. The
that employs a simulator to generate benign data and then authors’ purpose was to build a network with a unique path
compare them to the available SCADA readings to detect for each threat scenario. In other words, each scenario must
anomalous behaviors. This approach is composed of three be described as a sequence of system states, actions, and
main phases: 1) available SCADA data are used in a Mixed- events that uniquely identify it. For each threat identified,
Integer Linear Program to estimate the water demand in each the system collected related measurable variables and events.
node; 2) EPANET simulator is used to generate reference Then, each scenario is divided into actions that cause the
values, which are used to produce simulation errors when system state transition. Finally, the Bayesian network is built
compared to actual readings; and 3) a multi-level classification on an independent path of states, computed for each threat.
approach is implemented to classify the obtained simulation An IDS was implemented starting from the Bayesian network
errors into events and normal conditions. The result shows obtained, which reads states and logs to track the system
a Precision of 0.987, and Accuracy of 0.989, and an F1- states. The obtained IDS can classify ten different scenarios
Score of 0.970. In [216] Kravchik and Shabtai present a containing both faults and cyber-attacks by monitoring the
detection approach base on under-complete Autoencoder in state transitions, with different precision based on the relay
the frequency domain, which could reach an F1-Score of location.
0.937, which is high, considering the simplicity and non- Dataset 5: Energy Management System Data [141] is
specificity of the used algorithm. The presented paper was a large anonymized log collected by an Energy Management
applied to the first version of BATADAL. Finally, we must System (EMS) in a utility in the United States of America.
consider that BATADAL is synthetically generated. Therefore The dataset’s csv contains the timestamp and ID of each
this dataset does not suffer from significant noisy problems, event, the SCADA category (i.e., information of the type of
making anomaly detection easier. event), each device type, the event message, the priority code,
Morris et al. presented different ICS datasets, which are the name of the substation, and the area of responsibility (i.e.,
available online [141]. Each dataset’s name is labeled with a the controlling authority). Data are collected in a period of
number from 1 to 5 and the involved industrial sector. 30 days. Since the dataset contains only normal operation, no
Dataset 1: Power System Datasets are a collection of three attacks are provided. For this reason, to the best of the authors’
datasets provided by Morris et al. [141], [217] containing the knowledge, there are not IDS implemented on this dataset.
same data but with various labels. One dataset has binary HAI Dataset (HIL-based Augmented ICS) [138], [139],
labels (i.e., Normal and Attack). The second dataset has three- [222] is a collection of physical data from three physical con-
class labels (i.e., Attack, Natural, and No Events), which trol systems (a GE’s turbine, Emerson’s boiler, and a FESTO’s
identity as “Natural” events single line-to-ground (SLG) faults water treatment systems) combined through the dSPACE HIL
 33

simulator [91]. Data were sampled every second in 59 points domain. With respect to SWaT [146] and BATADAL [208],
representing the variables measured or controlled by the con- the authors also mention that it was impossible to apply the
trol system. Basing on the GitHub repository [139] (which AutoEncoder on the frequency domain because most of the
currently differs from [222]), the data collected contains seven features do not have a clear dominant frequency. However, the
days of normal system behavior, a day with 20 different best results on WADI was obtained by DAICS [207], a deep
attack scenarios on each control loop, and two days with 14 learning solution for anomaly detection in ICSs. The authors
attacks on multiple control loops, for a total of 10 days of propose a 2-branch feature extraction framework. The wide
capturing. Totally, there are around 1 million samples, 1.83% branch, containing only one fully connected layer, is used to
of them are labeled as under MitM attacks, in particular relay memorize the normal state of sensors and actuators. Instead,
and modification attacks. However, all the attacks are deeply the deep branch comprises two fully connected layers between
explained in [222]. Data are provided in a csv format with a two convolution layers and provides the generalization degree
document that accurately depicts the testbed architecture and required to handle events not covered in the training set. More-
the dataset’s data. over, DAICS introduces the few-time-steps algorithm which
Due to the novelty of the dataset, released in 2020, there is can be used to efficiently reconfigure DAICS in a production
a lack of IDS implemented on this dataset. However, in [211] environment when operators encounter false alarms. DAICS
the authors present an anomaly detection strategy based on can achieve a Precision of 0.919 and an F1-Score of 0.804 on
clustered deep one-class classification (CD-OCC). It is an WADI.
unsupervised approach that combines clustering algorithms
with deep learning (DL) models. In particular, K-means was
applied for clustering on the training set. Then, different E. Network Level
types of neural networks (e.g., DNN, CNN, RNN) were CyberCity Dataset [64], [170], [171] is a dataset collected
implemented to predict the clusters and return softmax values by the SANS Institute from their own ICS CyberCity testbed.
classified with the iForest algorithm. Currently, on the HAI CyberCity testbed is a complete simulation of an entire city
Dataset, the higher precision is achieved using DNN as cluster containing a bank, a hospital, a power plant, and many other
predictor (0.95) while the overall higher scores are obtained generally available components in a small town. There is also
with CNN as cluster predictor (F1-score: 0.78; Precision: a tabletop scale model of the city, which shows an electric
0.78). To complete the research, the same algorithms are tested train’s behavior, a water tower, and a miniature traffic light. A
on another popular dataset, SWaT [146], showing the best pcap file is freely downloadable online [171] containing over
results with the same algorithms (i.e., CNN and DNN). 170k network packets recorded as a dataset for the Holiday
WADI [103], [104] is a dataset with data collected from Hack cybersecurity challenge in 2013. The data are unlabeled,
WADI, a water distribution testbed, created as an extension but in [64] the authors estimate that about 16% of the data is
of the SWaT testbed [99]. The system comprises three sub- under attack. Various attacks are included, such as scanning,
systems: a primary grid, a secondary grid, and a return water information disclosure, command injection, MitM, and DoS.
grid. It is also able to simulate water consumption following The ICS components use Modbus/TCP, EtherNet/IP, and Net-
time-varying demand patterns. The dataset collects 16 days of BIOS as communication protocols. For each attack presented,
continuous operation: 14 under regular operation and two days some preventative measures are proposed and evaluated. Some
within an attack scenario (a total of 15 attacks). The adversary examples are awareness training, system patching, IDS, or
aimed to cut off the water supply to the consumer tanks. In the anti-virus, but it is remarked that neither one is 100% effective.
attacker model, the adversary has remote access to the SCADA It is worth noting that, at the best of the authors’ knowledge,
system. The data recorded represent the state of all the 123 there is no precise and official documentation of the dataset
sensors and actuators connected using Modbus/TCP protocol. provided by the SANS Institute.
The dataset is free upon request [204], and it is provided as Dataset 2: Gas Pipeline Datasets [141], [192] contains
csv files. a collection of labeled Modbus/RTU telemetry streams from
There are many IDSs designed and tested on WADI Dataset a gas pipeline system in Mississipi State University’s Critical
in literature. MAD-GAN [223] is an unsupervised multivariate Infrastructure Protection Center [95]. Each stream is composed
anomaly detection method based on Generative Adversarial of some selected features, including, for instance, an identi-
Networks (GANs). This method uses a generative model to fication bit to discriminate between command and responses,
create a fake time serie and a discriminator to distinguish states of components, length of data, and physical measure-
between normal and abnormal data. A peculiarity of this work ments. The authors include different command injection and
is that, instead of considering each data stream independently, data injection attacks, alongside some data in normal behavior.
the framework considers the entire variable set concurrently to The dataset contains about 397k samples, divided into csv files
capture the latent interactions among variables. To do so, the with a name indicating the particular attack. The dataset also
authors implement a sliding-window approach to divide the includes a feature to identify the samples that are effectively
multivariate time series into sub-sequences. On WADI, MAD- part of an attack, with information about the attacker’s action
GAN obtains a precision of 0.53 and an F1-Score of 0.62. in the particular moment. The total percentage of samples with
Better results were achieved by Kravchik and Shabtai [216] abnormal behavior is 0.97%. Unfortunately, the dataset does
which obtain a Precision of 0.83 and an F1-Score of 0.75. They not include each sample’s timestamps, making it impossible
employ an Autoencoder with sequences of length 7 in the time to analyze timing information. The dataset was used to test
 34

different machine learning algorithms as a discriminator of ma- on Dataset 4: New Gas Pipeline using two LSTM layers
licious RTU transactions to detect the deployed attacks [192]. of 256 nodes, each achieving a Precision of 0.94, Accuracy
K-Nearest Neighbors and Random Forest are the two algo- of 0.92, and an F1-Score of 0.85. The most problematic
rithms that provided better results across all the attacks, with attack to be detected was the injection of malicious state
a Recall/Precision of 0.75 or higher for five of the seven commands for which a Gaussian Mixture Model performed
attacks. More in detail, the most problematic attacks were better. Demertizis et al. [193] proposed the Spiking One-Class
burst values (i.e., sending multiple successive pressure values, Anomaly Detection Framework (SOCCADF), which employs
faster than the data display rate, to the operator interface) and the advanced evolving Spiking Neural Netowork (eSNN).
setpoint value injection (i.e., the attacker sends false pressure eSNN is a modular connectionist-based system that evolves its
values equal to the setpoint). Yüksel et al. [224] formally structure and functionality in a continuous, self-organized, on-
describes the user-understandable framework with effective line, adaptive, and interactive way using incoming information.
anomaly detection techniques for ICSs. The test implemented The framework is supervised and was tested on both the
using Modbus/RTU employs the Dataset 2: Gas Pipeline by Dataset 3: Water Storage Tank (Precision 0.981; Accuracy
dividing the attacks into scanning, illegal values, timing, and 0.981; F1-Score 0.981) and the Dataset 4: New Gas Pipeline
illicit command. The results are highly variable depending on (Precision 0.988; Accuracy 0.988; F1-Score 0.988). The same
the trade-off between the detection rate and the false positive authors adopted eSNN on GRYPHON [226], which simplifies
rate. However, by fine-tuning the algorithm, it was possible to the validation mechanisms to work in a semi-supervised way,
achieve a detection rate of 0.9991 and a false positive rate of getting as input only data in standard behavior (i.e., labeled
0.001. as normal packets). This approach was able to get a Precision
Dataset 3: Gas Pipeline and Water Storage Tank by of 0.980, an Accuracy of 0.980, and an F1-Score of 0.980
Morris et al. [141], [183] are two different datasets from on the Dataset 3: Water Storage Tank, while a Precision of
physical testbeds containing both physical data field and 0.975, an Accuracy of 0.977, and an F1-Score of 0.970 on
network traffic. The first comprises data deriving from a gas the Dataset 4: New Gas Pipeline. Another interesting work is
pipeline, while the second contains data from a water storage the metaheuristic approach by Mansouri et al. [227]. In this
tank. Both the datasets come from testbeds at the Mississipi work, the authors provide an anomaly detector based on neural
State University’s Critical Infrastructure Protection Center [95] networks with a pre-processing step able to act with a different
and are shared as ARFF files. A bump-in-the-wire approach algorithm based on the packet’s delay to have as little impact
was used to capture data logs and inject attacks in Modbus as possible on the real-time communications. When computa-
communication in both cases. The implemented attacks are tional speed is required, computationally efficient Evolutionary
reconnaissance, response and command injection, and DoS. System [228] optimization is used. Insted, a more accurate but
They cover around 27% of the total data. The authors also computationally expensive Grey Wolf optimizer [229] is used
provide two short datasets created using 10% of the complete if with higher latency scenarios. A neural network is then used
datasets, suited for rapid tests during the preliminary IDS to detect malicious data with an accuracy up to 98% on the
development phases. As explained in [111], [141], the gas Dataset 4 New Gas Pipeline.
pipeline dataset contains unintended patterns that cause some Electra [55] dataset was obtained from a real scenario of
algorithms to identify attacks and non-attacks in unrealistic an electric traction station used in the railway industry. Electra
ways easily. Therefore, we do not report this work in the is composed of 5 PLCs, a SCADA system, a switch, and
corresponding dataset tables. Instead, we consider the second a firewall. All the communications between the components
version of this dataset, called Dataset 4: New Gas Pipeline. implement Modbus and S7comm over TCP/IP with a master-
Dataset 4: New Gas Pipeline [111], [141] is a new version slave model. There are two different datasets, one for each
of the Dataset 3: Gas Pipeline dataset. This version was communication protocol. The implemented and labeled attacks
proposed to fix dataset problems causing machine learning are false data injection, replay attack, and reconnaissance
algorithms models that do not match real system behav- attacks in both cases. The attacks were deployed with a new
ior and lead to overly optimistic classification accuracy. In device attached to the network with a MitM configuration.
this version, the authors implement 35 attacks and precisely In both Electra Modbus and Electra S7comm datasets, the
document them in the paper and the dataset. The dataset capture lasts about 12 hours in which the 94% and the 98% of
includes different labels for each attack, which cover 21.86% the data are in normal condition, respectively. The data amount
of the capture. Like the previous version, the protocol used is is enormous, containing 387M entries for S7Comm (36.8GB)
Modbus and data are available as an ARFF dataset containing and 16M for Modbus (1.5GB). The two datasets are freely
both physical data and information about the network packets. available on the web [194] in csv format.
D3 and D4 datasets are widely used in the study of IDS Together with the datasets’ presentation, Gómez et al.
for ICS. Feng et al. [225] presented a multi-level anomaly provided an implementation of the main algorithms used
detector using package signatures and LSTM networks. The for anomaly detection. The authors try both supervised and
detection architecture provided is composed of two-level. First, semi-supervised algorithms. On Electra Modbus, a simple
a packet-level anomaly detector based on a Bloom Filter is supervised Random Forest with 200 estimators was sufficient
applied; second, the first-level not-anomaly data are used as to achieve a Precision of 0.988 and an F1-Score of 0.987,
input to a stacked LSTM neural network model time-series while a single layer supervised Neural Network with 128
level anomaly detection. The anomaly detector was tested neurons was able to reach a Precision of 0.9999 and an F1-
 35

Score of 0.996 on the S7Comm version. On the other hand, tured data are divided into various collections with an explicit
the semi-supervised OCSVM performed properly on both the name indicating the types of implemented attacks. The authors
dataset, reaching 0.996 of Precision in Electra S7Comm. In also implemented a cover-channel attacks dataset presented as
the successive year, the same authors proposed SafeMan [230], Lemay Covert. In these attacks, the least significant bit of
a framework to manage both cybersecurity and safety in the the Modbus packets are used to carry information. To the best
manufacturing industry. It is composed of a set of applications of our knowledge, this is the only available dataset contain-
and services used to monitor and analyze the industrial process ing side-channel attacks. Unfortunately, none of the attacks
in real-time. SafeMan is based on Edge Computing (EC) to were designed considering Modbus protocol vulnerabilities.
achieve low latency and fast deployment of applications and Instead, they are implemented with Off-the-Shelf Tools (i.e.,
services. Furthermore, EC allows performing the necessary Metasploit [182]). Data collection lasts about 6.25h, and the
computing tasks close to the manufacturing activity or the samples labeled as attacks are about 0.15% of the total for the
network edge. The framework contains several components first attack, while the covert channel packets are present in the
to assist the deployment, and the risk assessment, together whole capture. The datasets are shared in both pcap and csv
with the cyber threats detection application proposed in [55]. format.
A different and innovative approach was introduced by Li Schneider and Böttinger [234] proposed an unsupervised
et al. [231] who design an anomaly detection method based anomaly detection framework. They employ deep autoen-
on cross-domain knowledge transferring. The authors employ coders with pipelining parallel processing strategies to speed
the TrAdaBoost algorithm to train a neural network using up the training. While the proposed framework performs
not only a part of the data of the Electra Datasets but well on the SWaT dataset [146], it shows very different
also employing data from different domains, both from other results depending on the attack type when applied on the
ICS (e.g., SWaT Dataset [146]) or other CPS fields (e.g., Lemay SCADA dataset. In particular, to correctly detect an
KDDCup99 Dataset [232]). Then, they compared the error attack, the framework requires a minimum duration of it.
rate with respect to a standard SVM and a standard LSTM, For attacks lasting longer than the minimum threshold, the
showing better results, especially when employing a small Precision reaches 100%. Anton et al. [198] implemented
fraction (< 10%) of the Electra Dataset in the training phase. different standard classification machine learning algorithms
HVAC Traces by Ndonda and Sadre [195], [196] is a on Lemay SCADA. The authors extract 14 basic features from
dataset recorded on a Heating, Ventilation, and Air Condi- the packets and nine additional features derived from timing
tioning (HVAC) system powered by Honeywell and used to and frequency information. Algorithms are tested on three
provide thermal comfort and acceptable indoor air quality different batches of packets resulted from merging different
on a university campus. The Building Management System Lemay SCADA datasets. Both Random Forest and SVM
(BMS) is fully automated, and it is suited to monitor from 15 result in an F1-Score and an Accuracy greater than 0.999
to 20 structures, each containing different PLCs and RTUs. with all the batch, while k-means clustering report the lowest
Operators can access the system through the HMI. The proto- results. In a follow-up work of Anton et al. [235] data are
cols implemented are proprietary (e.g., DCE/RPC, NetBIOS, considered as time series. Each second of network traffic was
S7Comm) and use TCP/IP at the transport layer. The data aggregated into a single data point. Three different algorithms
capture was produced using tcpdump, at two routers via port were implemented to detect anomalies inside the three batches
mirroring. To obtain an accurate timestamp on each packet in of captures defined in [198]. The first algorithm implemented
two separate recording points, the authors synchronized the was Matrix Profiles, and it performs well on data with periodic
clocks using Network Time Protocol (NTP) [233]. However, characteristics, requiring only one hyperparameter. Second,
it was not sufficient to ensure good accurate timing. To over- the Seasonal ARIMA-process performed well on periodical
come this problem, the authors introduced a correction factor data and is more resistant to noise but requires a more
calculated using ad-hoc ICMP messages sent periodically on complicated tuning of the three hyperparameters. Finally, the
the network. The anonymized dataset is publicly accessible in authors implemented LSTM, which requires a high training
pcap files, where each file contains one hour of traffic. In total, effort compared with the other two light-weight approaches.
there are about 7 days of collected data in normal conditions, They tested the algorithms on a subset of the Lemay SCADA
without any attacks. Since the dataset does not contain attacks datasets containing seven attacks divided into three categories
and is a novel collection, there are no IDS tested to the best (fake command, executable upload, file moving). The attacks
of our knowledge. are almost all correctly classified with every algorithm. With
Lemay et al. [185] present a dataset of a SCADA net- LSTM, the accuracy is always greater than 0.90, while the
work, also called Lemay SCADA, virtually implemented with F1-Score is really variable based on the threshold selection
SCADA Sandbox. The simulations contain different MTUs methodology.
and controllers connected with the Modbus/TCP protocol. Modbus SCADA #1 [199], [200] by Cruz et al. is a
The attacks are generated with an infected machine that dataset containing data recorded from a small physical testbed
launches various exploits to infect other devices. Then, the simulating a liquid pump. The testbed comprises an HMI,
compromised machines launch different attacks by leveraging an Adruino-based RTU, a PLC, a Variable-Frequency Drive
Metasploit [182] (e.g., Malware Injection, Reconnaissance). (VFD), and a 3-phase motor. The protocols used are Mod-
The authors give particular attention to the labeling process bus/TCP and Modbus/RTU. Data are divided into subfolders
and to maintain normal intra-packet time properties. The cap- based on the attack deployed. Moreover, each pcap file is
 36

named with an intuitive strategy that includes the duration node capture perspective: a file collected from the attacker’s
of both the capture and the attack. The attacks implemented point of view, one from the HMI, and one from the master
range from MitM to different flooding types: ping flooding, PLC. This particular composition could be initially complex
TCP SYN flooding, and Modbus Query flooding. All these to use, but on the other hand, it provides higher flexibility
flooding attacks are aimed at the generation of DoS. The data with respect to datasets with the entire capture. The dataset is
recording lasts for 24 days, containing 4.81% of data flagged available on Github [189].To the best of our knowledge, there
as under attack. are no IDSs implemented on this dataset.
This dataset was used by Radoglou et al. [201] to test an 4SICS [190] is a pcap dataset collected by Netresec from
IDS. Firstly, the authors present an expansion of Smod [236], an ICS lab at the Industrial Cyber Security Conference. At this
a penetration testing tool for Modbus/TCP, to enable the conference, there was an ICS testbed composed of heteroge-
generation of DoS, MitM, and replay attacks. Then, they neous devices such as PLCs, RTUs, servers, and industrial
deployed an IDS to detect DoS attacks and a server for network equipment (e.g., switch, firewalls). It was available
machine learning offloading computation. Among the various for hands-on testing by the conference attendees and, since the
algorithms tested, AdaBoost [221] and Random Forest achieve testbed was left almost uncontrolled, the data recorded are not
the best results with a Precision of 0.96, an Accuracy of 0.81 labeled. Furthermore, it is impossible to know what the users
and an F1-Score of 0.77. have done and, eventually, what kinds of attacks are present.
QUT DNP3 [56], [187] is a dataset presented in the Ph.D. The dataset includes a wide variety of ICS protocol traffic such
dissertation of the author. The dataset contains data collected as S7Comm, Modbus/TCP, EtherNet/IP, and DNP3.
from a small section of a transmission substation SCADA S4x15 ICS Village CTF Dataset [202] provided by Digital
network. The testbed involves GOOSE and DNP3 protocols, Bond, contains network traffic collected during a capture-
enabling the communication between the Master, the Slave, the-flag (CTF) competition in the ICS Village. The system
the IED, and the attacker machine. All the communications was composed of different interconnected PLCs, and the
pass through an industrial switch. The attacks are categorized dataset contains, without labeling, the attacks launched by the
into six categories: Injection, Flooding, Masquerading, Replay, players to the system. The dataset contains pcap files with
MitM, and all attacks. Each category also contains Recon- Modbus/TCP and BACnet packets.
naissance packets. The attacks are launched by an attacker Basing on this dataset, Yu et al. [191] proposed an anomaly
machine, which also generates a log providing information detection method based on TCP and UDP payload inspection.
about each attack sequence’s start and end. Each dataset file The detector’s architecture comprises an offline module for
has a different duration based on the attack frequency during the expected behavior model and an online module containing
the capture creation since the authors implement a random the actual anomaly detector and a packet signature generator.
time between two attacks. Moreover, the dataset is divided into In the proposed work [191], the authors use 4SICS [190]
two categories based on the attack frequency: frequent attacks dataset to model the normal traffic behavior for Modbus/TCP
(i.e., approximately an attack every half an hour) and infre- protocol, while the normal traffic behavior of BACnet protocol
quent attacks (i.e., approximately an attack every random time is based S4x15 dataset. Instead, malicious packets are retrieved
between one and four hours). For each frequency category, from Quickdraw-Snort [237], a collection of Snort rules for
the authors provide two datasets, respectively, for training and ICS environments, which also provides some testing packets.
testing. Furthermore, a control dataset with only legitimate Results show Accuracy and Recall close to 100% and a very
communications (i.e., without any attacks) is available, and low false alarm rate.
it covers 24 hours of recording. In total, the dataset contains WUSTL-IIOT-2018 [101], [147] is a dataset recorded from
40 days of recording. It is worth mention that the labeling a testbed simulating a water tank control system. Network
process was performed with particular care since it was the traffic was monitored for 25 hours, collecting 25 features.
main topic of the thesis work. The dataset is available on Then, the authors performed a data cleaning process to delete
Github [187].However, to the best of our knowledge, there corrupted or missing values and outliers. In this phase, about
are no IDS tested on this dataset. 10k observations were erased, leaving the final version with
QUT S7Comm [188], [189] developed by Rodofile et al. about 7037k entries. Furthermore, only the six more relevant
is an open-source dataset collected in a three time-based sub- features are available in the provided csv file, together with
processes testbed of a mining refinery plant. The plant testbed a column indicating if the observations are related to an
is composed of one Siemens PLC acting as Master and three attack. Various attacks have been launched during the capture:
PLCs actings as slaves, all connected with a switch to an port scanning using Nmap [238], address scan attacks, device
HMI and communicating using S7Comm protocol. The attack identification, and unauthorized access to actuators status by
dataset comprises 9 hours of data and 64 attacks from 13 using known exploits of the Mobus protocol. The final dataset
different possible typologies. Data are provided with pcap files contains 6.07% of data under attack. On the same paper [101],
and four process logs: a master log, a conveyor log, a tank the authors developed IDSs employing standard Machine
log, and a reactor log. The labels of the attack samples are Learning algorithms. Best results were achieved by Decision
contained in separated csv files. The control dataset comprises Tree and KNN with an accuracy of up to 100% considering
8.5 hours of network traffic and process log data, with 32 the offline evaluation. Instead, regarding the online phase,
different processes. This dataset’s peculiarity is the particular Decision Tree and Random Forest obtain the best results with
division of the network traffic in separate files based on the an accuracy of 0.999. Furthermore, these last two models
 37

performed well in terms of False Alarm Rate (i.e., percentage (the last one is dated June 2020). The first version (December
of the normal flows misclassified as abnormal flows) and 2015, described in [146], [204]) is the largest and most used.
Un-Detection Rate (i.e., the fraction of the abnormal flows This version includes both network traffic and recordings from
misclassified as normal flows), which are close to 1. all the 51 sensors and actuators for eleven days. Of these
eleven days, seven days are under normal operation, and four
F. Physical and Network levels days contain 36 different attacks, classified into four types
Electric Power and Intelligent Control (EPIC) is a based on the attack number of stage and devices affected. The
collection of data from 8 scenarios collected with the EPIC first version of SWaT contains about 944k physical samples
testbed [89], [90]. Each collection scenario lasts for 30 minutes (5.76% are labeled as attack samples). The singular network
under normal operations, resulting in more than 5000 readings packets are instead unlabelled, with only a flag indicating the
of sensors and actuators, together with the corresponding presence (or not) of malicious data in each packet’s batch.
Modbus network traffic. Blaq 0 [204] is a dataset obtained In 2017, a new version was released, which collected about
from the same testbed under different attacks. Blaq 0 contains 136 hours of network traffic together with measurements of
network-level data collected from a three-day Hackaton 2018 sensors and actuators provided in csv form. No attacks were
where different teams attack the EPIC testbed. Both datasets performed during the recording phase. In 2019 about 4 hours
are free upon request, but the second one is not widely used. of recording were saved as a dataset. About one hour contains
The EPIC dataset contains both pcap and csv files. Both the 6 different attacks like spoofing and tampering with some
datasets are free upon request on the iTrust website [204]. switch. In this case, physical data are provided as xlsx files. In
QUT S7 (Myers) [205] is a dataset generated from a small the same year, another version was released containing both
scale ICS testbed. It is composed of a bi-directional conveyor network and physical data of about 3 hours, during which
belt system, a water pump system, and a “reactor” pressure two malware attacks were launched. The first try to exfiltrate
vessel system. All these devices are connected to a power historian data, while the second disrupt sensors reading and
meter, and an HMI is used to collect logs. The protocol used is process. The most recent version is dated 2020 and contains
S7 Communication, the standard protocol for Siemens PLCs. 4 runs, each one lasted 2 or 4 hours. Physical data without
The dataset contains device logs with information about each any attacks are provided in xlsx form. In 2017 during the
component’s state and pcap files with network traffic. Data SUTD Security Showdown, a competition where researchers
are divided into two folders containing control data (i.e., data could access and attack the SWaT testbed, all the network
in a normal behavior) and attack data. 21 cyberattacks were flows are saved in a dataset called S317 [239]. The dataset
launched, consisting of two major types: Injection Attacks and records three days of competition and contains historical data
Flooding Attacks. Furthermore, the authors also provide an and the description of the attack scenarios. Both datasets are
xlsx file containing pre-processed data. The dataset is freely free upon request, but the second one is not widely used.
available for the download [206]. In the same paper, Myers Data are provided in different csv, xlsx, and pcap formats. As
et al. [205] proposed a novel process mining based anomaly reported in [240], the various SWAT releases are very different
detection technique. The detector idea is to collect logs in from the operational point of view, also implementing different
order to produce a record of each device’s status. From this actuators control logic. It makes it difficult to transfer the
data, the authors compute a model containing the expected detection framework among the dataset releases. Furthermore,
behavior of the ICS. The model is designed to manage the even in the same SWaT version, the systems behave very
entire process instance from start to finish with only acceptable differently. It is probably due to the testbed recovery time after
events. Finally, process mining is used to perform conformance an attack. To overcome this problem, in more recent versions,
checking activity, calculating the fit of a given event log by the authors restart the testbed after each attack. All the datasets
replaying it on the model. Results show that only 16 attacks out are free upon request at the iTrust datasets webpage [204].
of 21 were correctly identified with several false positives (i.e., The SWaT dataset is probably the most used dataset on
Precision: 0.727; F1-Score: 0.744; Recall: 0.762). The authors which researchers try their IDSs. In almost all the papers
motivate that for most false positives, the starting condition using the SWaT testbeds, there is no explicit mention of the
was altered by previous attacks. It is a common problem that dataset version used. However, from the data description, we
can be mitigated with a correct generation of the dataset, as can infer that the first version is used in almost all the cases.
will be discussed in Section VIII, especially taking care of the Several innovative detection methodologies have been tested
labeling part. on this dataset, ranging from sensor noise fingerprint [241],
SWaT [100], [146] is the most popular dataset in the ICS a graphical-based detection approach [242] and a framework
field. It contains monitoring data from a fully operational to generate invariants with association rules mining [243].
scaled-down water treatment plant. The testbed contains two Kravchik and Shabtai [216] employ the SWaT physical data
separate networks: a level 1 star network that allows the to test different detection approaches such as Dynamic PCA,
communication between the SCADA system and the six PLCs 1D-CNN, and AutoEncoder, both in the time and frequency
and a level 0 ring network that transmits sensor and actuator domains. Thanks to the linearity of many relations between
data to the corresponding PLC. The protocols employed for sensors and actuators in SWaT, PCA performed very well,
communications are CIP and EtherNet/IP. The dataset received especially using a sliding-window approach, which results in
various updates and improvements over the years. More pre- 0.92 of Precision and 0.879 F1-Score. Also AutoEncoder in
cisely, up to today, there are seven different data collections the frequency domain reaches similar scores (i.e., Precision:
 38

0.924; F1-Score: 0.873). Similarly to WADI, excellent results Generates

on SWaT physical data were achieved by Abdelaty et al. [207].
This paper introduces DAICS, 2-branch neural networks with
Testbed Dataset
automatic tuning mechanisms to update the system model.
DAICS scores 0.9185 of Precision and 0.889 of F1-Score. It is
worth noting that, despite a not high Precision (i.e., 0.70) and Represents
F1-Score (i.e., 0.81), MAD-GAN [223] on SWaT achieved a Used
Recall of 0.954, the higher one to the best of our knowledge. Integrates
by
Instead, by relying on SWaT network traffic, Schneider and
Böttinger [234] implement an autoencoder-based unsupervised Implemented Based
anomaly detection framework leveraging pipelining parallel on on
processing strategies to speed up the training. To overcome Intrusion
the problem of unlabelled network packets and generate the Detection
ground truth, the authors developed a semi-automatic label System
estimation mechanism to detect the packets with a higher
probability of being anomalous and then use a manual investi-
Fig. 12: Relations between Testbed, Dataset, and IDS.
gation to label them. Results reveal a Precision of 0.998 and an
F1-Score of about 0.988 in scenarios like TCP session reset
attack and SYN Flood attack, while other types of attacks
as duplicate acknowledgments or TCP retransmissions are and iii) Education, to use the testbed to educate students,
essentially not detected. researchers, and stakeholders. Every scope implies different
requirements to deal with and different funding. For instance,
if a testbed is specifically designed for IDS development,
VIII. L ESSON L EARNED : G OOD P RACTICES
the authors must consider developing an attack chain and
Basing on the knowledge acquired on surveying the different data collection accurately. On the contrary, the Educational
works and analyzing the common mistakes and solutions testbeds do not have this requirement. Instead, they should
implemented, in this section, we summarize concepts and be composed of an easily understandable and representative
good practices to consider when selecting a testing system. process. In this case, water systems are an excellent choice due
In particular, we summarize the good practices in creating an to their immediate visualization. Once the scope is identified,
effective testbed in Section VIII-A and to develop a dataset the designer can give the system’s specific layer adequate
in Section VIII-B. Furthermore, we also provide additional importance to satisfy the scope.
insight to help the standardization of the IDS results in Fidelity. If the testbed is used for Discovery or Education,
Section VIII-C. During the designing phase of each of the data’s perfect fidelity is generally not needed. In these cases,
three resources, the designer must consider the final use of Virtual or Hybrid testbeds are the preferable platforms to be
such resources and the other two resources’ requirements used due to their flexibility and cheapness. Contrarily, in the
in future integration. Figure 12 graphically represents the case of validation tests, Physical testbeds are the best solutions
relation between the three resources. More precisely, a testbed since the smallest variation of measures is fundamental for the
should allow an efficient data collection to produce a well research. A complete work that can help researchers to identify
representative dataset and integrate IDSs to validate the case the correct design criteria is [23]. The US National Institute
studies in a real scenario. A dataset must be designed to be an of Standard and Technology (NIST) has recommended that
exhaustive and precise representation of a testbed and easily a SCADA testbed for security assessment should consider
allow data analysis tasks and the implementation of an IDSs. four general areas [2]: the control center, the communication
Finally, an IDS, which represents the higher-level products architecture, the field devices, and the physical process itself.
with respect to the other two resources, should generalize on
Expensiveness. Expensiveness in the construction and the
different datasets to avoid construction biases. Moreover, the
maintenance of Physical testbeds are the first barriers a re-
design of a dataset should consider an easy integration into a
search group will encounter when deciding to build one. Sup-
real-world scenario such as a testbed.
pose a research group can deal with this limitation. In that case,
it is useful to share with the community datasets collected from
A. Good Practices: Testbed the Physical testbed and the related documentation, as iTrust
An effective testbed development passes through various laboratory of SUTD [212] is doing with SWaT and WADI.
steps and challenges, each composed of a notable complexity Furthermore, provide a simple way for other researchers to
level. These challenges should be considered during the design access the testbed can be an added value not only for the
phase. community, which can take advance of it but also for the owner
Scope Identification. During the design phase, the designer who can have a more critical view of the system.
must consider the final application. The applications of a Reproducibility and Comparability. Robust and innovative
testbed can be [133]: i) Discovery, to study and obtain knowl- researches need to be reproducible and peer validated. Basing
edge about a particular ICS field or system functioning; ii) on this, testing on physical testbeds is not recommended since
Demonstration, to validate or experiment the research findings; it creates difficulties in reproducibility. An intelligent solution
 39

is to create one or more datasets capturing network traffic and Semi-supervised approaches provide a trade-off that efficiently
physical measures from the testbed and share them with the spends an expert’s work supported by a visualization platform
community. In this way, the reviewer can easily verify the such as RiskID [244].
study, while the community will benefit from newly available Documentation. Many of the datasets surveyed lack in
datasets. On the other hand, if a virtual testbed is used, it documentation. To allow correct and easy use of the dataset,
is not required to provide a dataset to support the research. the designer should include detailed information with the
Instead, it is possible to provide the software with the entire dataset’s characteristics or a description of the source testbed
simulation. However, it is fundamental to precisely indicate the design. Exhaustive documentation should include the system’s
architecture and the state of the ICS at the beginning of the control logic, a description of the implemented attacks, and
experiment to avoid reproducibility errors due to a different the configuration settings. In the SWaT case, as reported
scenario. in [240], the recent versions of the dataset implement different
Missing Representation. We identified that the most common control logic. However, the authors never mentioned such
scenario represented with a testbed is relative to water man- modifications.
agement (e.g., Water Distribution, Water Treatment). This is Attacks Similarity. A complete dataset should also include
probably because Water Systems are the most easier scenario attacks. Similarly, in a testbed, a researcher needs to be able
to implement in terms of equipment and maintenance costs. to deploy attacks easily. However, the designer must approach
Another very represented scenario is related to the Electric this phase with caution. Attacks should be as similar as
Plant (e.g., Power Grid, Power Power). However, IDS rarely possible to real cases. If a dataset is collected from a testbed,
investigate this scenario. We believe this is due to the difficulty it is sufficient to launch the attacks following an adversary
in developing detection systems that deal with high-dynamic approach and various system information. The authors should
environments such as Electricity. Also, the majority of the also include a clear and complete analysis of the attacker
related dataset does not include attacks scenario. For future model. On the other hand, it is important and challenging to
organizations that want to approach a testbed development, capture traffic while the attack is occurring in order to generate
we identified a low contribution in scenarios such as large- the datasets. Adding synthetic packets in the resulting capture,
scale manufacturing, Nuclear Plants, Transportation Systems, if not accurately managed, could disrupt the fidelity of the
health-care infrastructure, IIoT, or HVAC. dataset, making it unrealistic. Furthermore, if data are captured
in different monitoring points inside the ICS, it is required a
synchronization mechanism to provide consistent data.
B. Good Practices: Dataset Domain Shift. A common problem in a dataset is the so-
A well-designed dataset should exhaustively represent a called Domain Shift, i.e., the difference between the training
testbed’s behavior and allow easy implementation of research entries and the testing data [207], [240]. To support researchers
findings. To do this, the design process of an effective dataset and IDS development, datasets should be released with com-
must consider the following points. plete documentation explaining the system’s initial state. An-
Labeling. When designing a dataset, the labeling process other problem observed in [240] is related to the testbed
must be precisely described in the documentation to allow remains unstable for a long time. More precisely, after the end
researchers to process the data accordingly. Packets that are of an attack, a system’s behavior may need time to recover,
part of attacks must be carefully labeled to provide ground remaining unstable. In this case, its behavior will be identified
truth to researchers. Furthermore, in a valuable dataset, labels as anomalous by detectors even if flagged as Normal. To
must also contain information about the attack type (e.g., In- deal with this problem, when designing a dataset, the authors
jection, Replay, DoS) and the attack phase. This last element is should consider adding another label to classify the dataset,
essential due to the recovery time of many ICSs: after an attack e.g., “System Unstable”. If the authors can directly interact
occurs, the system may need some time to stabilize itself. This with the testbed, another solution could be to restart the system
behavior can be wrongly considered part of the attack by an after an attack. An imbalanced dataset is a collection of data
inaccurate labeling process. Hence, a good strategy is to flag that contains a significantly low number of samples from one
this kind of packet as recovery, leaving the decision on how to class with respect to the other [245]. It is a critical issue that
consider them to researchers. The work [186] explained that can influence the performances of Machine Learning based
the authors of the SWaT dataset decided to label a process classifiers. Some datasets provided by researchers contain a
data sample as “Attack” when the attack was launched, instead low percentage of data classified as under attack, as reported
of when the system behavior started to change. This approach in Table IV. This happens because attacks generally last for
can lead to a ground truth problem if not correctly documented seconds or minutes, while the ICS is expected to run for much
or managed. Furthermore, it is important to consider the label longer. There are different techniques to get better results from
generation methodology accurately. A manual approach to flag an imbalanced dataset [246]. One solution is to act at the data
each entry of an attack as malicious is costly, and if the data level by re-balance the data in a pre-processing phase using
amount is large may be impracticable. On the other hand, different sampling strategies (e.g., down-sampling). Another
fully automatic strategies are possible, and they work quite novel solution is Data Augmentation, recently introduced
well if attacks are at the same time automatically generated. to improve Anomaly detection performances in [247]. This
However, automatic labeling cannot provide high accuracy technique leverages generative models, such as GAN, to
in case of complex attacks on highly distributed systems. generate synthetic samples. In [245] the authors performed an
 40

experiment to understand the impact of an imbalanced dataset Set significantly differ. However, even if the SWaT dataset
in the ICS security field. Different datasets were obtained currently represents the most used dataset to test IDSs, no
from an extensive network traffic capture collected from a previous works analyzed the statistical distribution through
water control system testbed. To do this, the authors associated statistical tests. Finally, to allow the community to validate the
to a fixed number of attack samples a variable number of research approach and improve it, a good practice is to release
normal samples to create five different datasets with different the code repository source code (e.g., GitHub, Bitbucket).
imbalance ratios (i.e., the percentage of data under attack
over the whole dataset). Ratios span from 0.1% to 10.0%. IX. C ONCLUSION
Results show a high Recall variance with better results on
higher ratios (Recall>0.99 for ratio≥0.70%; Recall<0.12 for In recent years the interconnection between IT and OT
ratio≤0.30). At the same time, the Undetected Rate (UR) (i.e., networks has opened up modern ICSs to new risks and novel
the fraction of the attack samples classified as normal) shows vulnerability surfaces. These vulnerabilities were highlighted
near-zero values for high ratio and large misclassification on in many works but also by the dangerous malware targeting
low ratio datasets (UR<0.01 for ratio≥0.70%; UR>0.88 for industrial companies. Therefore, it is vital to develop new
ratio≤0.30). Basing on these results, the authors conclude that security mechanisms to protect such systems.
it is advisable to generate datasets with at least 1% of data In this paper, we provide a comprehensive overview of the
under attack to reduce imbalance problems when testing IDSs. ICS field by presenting the architecture and the typical devices
employed. We then proposed an analysis of the industrial pro-
tocols used in ICSs, highlighting security measures offered by
C. Good practices: Intrusion Detection System the protocols, their expansions, and analysis of their diffusion
Nowadays, the majority of IDS are based on Machine in the real world. Furthermore, we surveyed and analyzed the
Learning and Deep Learning techniques. To build a model different platforms to test new security mechanisms in the
using such techniques is required a notable amount of well- ICS field. To do this, we categorized the testbeds as Physical,
organized dataset (e.g., balanced, labeled). Since these tech- Virtual, or Hybrid based on their functioning and explaining
niques are not always straightforward to understand and imple- the various challenges and requirements to consider during
ment, researchers should implement a clear and well-approved the development or selection phases. Also, we presented the
pipeline. The following aspects can help the development of different ICS datasets dividing them based on the type of
an effective IDSs. data provided and useful information that can help the reader
Results Baseline. While reading the different papers concern- choose the dataset (e.g., attack implemented, format, and
ing the implementation of IDSs, we noticed the absence of an various data information). To do this, we accessed every
evaluation baseline in many cases. Defining a good baseline dataset and analyzed it separately. We also reported the IDS
could help researchers evaluate if their proposed research is with the best performance present in the literature to offer a
effective and improve the current state of the art. Furthermore, baseline to further works for each dataset. Finally, we depicted
various works base their results on a subset of an available different good practices and suggestions for researchers who
dataset. If not used for a specif reason (e.g., isolate and test want to use this kind of testing method and institutions that
a specific attack), this approach could cause a problem in un- want to build testbeds or collect datasets.
derstanding an IDS’s effectiveness. For this reason, we believe We believe this survey can help address future research on
that our Table IV can support the future evaluation baseline. this field and new researcher approaching the ICS area. In the
We also identified issues in the evaluation metrics. Many future, we aim to continue collecting new testbeds and datasets
research not always use the same metrics, making it difficult on the website to create a collection of useful information the
to compare different approaches. We suggest using as many research community can exploit for researches and studies on
common metrics as possible, such as F1-Score, Accuracy, this essential field.
Precision, and Recall. Baseline problems are also mentioned
in other fields, such as Review Helpfulness predictions [248], ACKNOWLEDGMENT
where the authors proposed to used the same features on the
different models proposed by researchers. In this way, it is This work was supported by a grant from the Cariparo
easier to compare the efficiency of a prediction model. In the Foundation and Yarix S.r.l. which we would like to thank.
ICS field, IDSs model features can be very different and based
on diverse approaches. However, comparing a proposed model R EFERENCES
with the best IDSs of the art state could help future researchers
[1] C. Alcaraz, “Secure interconnection of it-ot networks in industry 4.0,”
identify the right research directions. Furthermore, to avoid the in Critical Infrastructure Security and Resilience. Springer, 2019, pp.
effect of design bias of the dataset, an IDS should be validated 201–217.
on multiple datasets. [2] K. Stouffer, V. Pillitteri, S. Lightman, M. Abrams, and A. Hahn, “Guide
to Industrial Control Systems (ICS) Security NIST Special Publication
Data Verification. Generally, researchers rarely investigate 800-82 Revision 2,” Tech. Rep., 2015. [Online]. Available: https:
the causes of the weak performance of IDSs. Sometimes, the //nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r2.pdf
reasons may be due to a problem related to the distribution of [3] B. Filkins, D. Wylie, and A. J. Dely, “SANS 2019 State of OT / ICS
Cybersecurity Survey,” SANS Institute, no. June, 2019.
the dataset. In [240], the authors showed that SWaT dataset [4] L. Neitzel and B. Huba, “Top ten differences between ICS and IT
behavior and data distribution between Training Set and Test cybersecurity,” InTech Magazine, no. June, 2014.
 41

[5] B. Miller and D. Rowe, “A survey scada of and critical infrastructure [28] D. Sullivan, E. Luiijf, and E. J. M. Colbert, “Components of
incidents,” in Proceedings of the 1st Annual conference on Research Industrial Control Systems,” 2016, vol. 66, pp. 15–28. [Online].
in information technology, 2012, pp. 51–56. Available: http://link.springer.com/10.1007/978-3-319-32125-7http://
[6] N. Falliere, L. O. Murchu, and E. Chien, “W32. Stuxnet Dossier, link.springer.com/10.1007/978-3-319-32125-7 2
Symantec Security Response, Version 1.4, February 2011,” Symantec [29] IANA. Service Name and Transport Protocol Port
Security Response, vol. 4, no. February, pp. 1–69, 2011. Number Registry. Accessed: 02-02-2021. [Online]. Avail-
[7] Israeli Test on Worm Called Crucial in Iran Nuclear Delay. Accessed: able: https://www.iana.org/assignments/service-names-port-numbers/
25-01-2021. [Online]. Available: https://www.nytimes.com/2011/01/ service-names-port-numbers.xhtml
16/world/middleeast/16stuxnet.html [30] Modbus - IDA, “Modbus Application Protocol Specification,” Tech.
[8] S. Shrivastava, “BlackEnergy - Malware for Cyber-Physical Attacks,” Rep., 2006. [Online]. Available: http://www.modbus-ida.org
no. May, 2016. [31] Modbus Organization, “MODBUS/TCP Security, Protocol
[9] A. D. Pinto, Y. Dragoni, and A. Carcano, “TRITON: The First Specification,” Tech. Rep., 2018. [Online]. Available: https:
ICS Cyber Attack on Safety Instrument Systems Understanding the //modbus.org/docs/MB-TCP-Security-v21 2018-07-24.pdf
Malware, Its Communications and Its OT Payload,” Black Hat USA, [32] A. Shahzad, M. Lee, Y.-K. Lee, S. Kim, N. Xiong, J.-Y. Choi, and
2018. Y. Cho, “Real time modbus transmissions and cryptography security
[10] Kaspersky Lab, “Threat landscape for industrial automation systems in designs and enhancements of protocol sensitive information,” Symme-
the second half of 2016,” AO Kaspersky Lab, 1997 - 2017, pp. 1–12, try, vol. 7, no. 3, pp. 1176–1210, 2015.
2016. [33] G. Bernieri, S. Cecconello, M. Conti, and G. Lain, “Tambus: A novel
[11] G. Barbieri, M. Conti, N. O. Tippenhauer, and F. Turrin, “Sorry, shodan authentication method through covert channels for securing industrial
is not enough! assessing ics security via ixp network traffic analysis,” networks,” Computer Networks, vol. 183, p. 107583, 2020.
arXiv preprint arXiv:2007.01114, 2020. [34] I. Triangle MicroWorks, “DNP3 overview,” Tech. Rep., 2002.
[12] M. Nawrocki, T. C. Schmidt, and M. Wählisch, “Uncovering vulnerable [35] GE Harris, “Overview of DNP3 Security Ver-
industrial control systems from the internet core,” in NOMS 2020-2020 sion 6,” Tech. Rep., 2020. [Online]. Avail-
IEEE/IFIP Network Operations and Management Symposium. IEEE, able: https://www.dnp.org/LinkClick.aspx?fileticket=hyvYMYugaQI%
2020, pp. 1–9. 3D&tabid=66&portalid=0&mid=447&forcedownload=true
[13] N. Networks. The Cost of OT Cybersecurity Incidents [36] S. Bagaria, S. B. Prabhakar, and Z. Saquib, “Flexi-dnp3: Flexible
and How to Reduce Risk. Accessed: 02-02-2021. [On- distributed network protocol version 3 (dnp3) for scada security,” in
line]. Available: https://www.nozominetworks.com/solutions/challenge/ Recent Trends in Information Systems (ReTIS), 2011 International
cost-of-ot-cyber-security-incidents/ Conference on. IEEE, 2011, pp. 293–296.
[14] IBM. IBM Study: Businesses More likely to Pay Ransomware [37] M. Majdalawieh, F. Parisi-Presicce, and D. Wijesekera, “Dnpsec:
than Consumers. Accessed: 02-02-2021. [Online]. Available: https: Distributed network protocol version 3 (dnp3) security framework,”
//www-03.ibm.com/press/us/en/pressrelease/51230.wss in Advances in Computer, Information, and Systems Sciences, and
Engineering. Springer, 2007, pp. 227–234.
[15] Coverware. Ransomware Costs Double in Q4 as
[38] A. Kleinmann and A. Wool, “Accurate Modeling of the Siemens
Ryuk, Sodinokibi Proliferate. Accessed: 02-02-2021.
S7 SCADA Protocol for Intrusion Detection and Digital Forensics,”
[Online]. Available: https://www.coveware.com/blog/2020/1/22/
Journal of Digital Forensics, Security and Law, vol. 9, no. 2, 2014.
ransomware-costs-double-in-q4-as-ryuk-sodinokibi-proliferate
[39] C. Lei, L. Donghong, and M. Liang, “The spear to break the security
[16] Dragos. EKANS Ransomware and ICS Operations. Accessed:
wall of s7commplus,” Blackhat USA, 2017.
02-02-2021. [Online]. Available: https://www.dragos.com/blog/
[40] PROFIBUS, “PROFINET System Description - Technology and Ap-
industry-news/ekans-ransomware-and-ics-operations/
plication,” Tech. Rep., 2014.
[17] H. Holm, M. Karresand, A. Vidström, and E. Westring, “A Survey of
[41] Profibus International, “Security Extensions for PROFINET - PI White
Industrial Control System Testbeds,” NordSec 2015: Secure IT Systems,
Paper for PROFINET,” Tech. Rep., 2019.
vol. 9417, pp. 213–230, 2015.
[42] V. Schiffer, “Common Industrial Protocol (CIP™) and the Family of
[18] H. Christiansson and E. Luiijf, “Creating a European SCADA security CIP Networks,” ODVA, Inc., no. February, pp. 1–134, 2016.
testbed,” IFIP Advances in Information and Communication Technol- [43] ODVA. CIP Secuirty. Accessed: 8-10-2020. [Online]. Available:
ogy, vol. 253, pp. 237–247, 2008. www.odva.org/technology-standards/distinct-cip-services/cip-security
[19] S. McLaughlin, C. Konstantinou, X. Wang, L. Davi, A. R. Sadeghi, [44] I. González, A. J. Calderón, J. Figueiredo, and J. M. Sousa, “A
M. Maniatakos, and R. Karri, “The Cybersecurity Landscape in Indus- literature survey on open platform communications (OPC) applied to
trial Control Systems,” Proceedings of the IEEE, vol. 104, no. 5, pp. advanced industrial environments,” Electronics (Switzerland), vol. 8,
1039–1057, 2016. no. 5, pp. 1–29, 2019.
[20] M. H. Cintuglu, O. A. Mohammed, K. Akkaya, and A. S. Uluagac, [45] B. Rolston, “Security implications of opc, ole, dcom, and rpc in control
“A Survey on Smart Grid Cyber-Physical System Testbeds,” IEEE systems,” Idaho National Laboratory, 2006.
Communications Surveys and Tutorials, vol. 19, no. 1, pp. 446–464, [46] H. Renjie, L. Feng, and P. Dongbo, “Research on OPC UA security,”
2017. Proceedings of the 2010 5th IEEE Conference on Industrial Electronics
[21] Y. Geng, Y. Wang, W. Liu, Q. Wei, K. Liu, and H. Wu, “A survey of and Applications, ICIEA 2010, pp. 1439–1444, 2010.
industrial control system testbeds,” IOP Conference Series: Materials [47] G. Clarke, D. Reynders, and E. Wright, Practical Modern SCADA
Science and Engineering, vol. 569, no. 4, 2019. Protocols: DNP3, 60870.5 and Related Systems, 2004.
[22] S. Choi, J. H. Yun, and S. K. Kim, A comparison of ICS datasets [48] P. Maynard, K. McLaughlin, and B. Haberler, “Towards understanding
for security research based on attack paths. Springer International man-in-the-middle attacks on iec 60870-5-104 scada networks,” in ICS-
Publishing, 2019, vol. 11260 LNCS. [Online]. Available: http: CSR, 2014.
//dx.doi.org/10.1007/978-3-030-05849-4 12 [49] D. Baigent, M. Adamiak, and R. Mackiewicz, “IEC 61850 Communi-
[23] U. P. D. Ani, J. M. Watson, B. Green, B. Craggs, and J. R. Nurse, “De- cation Networks and Systems In Substations : An Overview for Users,”
sign considerations for building credible security testbeds: Perspectives pp. 1411–1423, 2013.
from industrial control system use cases,” Journal of Cyber Security [50] S. M. Hussain, T. S. Ustun, and A. Kalam, “A Review of IEC
Technology, pp. 1–49, 2020. 62351 Security Mechanisms for IEC 61850 Message Exchanges,” IEEE
[24] B. Green, R. Derbyshire, W. Knowles, J. Boorman, P. Ciholas, Transactions on Industrial Informatics, vol. 16, no. 9, pp. 5643–5654,
D. Prince, and D. Hutchison, “{ICS} testbed tetris: Practical building 2020.
blocks towards a cyber security resource,” in 13th {USENIX} Workshop [51] H. Yoo and T. Shon, “Challenges and research directions for hetero-
on Cyber Security Experimentation and Test ({CSET} 20), 2020. geneous cyber-physical system based on IEC 61850: Vulnerabilities,
[25] F. Khorrami, P. Krishnamurthy, and R. Karri, “Cybersecurity for security requirements, and security architecture,” Future Generation
Control Systems: A Process-Aware Perspective,” IEEE Design and Test, Computer Systems, vol. 61, pp. 128–136, 2016.
vol. 33, no. 5, pp. 75–83, 2016. [52] R. American Society of Heating and A.-C. Engineers. BACnet Website.
[26] L. Obregon, “InfoSec Reading Room Secure Architecture for Indus- Accessed: 11-10-2020. [Online]. Available: http://www.bacnet.org/
trial Control Systems,” SANS Institute InfoSec, GIAC (GSEC) Gold [53] W. Xu, Y. Tao, and X. Guan, “The landscape of industrial control sys-
Certification, vol. 1, pp. 1–27, 2014. tems (ics) devices on the internet,” in 2018 International Conference on
[27] Promotic Software. Accessed: 10-02-2021. [Online]. Available: Cyber Situational Awareness, Data Analytics and Assessment (Cyber
https://www.promotic.eu/en/index.htm SA). IEEE, 2018, pp. 1–8.
 42

[54] T. Carlsson. Industrial network market shares 2020 according to HMS [75] G. Koutsandria, R. Gentz, M. Jamei, A. Scaglione, S. Peisert, and
Networks. Accessed: 29-01-2021. [Online]. Available: https://www. C. McParland, “A real-time testbed environment for cyber-physical
hms-networks.com/news-and-insights/news-from-hms/2020/05/29/ security on the power grid,” in Proceedings of the First ACM Workshop
industrial-network-market-shares-2020-according-to-hms-networks on Cyber-Physical Systems-Security and/or PrivaCy, ser. CPS-SPC ’15.
[55] Á. L. P. Gómez, L. F. Maimó, A. H. Celdrán, F. l. J. a. Clemente, New York, NY, USA: Association for Computing Machinery, 2015, p.
C. C. Sarmiento, C. J. D. C. Masa, and R. n.M.ndez Nistal, “On 67–78.
the Generation of Anomaly Detection Datasets in Industrial Control [76] P. Čeleda, J. Vykopal, V. Švábenskỳ, and K. Slavı́ček, “Kypo4industry:
Systems,” IEEE Access, vol. 7, pp. 177 460–177 473, 2019. A testbed for teaching cybersecurity of industrial control systems,”
[56] N. R. Rodofile, “Generating Attacks and Labelling Attack Datasets in Proceedings of the 51st ACM Technical Symposium on Computer
for Industrial Control Intrusion Detection Systems,” Ph.D. dissertation, Science Education, 2020, pp. 1026–1032.
2013. [77] J. Rubio-Hernan, J. Rodolfo-Mejias, and J. Garcia-Alfaro, “Security
[57] R. Mitchell and I. R. Chen, “A survey of intrusion detection techniques of cyber-physical systems,” in International Workshop on the Security
for cyber-physical systems,” ACM Computing Surveys, vol. 46, no. 4, of Industrial Control Systems and Cyber-Physical Systems. Springer,
2014. 2016, pp. 3–18.
[58] Y. Hu, A. Yang, H. Li, Y. Sun, and L. Sun, “A survey of intrusion [78] LegoSCADA Testbed. Accessed: 11-01-2021. [Online]. Available:
detection on industrial control systems,” International Journal of Dis- http://j.mp/legoscada
tributed Sensor Networks, vol. 14, no. 8, 2018. [79] F. Guo, L. Herrera, M. Alsolami, H. Li, P. Xu, X. Lu, A. Lang,
[59] G. Bernieri, M. Conti, and F. Turrin, “Kingfisher: An industrial J. Wang, and Z. Long, “Design and development of a reconfigurable
security framework based on variational autoencoders,” in Proceedings hybrid Microgrid testbed,” 2013 IEEE Energy Conversion Congress
of the 1st Workshop on Machine Learning on Edge in Sensor Systems, and Exposition, ECCE 2013, pp. 1350–1356, 2013.
ser. SenSys-ML 2019. New York, NY, USA: Association for [80] W. Xu, Y. Tao, C. Yang, and H. Chen, “MSICST: Multiple-scenario
Computing Machinery, 2019, p. 7–12. [Online]. Available: https: industrial control system testbed for security research,” Computers,
//doi.org/10.1145/3362743.3362961 Materials and Continua, vol. 60, no. 2, pp. 691–705, 2019.
[60] Y. Xie, W. Wang, F. Wang, and R. Chang, “VTET: A Virtual Industrial [81] R. Candell, T. Zimmerman, and K. Stouffer, “An Industrial Control
Control System Testbed for Cyber Security Research,” 2018 3rd Inter- System Cybersecurity Performance Testbed,” National Institute of Stan-
national Conference on Security of Smart Cities, Industrial Control dards and Technology, U.S. Department of Commerce, no. November,
System and Communications, SSIC 2018 - Proceedings, 2018. 2015.
[61] H. G. Aghamolki, Z. Miao, and L. Fan, “A hardware-in-the-loop scada [82] T. Edgar, D. Manz, and T. Carroll, “Towards an experimental testbed
testbed,” in 2015 North American Power Symposium (NAPS), 2015, facility for cyber-physical security research,” ACM International Con-
pp. 1–6. ference Proceeding Series, pp. 4–7, 2011.
[62] T. Alves, R. Das, and T. Morris, “Virtualization of industrial control [83] C. Queiroz, A. Mahmood, J. Hu, Z. Tari, and X. Yu, “Building a
system testbeds for cybersecurity,” in Proceedings of the 2nd Annual SCADA security testbed,” NSS 2009 - Network and System Security,
Industrial Control System Security Workshop, 2016, pp. 10–14. pp. 357–364, 2009.
[63] T. Cruz, L. Rosa, J. Proença, L. Maglaras, M. Aubigny, L. Lev, J. Jiang, [84] V. Urias, B. Van Leeuwen, and B. Richardson, “Supervisory Command
and P. Simões, “A cybersecurity detection framework for supervisory and Data Acquisition (SCADA) system cyber security analysis using
control and data acquisition systems,” IEEE Transactions on Industrial a live, virtual, and constructive (LVC) testbed,” Proceedings - IEEE
Informatics, vol. 12, no. 6, pp. 2236–2246, 2016. Military Communications Conference MILCOM, no. Lvc, pp. 1–8,
[64] R. C. Borges Hink and K. Goseva-Popstojanova, “Characterization of 2012.
Cyberattacks Aimed at Integrated Industrial Control and Enterprise [85] D. C. Bergman, D. Jin, D. M. Nicol, and T. Yardley, “The virtual power
Systems: A Case Study,” Proceedings of IEEE International Sympo- system testbed and inter-testbed integration,” 2nd Workshop on Cyber
sium on High Assurance Systems Engineering, vol. 2016-March, pp. Security Experimentation and Test, CSET 2009, no. August, 2009.
149–156, 2016. [86] I. Ahmed, V. Roussev, W. Johnson, S. Senthivel, and S. Sudhakaran,
[65] C. Siaterlis, B. Genge, and M. Hohenadel, “EPIC: A testbed for “A SCADA system testbed for cybersecurity and forensic research and
scientifically rigorous cyber-physical security experimentation,” IEEE pedagogy,” ACM International Conference Proceeding Series, pp. 1–9,
Transactions on Emerging Topics in Computing, vol. 1, no. 2, pp. 319– 2016.
330, 2013. [87] P. Blazek, R. Fujdiak, P. Mlynek, and J. Misurec, “Development
[66] Epic testbed website. Accessed: 08-05-2020. [Online]. Available: of cyber-physical security testbed based on iec 61850 architecture,”
http://sourceforge.net/projects/amici/ Elektronika ir Elektrotechnika, vol. 25, no. 5, pp. 82–87, 2019.
[67] H. Gao, Y. Peng, K. Jia, Z. Dai, and T. Wang, “The design of [88] E. Korkmaz, A. Dolgikh, M. Davis, and V. Skormin, “Industrial control
ICS testbed based on emulation, physical, and simulation (EPS-ICS systems security testbed,” in 11th Annual Symposium on Information
Testbed),” Proceedings - 2013 9th International Conference on Intelli- Assurance, 2016.
gent Information Hiding and Multimedia Signal Processing, IIH-MSP [89] S. Adepu, N. K. Kandasamy, and A. Mathur, “EPIC: An electric power
2013, pp. 420–423, 2013. testbed for research and training in cyber physical systems security,”
[68] R. E. Gillen, L. A. Anderson, C. Craig, J. Johnson, A. Columbia, Lecture Notes in Computer Science (including subseries Lecture Notes
R. Anderson, A. Craig, and S. L. Scott, “Design and Implementation in Artificial Intelligence and Lecture Notes in Bioinformatics), vol.
of Full-Scale Industrial Control System Test Bed for Assessing Cyber- 11387 LNCS, no. November, pp. 37–52, 2018.
Security Defenses,” Proceedings - 21st IEEE International Symposium [90] Singapore University of Technology and Design (SUTD).
on a World of Wireless, Mobile and Multimedia Networks, WoWMoM Electric Power and Intelligent Control (EPIC) testbed Webpage.
2020, pp. 341–346, 2020. Accessed: 13-01-2021. [Online]. Available: https://itrust.sutd.edu.sg/
[69] H. Henry, P. Maynard, and K. McLaughlin, “ICS Interaction Testbed: itrust-labs-home/itrust-labs epic/
A Platform for Cyber-Physical Security Research,” pp. 1–8, 2019. [91] H. K. Shin, W. Lee, J. H. Yun, and H. C. Kim, “Implementation of
[70] G. Bernieri, F. Del Moro, L. Faramondi, and F. Pascucci, “A testbed programmable CPS testbed for anomaly detection,” in 12th USENIX
for integrated fault diagnosis and cyber security investigation,” Interna- Workshop on Cyber Security Experimentation and Test, CSET 2019,
tional Conference on Control, Decision and Information Technologies, co-located with USENIX Security 2019, 2019. [Online]. Available:
CoDIT 2016, pp. 454–459, 2016. https://www.usenix.org/conference/cset19/presentation/shin
[71] Hydra testbed repository. Accessed: 11-12-2020. [Online]. Available: [92] B. Green, A. Le, R. Antrobus, U. Roedig, D. Hutchison, and A. Rashid,
https://github.com/hydra-testbed/Part-list “Pains, gains and PLCs: Ten lessons from building an industrial control
[72] J. Jarmakiewicz, K. Maślanka, and K. Parobczak, “Development of systems testbed for security research,” 10th USENIX Workshop on
cyber security testbed for critical infrastructure,” in 2015 International Cyber Security Experimentation and Test, CSET 2017, co-located with
Conference on Military Communications and Information Systems USENIX Security 2017, 2017.
(ICMCIS), 2015, pp. 1–10. [93] F. Sauer, M. Niedermaier, S. Kießling, and D. Merli, “Licster–a low-
[73] M. Kaouk, F.-X. Morgand, and J.-M. Flaus, “A testbed for cybersecurity cost ics security testbed for education and research,” in 6th Interna-
assessment of industrial and iot-based control systems,” 2018. tional Symposium for ICS & SCADA Cyber Security Research 2019 6,
[74] J. Kim, K. Kim, and M. Jang, “Cyber-Physical Battlefield Platform 2019, pp. 1–10.
for Large-Scale Cybersecurity Exercises,” International Conference on [94] hsainnos. Low-cost ICS Testbed Github Repository. Accessed: 10-02-
Cyber Conflict, CYCON, vol. 2019-May, pp. 1–19, 2019. 2021. [Online]. Available: https://github.com/hsainnos/LICSTER
 43

[95] T. Morris, R. Vaughn, and Y. S. Dandass, “A testbed for SCADA con- [117] V. S. Koganti, M. Ashrafuzzaman, A. A. Jillepalli, and F. T. Shel-
trol system cybersecurity research and pedagogy,” ACM International don, “A virtual testbed for security management of industrial control
Conference Proceeding Series, 2011. systems,” in 2017 12th International Conference on Malicious and
[96] A. Hahn, B. Kregel, M. Govindarasu, J. Fitzpatrick, R. Adnan, Unwanted Software (MALWARE), 2017, pp. 85–90.
S. Sridhar, and M. Higdon, “Development of the PowerCyber SCADA [118] S. Lee, S. Lee, H. Yoo, S. Kwon, and T. Shon, “Design and implemen-
security testbed,” ACM International Conference Proceeding Series, tation of cybersecurity testbed for industrial iot systems,” The Journal
pp. 1–4, 2010. of Supercomputing, vol. 74, no. 9, pp. 4506–4520, 2018.
[97] N. Sayegh, A. Chehab, I. H. Elhajj, and A. Kayssi, “Internal security [119] P. Maynard, K. McLaughlin, and S. Sezer, “An Open Framework for
attacks on scada systems,” in 2013 Third International Conference on Deploying Experimental SCADA Testbed Networks,” 5th International
Communications and Information Technology (ICCIT). IEEE, 2013, Symposium for ICS & SCADA Cyber Security Research 2018: Proceed-
pp. 22–27. ings, 2018.
[98] Cx-270322: Idaho national laboratory (inl) smart grid test bed revision [120] ——. Ics testbed framework. Accessed: 08-05-2020. [Online].
2. Accessed: 30-04-2020. [Online]. Available: https://www.energy.gov/ Available: https://github.com/PMaynard/ICS-TestBed-Framework
sites/prod/files/2017/10/f38/CX-270322.pdf [121] D. Antonioli and N. O. Tippenhauer, “Minicps: A toolkit for security
[99] A. P. Mathur and N. O. Tippenhauer, “SWaT: A water treatment testbed research on cps networks,” in Proceedings of the First ACM workshop
for research and training on ICS security,” 2016 International Workshop on cyber-physical systems-security and/or privacy, 2015, pp. 91–100.
on Cyber-physical Systems for Smart Water Networks, CySWater 2016, [122] Minicps: a framework for cyber-physical systems real-time simulation,
no. Figure 1, pp. 31–36, 2016. built on top of mininet. Accessed: 08-05-2020. [Online]. Available:
[100] Singapore University of Technology and Design (SUTD). Secure Water https://github.com/scy-phy/minicps
Treatment (SWaT) testbed Webpage. Accessed: 13-01-2021. [Online]. [123] B. Reaves and T. Morris, “An open virtual testbed for industrial
Available: https://itrust.sutd.edu.sg/itrust-labs-home/itrust-labs swat/ control system security research,” International Journal of Information
[101] M. A. Teixeira, T. Salman, M. Zolanvari, R. Jain, N. Meskin, and Security, vol. 11, no. 4, pp. 215–229, 2012.
M. Samaka, “SCADA system testbed for cybersecurity research using [124] M. Almgren, P. Andersson, G. Björkman, M. Ekstedt, J. Hallberg,
machine learning approach,” Future Internet, vol. 10, no. 8, 2018. S. Nadjm-Tehrani, and E. Westring, “Rics-el: building a national
[102] I. N. Fovino, M. Masera, L. Guidi, and G. Carpi, “An experimental testbed for research and training on scada security (short paper),”
platform for assessing SCADA vulnerabilities and countermeasures in International Conference on Critical Information Infrastructures
in power plants,” 3rd International Conference on Human System Security. Springer, 2018, pp. 219–225.
Interaction, HSI’2010 - Conference Proceedings, pp. 679–686, 2010. [125] A. Ghaleb, S. Zhioua, and A. Almulhem, “Scada-sst: a scada secu-
[103] C. M. Ahmed, V. R. Palleti, and A. P. Mathur, “WADI: A water rity testbed,” in 2016 World Congress on Industrial Control Systems
distribution testbed for research in the design of secure cyber physical Security (WCICSS). IEEE, 2016, pp. 1–6.
systems,” Proceedings - 2017 3rd International Workshop on Cyber- [126] SCADA-SST - scada security testbed. Accessed: 12-01-2021. [Online].
Physical Systems for Smart Water Networks, CySWATER 2017, pp. Available: https://sourceforge.net/projects/scada-sst/
25–28, 2017.
[127] C. Queiroz, A. Mahmood, and Z. Tari, “SCADASim - A framework
[104] Singapore University of Technology and Design (SUTD). Water Dis-
for building SCADA simulations,” IEEE Transactions on Smart Grid,
tribution (WADI) testbed Webpage. Accessed: 13-01-2021. [Online].
vol. 2, no. 4, pp. 589–597, 2011.
Available: https://itrust.sutd.edu.sg/itrust-labs-home/itrust-labs wadi/
[128] T. Z. Queiroz Carlos, Mahmood Abdun. scadasim on github.
[105] Y. Yang, K. McLaughlin, S. Sezer, T. Littler, E. G. Im, B. Pranggono,
Accessed: 26-07-2020. [Online]. Available: https://github.com/
and H. F. Wang, “Multiattribute scada-specific intrusion detection
caxqueiroz/scadasim
system for power networks,” IEEE Transactions on Power Delivery,
vol. 29, no. 3, pp. 1092–1102, 2014. [129] A. Almalawi, Z. Tari, I. Khalil, and A. Fahad, “SCADAVT-A frame-
[106] F. Zhang, H. A. D. E. Kodituwakku, J. W. Hines, and J. Coble, work for SCADA security testbed based on virtualization technology,”
“Multilayer Data-Driven Cyber-Attack Detection System for Industrial Proceedings - Conference on Local Computer Networks, LCN, pp. 639–
Control Systems Based on Network, System, and Process Data,” IEEE 646, 2013.
Transactions on Industrial Informatics, vol. 15, no. 7, pp. 4362–4369, [130] P. Singh, S. Garg, V. Kumar, and Z. Saquib, “A testbed for scada cyber
2019. security and intrusion detection,” in 2015 International Conference
[107] C. M. Davis, J. E. Tate, H. Okhravi, C. Grier, T. J. Overbye, and on Cyber Security of Smart Cities, Industrial Control System and
D. Nicol, “SCADA cyber security testbed development,” in 2006 38th Communications (SSIC). IEEE, 2015, pp. 1–6.
Annual North American Power Symposium, NAPS-2006 Proceedings, [131] M. Mallouhi, Y. Al-Nashif, D. Cox, T. Chadaga, and S. Hariri, “A
2006, pp. 483–488. testbed for analyzing security of SCADA control systems (TASSCS),”
[108] M. Krotofil and J. Larsen, “Rocking the pocket book: Hacking chemical IEEE PES Innovative Smart Grid Technologies Conference Europe,
plants,” in DefCon Conference, DEFCON, 2015. ISGT Europe, pp. 1–7, 2011.
[109] DVCP-TE. Accessed: 12-01-2021. [Online]. Available: https://github. [132] C. Wang, L. Fang, and Y. Dai, “A simulation environment for SCADA
com/satejnik/DVCP-TE security analysis and assessment,” 2010 International Conference on
[110] A. A. Farooqui, S. S. H. Zaidi, A. Y. Memon, and S. Qazi, “Cyber Measuring Technology and Mechatronics Automation, ICMTMA 2010,
security backdrop: A scada testbed,” in 2014 IEEE Computers, Com- vol. 1, pp. 342–347, 2010.
munications and IT Applications Conference. IEEE, 2014, pp. 98–103. [133] N. O. Tippenhauer, “Design and realization of testbeds for security
[111] T. H. Morris, Z. Thornton, and I. Turnipseed, “Industrial research in the industrial internet of things,” in Security and Privacy
Control System Simulation and Data Logging for Intrusion Trends in the Industrial Internet of Things. Springer, 2019, pp. 287–
Detection System Research,” Seventh Annual Southeastern Cyber 310.
Security Summit, 2015. [Online]. Available: http://www.ece.uah.edu/ [134] M. Almgren, W. Aoudi, R. Gustafsson, R. Krahl, and A. Lindhé, “The
$\sim$thm0009/icsdatasets/cyberhuntsvillepaper v4.pdf nuts and bolts of deploying process-level IDS in industrial control
[112] B. Genge, C. Siaterlis, I. Nai Fovino, and M. Masera, “A systems,” ACM International Conference Proceeding Series, pp. 17–
cyber-physical experimentation environment for the security analysis 24, 2018.
of networked industrial control systems,” Computers and Electrical [135] J. Gardiner, B. Craggs, B. Green, and A. Rashid, “Oops I did it again:
Engineering, vol. 38, no. 5, pp. 1146–1161, 2012. [Online]. Available: Further adventures in the land of ICS security testbeds,” Proceedings
http://dx.doi.org/10.1016/j.compeleceng.2012.06.015 of the ACM Conference on Computer and Communications Security,
[113] A. Giani, G. Karsai, T. Roosta, A. Shah, B. Sinopoli, and J. Wiley, “A pp. 75–86, 2019.
testbed for secure and robust SCADA systems,” ACM SIGBED Review, [136] E. Eide, L. Stoller, and J. Lepreau, “An experimentation workbench
vol. 5, no. 2, pp. 1–4, 2008. for replayable networking research,” in NSDI, 2007.
[114] D. Formby, M. Rad, and R. Beyah, “Lowering the barriers to industrial [137] E. Eide, “Toward replayable research in networking and
control system security with {GRFICS},” in 2018 {USENIX} Work- systems,” Position paper presented at Archive, no. May, 2010.
shop on Advances in Security Education ({ASE} 18), 2018. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?
[115] ——. Grfics. Accessed: 11-01-2021. [Online]. Available: https: doi=10.1.1.170.9948&rep=rep1&type=pdf
//github.com/djformby/GRFICS [138] S. Choi, J.-h. Yun, B.-g. Min, and H. Kim, “POSTER: Expanding
[116] D. Jin, D. M. Nicol, and G. Yan, “An event buffer flooding attack in a Programmable CPS Testbed for Network Attack Analysis,”
dnp3 controlled scada systems,” in Proceedings of the 2011 Winter in Proceedings of the 15th ACM Asia Conference on Computer and
Simulation Conference (WSC). IEEE, 2011, pp. 2614–2626. Communications Security. New York, NY, USA: ACM, oct 2020, pp.
 44

928–930. [Online]. Available: https://dl.acm.org/doi/10.1145/3320269. [163] OMNeT++ Discrete Event Simulator. An extensible, modular,
3405447 component-based C++ simulation library and framework, primarily
[139] H.-K. Shin, W. Lee, J.-H. Yun, and H. Kim. IL-based Augmented ICS for building network simulators. Accessed: 15-12-2020. [Online].
(HAI) Security Dataset. Accessed: 22-10-2020. [Online]. Available: Available: https://omnetpp.org/
https://github.com/icsdataset/hai [164] Common Open Research Emulator (CORE). Accessed: 15-12-2020.
[140] E. Korkmaz, A. Dolgikh, M. Davis, and V. Skormin, “ICS security [Online]. Available: https://www.nrl.navy.mil/itd/ncs/products/core
testbed with delay attack case study,” Proceedings - IEEE Military [165] EPANET. Application for Modeling Drinking Water Distribution
Communications Conference MILCOM, pp. 283–288, 2016. Systems. Accessed: 15-12-2020. [Online]. Available: https://www.epa.
[141] Industrial control system (ics) cyber attack datasets by morris et al. gov/water-research/epanet
Accessed: 27-04-2020. [Online]. Available: https://sites.google.com/a/ [166] S. Hariri, G. Qu, H. Chen, Y. Al-Nashif, and M. Yousif, “Autonomic
uah.edu/tommy-morris-uah/ics-data-sets network security management: Design and evaluation,” ACM Transac-
[142] I. Moreno-Garcia, A. Moreno-Munoz, V. Pallares-Lopez, M. Gonzalez- tions on Autonomous and Adaptive Systems-Special Issue on Adaptive
Redondo, E. J. Palacios-Garcia, and C. D. Moreno-Moreno, “Develop- Learning in Autonomic Communication, 2007.
ment and application of a smart grid test bench,” Journal of Cleaner [167] OPNET Network simulator. Accessed: 14-12-2020. [Online].
Production, vol. 162, pp. 45–60, 2017. Available: https://opnetprojects.com/opnet-network-simulator/
[143] J. Goh, S. Adepu, K. N. Junejo, and A. Mathur, “A dataset to [168] NetToPLCsim - Network extension for Plcsim. Accessed: 16-12-2020.
support research in the design of secure water treatment systems,” [Online]. Available: http://nettoplcsim.sourceforge.net/
in International Conference on Critical Information Infrastructures [169] Getting started with S7-PLCSIM Advanced and sim-
Security. Springer, 2016, pp. 88–99. ulation tables. Accessed: 16-12-2020. [Online]. Avail-
[144] Y. Chen, C. M. Poskitt, and J. Sun, “Learning from mutants: Using code able: https://cache.industry.siemens.com/dl/files/047/109759047/att
mutation to learn and monitor invariants of a cyber-physical system,” 962042/v3/109759047 PLCSIMAdv SimTable DOC V10 en.pdf
in 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 2018, [170] CyberCity allows government hackers to train
pp. 648–660. for attacks. Accessed: 10-01-2021. [Online].
[145] SWaT Simulator. Accessed: 12-01-2021. [Online]. Available: https: Available: https://www.washingtonpost.com/investigations/
//github.com/yuqiChen94/Swat Simulator cybercity-allows-government-hackers-to-train-for-attacks/2012/
[146] J. Goh, S. Adepu, K. N. Junejo, and A. Mathur, “A dataset to support 11/26/588f4dae-1244-11e2-be82-c3411b7680a9 story.html
research in the design of secure water treatment systems,” Lecture Notes [171] CyberCity SANS Holiday Hack 2013 Dataset.
in Computer Science (including subseries Lecture Notes in Artificial Accessed: 10-01-2021. [Online]. Available: https://assets.
Intelligence and Lecture Notes in Bioinformatics), vol. 10242 LNCS, contentstack.io/v3/assets/blt36c2e63521272fdc/bltff8e7c1232f3bcbc/
no. October, pp. 88–99, 2017. 5fbd7be072a3526f28dbed75/sansholidayhack2013.pcap
[147] WUSTL-IIOT-2018 Dataset for ICS (SCADA) Cybersecurity Research. [172] Matlab/Simulink Coder: Generate C and C++ code from Simulink
Accessed: 11-01-2021. [Online]. Available: https://www.cse.wustl.edu/ and Stateflow models. Accessed: 14-12-2020. [Online]. Available:
∼jain/iiot/index.html https://www.mathworks.com/products/simulink-coder.html
[148] M. Zolanvari, M. A. Teixeira, L. Gupta, K. M. Khan, and R. Jain, [173] Summit - Oak Ridge National Laboratory’s 200 petaflop
“Machine Learning-Based Network Vulnerability Analysis of Industrial supercomputer. Accessed: 22-12-2020. [Online]. Available:
Internet of Things,” IEEE Internet of Things Journal, vol. 6, no. 4, pp. https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit/
6822–6834, 2019. [174] November 2019 - TOP500 Supercomputer Sites. Accessed: 22-12-
[149] Sutd-mit international design centre (idc), water distribution (wadi) 2020. [Online]. Available: https://www.top500.org/lists/2019/11/
testbed. Accessed: 06-05-2020. [Online]. Available: https://itrust.sutd. [175] Teach, Learn and Make with Raspberry Pi. Accessed: 14-12-2020.
edu.sg/itrust-labs-home/itrust-labs wadi/ [Online]. Available: https://www.raspberrypi.org/
[150] PowerWorld: The visual approach to electric power systems. Accessed: [176] OSIsoft. Accessed: 12-01-2021. [Online]. Available: http://www.
15-12-2020. [Online]. Available: https://www.powerworld.com/ osisoft.com/
[151] M. Liljenstam, J. Liu, D. Nicol, Y. Yuan, G. Yan, and C. Grier, “Rinse: [177] M. Rollins, Beginning Lego Mindstorms Ev3. Apress, 2014.
The real-time immersive network simulation environment for network [178] Fischertechnik - The Fischertechnik learning environment is used
security exercises (extended version).” Simulation, vol. 82, pp. 43–59, for learning and understanding industry 4.0 applications. Accessed:
01 2006. 05-01-2021. [Online]. Available: https://www.fischertechnik.de/en
[152] P. R. Lyman and C. Georgakis, “Plant-wide control of the tennessee [179] NIST. Tesim on github. Accessed: 31-07-2020. [Online]. Available:
eastman problem,” Computers & chemical engineering, vol. 19, no. 3, https://github.com/usnistgov/tesim
pp. 321–331, 1995. [180] Pnnl control testbed. Accessed: 03-08-2020. [Online]. Available:
[153] L. Argenta and M. Morykwas, “Vacuum-assisted closure: a new method https://controls.pnnl.gov/testbed/
for wound control and treatment: clinical experience.” 1997. [181] Arion: simplifying models for controls research. Accessed: 14-12-
[154] Emulab: A time- and space-shared platform for research, education, 2020. [Online]. Available: https://arion.labworks.org/
and development in distributed systems and networks. Accessed: [182] D. Maynor, Metasploit toolkit for penetration testing, exploit develop-
14-12-2020. [Online]. Available: https://www.emulab.net/ ment, and vulnerability research. Elsevier, 2011.
[155] Matlab/Simulink: Simulation and Model-Based Design. Accessed: [183] T. Morris and W. Gao, “Industrial control system traffic data sets
14-12-2020. [Online]. Available: https://mathworks.com/products/ for intrusion detection research,” IFIP Advances in Information and
simulink.html Communication Technology, vol. 441, pp. 65–78, 2014.
[156] T. R. Alves, M. Buratto, F. M. De Souza, and T. V. Rodrigues, [184] WEKA – The workbench for machine learning. Accessed: 15-01-2021.
“Openplc: An open source alternative to automation,” in IEEE Global [Online]. Available: https://www.cs.waikato.ac.nz/ml/weka/
Humanitarian Technology Conference (GHTC 2014). IEEE, 2014, pp. [185] A. Lemay and J. M. Fernandez, “Providing SCADA network data
585–589. sets for intrusion detection research,” 9th USENIX Workshop on Cyber
[157] AdvancedHMI software. Accessed: 11-01-2021. [Online]. Available: Security Experimentation and Test, CSET 2016, p. 8, 2016.
https://www.advancedhmi.com/ [186] G. Bernieri, M. Conti, and F. Turrin, “Evaluation of machine learning
[158] M. Roesch et al., “Snort: Lightweight intrusion detection for networks.” algorithms for anomaly detection in industrial networks,” in 2019 IEEE
in Lisa, vol. 99, no. 1, 1999, pp. 229–238. International Symposium on Measurements Networking (M N), 2019,
[159] Oracle VM VirtualBox: a powerful x86 and AMD64/Intel64 pp. 1–6.
virtualization product for enterprise as well as home use. Accessed: [187] N. Rodofile. Scada network attack datasets and process logs.
15-12-2020. [Online]. Available: https://www.virtualbox.org/ Accessed: 08-05-2020. [Online]. Available: https://github.com/
[160] K. Kaur, J. Singh, and N. S. Ghumman, “Mininet as software defined qut-infosec/2017QUT DNP3
networking testing platform,” in International Conference on Commu- [188] N. R. Rodofile, T. Schmidt, S. T. Sherry, C. Djamaludin, K. Radke, and
nication, Computing & Systems (ICCCS), 2014, pp. 139–42. E. Foo, “Process control cyber-attacks and labelled datasets on S7comm
[161] Z. Thornton and T. Morris, “ENHANCING A VIRTUAL SCADA critical infrastructure,” Lecture Notes in Computer Science (including
LABORATORY USING SIMULINK,” p. 331, 2015. [Online]. subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
Available: http://link.springer.com/10.1007/978-3-319-26567-4 Bioinformatics), vol. 10343 LNCS, pp. 452–459, 2017.
[162] CRATE - Cyber Range And Training Environment. Accessed: [189] N. Rodofile. Scada network attack datasets and process logs.
11-01-2021. [Online]. Available: https://www.foi.se/en/foi/resources/ Accessed: 08-05-2020. [Online]. Available: https://github.com/
crate---cyber-range-and-training-environment.html qut-infosec/2017QUT S7comm
 45

[190] Ics lab: 4sics ics lab pcap file. Accessed: 27-04-2020. [Online]. [210] M. Housh and Z. Ohar, “Model-based approach for cyber-physical
Available: https://www.netresec.com/?page=PCAP4SICS attack detection in water distribution systems,” Water Research, vol.
[191] T. C. Yu, J. Y. Huang, I. E. Liao, and K. F. Kao, “Mining anomaly 139, pp. 132–143, 2018.
communication patterns for industrial control systems,” Australasian [211] Y. Kim and H. K. Kim, “Anomaly Detection using Clustered
Universities Power Engineering Conference, AUPEC 2018, 2018. Deep One-Class Classification,” in 2020 15th Asia Joint Conference
[192] J. M. Beaver, R. C. Borges-Hink, and M. A. Buckner, “An evaluation on Information Security (AsiaJCIS). IEEE, aug 2020, pp. 151–157.
of machine learning methods to detect malicious SCADA communica- [Online]. Available: https://ieeexplore.ieee.org/document/9194140/
tions,” Proceedings - 2013 12th International Conference on Machine [212] iTrust, Centre for Research in Cyber Security, Singapore University
Learning and Applications, ICMLA 2013, vol. 2, pp. 54–59, 2013. of Technology and Design, iTrust Labs Dataset Info. Accessed:
[193] K. Demertzis, L. Iliadis, and S. Spartalis, “A Spiking One-Class 24-04-2020. [Online]. Available: https://itrust.sutd.edu.sg/itrust-labs
Anomaly Detection Framework for Cyber-Security on Industrial datasets/dataset info
Control Systems,” in EANN 2017: Engineering Applications of Neural [213] R. Taormina, S. Galelli, H. Douglas, N. O. Tippenhauer, E. Salomons,
Networks, 2017, vol. 2, no. August, pp. 122–134. [Online]. Available: and A. Ostfeld, “A toolbox for assessing the impacts of cyber-physical
http://link.springer.com/10.1007/978-3-319-65172-9 11 attacks on water distribution systems. environmental modelling soft-
[194] Dataset for cybersecurity research in industrial control systems. ware,” Environmental Modelling Software, vol. 112, pp. 46–51, 02
Accessed: 06-05-2020. [Online]. Available: http://perception.inf.um. 2019.
es/ICS-datasets/ [214] A. Erba, R. Taormina, S. Galelli, M. Pogliani, M. Carminati,
[195] G. K. Ndonda and R. Sadre, “A Public Network Trace of a S. Zanero, and N. O. Tippenhauer. Constrained concealment
Control and Automation System,” ArXiv, 2019. [Online]. Available: attacks on reconstruction-based anomaly detectors in industrial
http://arxiv.org/abs/1908.02118 control systems. Accessed: 07-12-2020. [Online]. Available: https:
[196] S. R. Ndonda Gorby Kabasele. Hvac traces. Accessed: 21-05-2020. //github.com/scy-phy/ICS-Evasion-Attacks
[Online]. Available: https://github.com/gkabasele/HVAC Traces [215] ——, “Constrained concealment attacks against reconstruction-based
[197] A. Lemay. Modbus dataset from cset 2016. Accessed: 08-05-2020. anomaly detectors in industrial control systems,” in Proceedings of the
[Online]. Available: https://github.com/antoine-lemay/Modbus dataset Annual Computer Security Applications Conference (ACSAC), 2020.
[198] S. D. Anton, S. Kanoor, D. Fraunholz, and H. D. Schotten, “Evaluation [216] M. Kravchik and A. Shabtai, “Efficient Cyber Attacks Detection
of machine learning-based anomaly detection algorithms on an indus- in Industrial Control Systems Using Lightweight Neural Networks
trial modbus/tcp data set,” in Proceedings of the 13th International and PCA,” ArXiv, pp. 1–18, 2019. [Online]. Available: http:
Conference on Availability, Reliability and Security, 2018, pp. 1–9. //arxiv.org/abs/1907.01216
[217] S. Pan, T. Morris, and U. Adhikari, “A specification-based intrusion
[199] I. Frazão, P. H. Abreu, T. Cruz, H. Araújo, and P. Simões, “Denial of
detection framework for cyber-physical environment in electric power
service attacks: Detecting the frailties of machine learning algorithms
system,” International Journal of Network Security, vol. 17, no. 2, pp.
in the classification process,” Lecture Notes in Computer Science
174–188, 2015.
(including subseries Lecture Notes in Artificial Intelligence and Lecture
[218] ——, “Classification of disturbances and cyber-attacks in power sys-
Notes in Bioinformatics), vol. 11260 LNCS, no. 700665, pp. 230–235,
tems using heterogeneous time-synchronized data,” IEEE Transactions
2019.
on Industrial Informatics, vol. 11, no. 3, pp. 650–662, 2015.
[200] I. Frazão, P. Henriques Abreu, T. Cruz, H. Araujo, and
[219] ——, “Developing a Hybrid Intrusion Detection System Using Data
P. Simoes. Modbus tcp scada #1 dataset. Accessed: 30-04-
Mining for Power Systems,” IEEE Transactions on Smart Grid, vol. 6,
2020. [Online]. Available: https://github.com/tjcruz-dei/ICS PCAPS/
no. 6, pp. 3104–3113, 2015.
releases/tag/MODBUSTCP%231
[220] J. Fürnkranz and G. Widmer, “Incremental reduced error pruning,” in
[201] P. Radoglou-Grammatikis, I. Siniosoglou, T. Liatifis, A. Kourouniadis, Machine Learning Proceedings 1994. Elsevier, 1994, pp. 70–77.
K. Rompolos, and P. Sarigiannidis, “Implementation and Detection [221] Y. Freund and R. E. Schapire, “A decision-theoretic generalization of
of Modbus Cyberattacks,” in 2020 9th International Conference on on-line learning and an application to boosting,” Journal of computer
Modern Circuits and Systems Technologies (MOCAST). IEEE, 2020, and system sciences, vol. 55, no. 1, pp. 119–139, 1997.
pp. 1–4. [Online]. Available: https://ieeexplore.ieee.org/document/ [222] H.-K. Shin, W. Lee, J.-H. Yun, and H. Kim, “HAI 1.0: Hil-based
9200287/ augmented ICS security dataset,” in 13th USENIX Workshop on Cyber
[202] Peterson, d., wightman, r.: Digital bond s4x15 ics village ctf pcap Security Experimentation and Test (CSET 20). USENIX Association,
files. Accessed: 27-04-2020. [Online]. Available: https://www.netresec. Aug. 2020. [Online]. Available: https://www.usenix.org/conference/
com/?page=DigitalBond S4 cset20/presentation/shin
[203] R. C. Borges Hink, J. M. Beaver, M. A. Buckner, T. Morris, U. Ad- [223] D. Li, D. Chen, B. Jin, L. Shi, J. Goh, and S.-K. Ng, “Mad-gan:
hikari, and S. Pan, “Machine learning for power system disturbance and Multivariate anomaly detection for time series data with generative
cyber-attack discrimination,” 7th International Symposium on Resilient adversarial networks,” in International Conference on Artificial Neural
Control Systems, ISRCS 2014, 2014. Networks. Springer, 2019, pp. 703–716.
[204] Singapore University of Technology and Design (SUTD). Dataset [224] Ö. Yüksel, J. D. Hartog, and S. Etalle, “Reading between the fields:
Characteristics. Accessed: 13-01-2021. [Online]. Available: https: Practical, effective intrusion detection for industrial control systems,”
//itrust.sutd.edu.sg/itrust-labs datasets/dataset info/ Proceedings of the ACM Symposium on Applied Computing, vol. 04-
[205] D. Myers, S. Suriadi, K. Radke, and E. Foo, “Anomaly detection 08-Apri, pp. 2063–2070, 2016.
for industrial control systems using process mining,” Computers [225] C. Feng, T. Li, and D. Chana, “Multi-level Anomaly Detection in In-
and Security, vol. 78, pp. 103–125, 2018. [Online]. Available: dustrial Control Systems via Package Signatures and LSTM Networks,”
https://doi.org/10.1016/j.cose.2018.06.002 Proceedings - 47th Annual IEEE/IFIP International Conference on
[206] QUT S7 Communication by Myers et al. dataset. Accessed: 19-12- Dependable Systems and Networks, DSN 2017, pp. 261–272, 2017.
2020. [Online]. Available: https://cloudstor.aarnet.edu.au/plus/index. [226] K. Demertzis, L. Iliadis, and I. Bougoudis, “Gryphon: a semi-
php/s/9qFfeVmfX7K5IDH supervised anomaly detection system based on one-class evolving
[207] M. Abdelaty, R. Doriguzzi-Corin, and D. Siracusa, “DAICS: A Deep spiking neural network,” Neural Computing and Applications, vol. 32,
Learning Solution for Anomaly Detection in Industrial Control Sys- no. 9, pp. 4303–4314, 2020.
tems,” ArXiv, pp. 1–12, 2020. [227] A. Mansouri, B. Majidi, and A. Shamisa, “Metaheuristic neural
[208] R. Taormina, S. Galelli, N. O. Tippenhauer, E. Salomons, A. Ostfeld, networks for anomaly recognition in industrial sensor networks with
D. G. Eliades, M. Aghashahi, R. Sundararajan, M. Pourahmadi, M. K. packet latency and jitter for smart infrastructures,” International
Banks, B. M. Brentan, E. Campbell, G. Lima, D. Manzi, D. Ayala- Journal of Computers and Applications, vol. 0, no. 0, pp. 1–10, 2018.
Cabrera, M. Herrera, I. Montalvo, J. Izquierdo, E. Luvizotto, S. E. [Online]. Available: https://doi.org/1206212X.2018.1533613
Chandy, A. Rasekh, Z. A. Barker, B. Campbell, M. E. Shafiee, [228] J. Jägersküpper, “How the (1+1) es using isotropic mutations minimizes
M. Giacomoni, N. Gatsis, A. Taha, A. A. Abokifa, K. Haddad, C. S. positive definite quadratic forms,” Theoretical Computer Science, vol.
Lo, P. Biswas, M. Fayzul, B. Kc, S. L. Somasundaram, M. Housh, and 361, no. 1, pp. 38–56, 2006.
Z. Ohar, “Battle of the Attack Detection Algorithms: Disclosing cyber [229] S. Mirjalili, S. M. Mirjalili, and A. Lewis, “Grey wolf optimizer,”
attacks on water distribution networks,” Journal of Water Resources Advances in engineering software, vol. 69, pp. 46–61, 2014.
Planning and Management, vol. 144, no. 8, 2018. [230] Á. L. Perales Gómez, L. Fernández Maimó, A. Huertas Celdrán, F. J.
[209] Batadal datasets. Accessed: 08-05-2020. [Online]. Available: http: Garcı́a Clemente, M. Gil Pérez, and G. Martı́nez Pérez, “SafeMan: A
//www.batadal.net/data.html unified framework to manage cybersecurity and safety in manufacturing
 46

industry,” Software - Practice and Experience, no. April, pp. 1–21,

2020.
[231] Y. Li, X. Ji, C. Li, X. Xu, W. Yan, X. Yan, Y. Chen, and W. Xu, “Cross-
domain Anomaly Detection for Power Industrial Control System,”
ICEIEC 2020 - Proceedings of 2020 IEEE 10th International Confer-
ence on Electronics Information and Emergency Communication, pp.
383–386, 2020.
[232] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed
analysis of the KDD CUP 99 data set,” IEEE Symposium on Com-
putational Intelligence for Security and Defense Applications, CISDA
2009, no. Cisda, pp. 1–6, 2009.
[233] D. L. Mills, “Internet time synchronization: the network time protocol,”
IEEE Transactions on communications, vol. 39, no. 10, pp. 1482–1493,
1991.
[234] P. Schneider and K. Böttinger, “High-performance unsupervised
anomaly detection for cyber-physical system networks,” Proceedings
of the ACM Conference on Computer and Communications Security,
pp. 1–12, 2018.
[235] S. D. Anton, L. Ahrens, D. Fraunholz, and H. D. Schotten, “Time is of
the essence: Machine learning-based intrusion detection in industrial
time series data,” in 2018 IEEE International Conference on Data
Mining Workshops (ICDMW). IEEE, 2018, pp. 1–6.
[236] J. Luswata, P. Zavarsky, B. Swar, and D. Zvabva, “Analysis of scada
security using penetration testing: A case study on modbus tcp pro-
tocol,” in 2018 29th Biennial Symposium on Communications (BSC).
IEEE, 2018, pp. 1–5.
[237] D. Bond. Quickdraw snort. Accessed: 09-10-2020. [Online]. Available:
https://github.com/digitalbond/Quickdraw-Snort
[238] Nmap: the Network Mapper - Free Security Scanner. Accessed:
11-12-2020. [Online]. Available: hhttps://nmap.org/
[239] Sutd security showdown 2017. Accessed: 27-04-2020.
[Online]. Available: https://itrust.sutd.edu.sg/scy-phy-systems-week/
2017-2/s317-event/
[240] F. Turrin, A. Erba, N. O. Tippenhauer, and M. Conti, “A statistical
analysis framework for ics process datasets,” in Proceedings of the
2020 Joint Workshop on CPS&IoT Security and Privacy, 2020, pp.
25–30.
[241] C. M. Ahmed, J. Zhou, and A. P. Mathur, “Noise matters: Using
sensor and process noise fingerprint to detect stealthy cyber attacks
and authenticate sensors in cps,” in Proceedings of the 34th Annual
Computer Security Applications Conference, 2018, pp. 566–581.
[242] Q. Lin, S. Adepu, S. Verwer, and A. Mathur, “Tabor: A graphical
model-based approach for anomaly detection in industrial control
systems,” in Proceedings of the 2018 on Asia Conference on Computer
and Communications Security, 2018, pp. 525–536.
[243] C. Feng, V. R. Palleti, A. Mathur, and D. Chana, “A systematic
framework to generate invariants for anomaly detection in industrial
control systems.” in NDSS, 2019.
[244] J. L. Torres, C. A. Catania, and E. Veas, “Active learning approach
to label network traffic datasets,” Journal of Information Security and
Applications, vol. 49, p. 102388, 2019.
[245] M. Zolanvari, M. A. Teixeira, and R. Jain, “Effect of Imbalanced
Datasets on Security of Industrial IoT Using Machine Learning,”
in 2018 IEEE International Conference on Intelligence and Security
Informatics (ISI). IEEE, nov 2018, pp. 112–117. [Online]. Available:
https://ieeexplore.ieee.org/document/8587389/
[246] D. Ramyachitra and P. Manikandan, “Imbalanced dataset classification
and solutions: a review,” International Journal of Computing and
Business Research (IJCBR), vol. 5, no. 4, 2014.
[247] S. K. Lim, Y. Loo, N. Tran, N. Cheung, G. Roig, and Y. Elovici,
“Doping: Generative data augmentation for unsupervised anomaly
detection with gan,” in 2018 IEEE International Conference on Data
Mining (ICDM), 2018, pp. 1122–1127.
[248] G. O. Diaz and V. Ng, “Modeling and prediction of online product
review helpfulness: a survey,” in Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), 2018, pp. 698–708.