See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344726578

Deep Reinforcement Adversarial Learning Against Botnet Evasion Attacks

Article in IEEE Transactions on Network and Service Management · October 2020

DOI: 10.1109/TNSM.2020.3031843

CITATIONS READS
75 584

5 authors, including:

Giovanni Apruzzese Mauro Andreolini

University of Liechtenstein Università degli Studi di Modena e Reggio Emilia
41 PUBLICATIONS 943 CITATIONS 57 PUBLICATIONS 960 CITATIONS

SEE PROFILE SEE PROFILE

Mirco Marchetti Andrea Venturi

Università degli Studi di Modena e Reggio Emilia Università degli Studi di Modena e Reggio Emilia
77 PUBLICATIONS 2,533 CITATIONS 7 PUBLICATIONS 99 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Giovanni Apruzzese on 18 November 2020.

The user has requested enhancement of the downloaded file.

IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 1

Deep Reinforcement Adversarial Learning

against Botnet Evasion Attacks
Giovanni Apruzzese∗ , Mauro Andreolini† , Mirco Marchetti‡ , Andrea Venturi‡ , Michele Colajanni§
∗ Hilti Chair of Data and Application Security – University of Liechtenstein, Vaduz, Liechtenstein
† Department of Physics, Computer Science and Mathematics – University of Modena, Italy
‡ Department of Engineering “Enzo Ferrari” – University of Modena, Italy
§ Department of Informatics, Science and Engineering – University of Bologna, Italy

{giovanni.apruzzese, mauro.andreolini, mirco.marchetti, andrea.venturi, michele.colajanni}@unimore.it

Abstract—As cybersecurity detectors increasingly rely on to severe consequences [6], [7]. Existing countermea-
machine learning mechanisms, attacks to these defenses sures are at an early stage and they suffer from several
escalate as well. Supervised classifiers are prone to adver- drawbacks. For example, they are effective only against
sarial evasion, and existing countermeasures suffer from
many limitations. Most solutions degrade performance in predictable attack strategies or they can be applied only
the absence of adversarial perturbations; they are unable to specific ML algorithms. Moreover, their detection rate
to face novel attack variants; they are applicable only tends to degrade when the system is not subject to
to specific machine learning algorithms. We propose the evasion strategies [2], [8]–[11].
first framework that can protect botnet detectors from In the context of adversarial evasion attacks against
adversarial attacks through Deep Reinforcement Learning
mechanisms. It automatically generates realistic attack sam- network intrusion detection systems (NIDS) based on
ples that can evade detection, and it uses these samples to ML, we propose a novel approach that leverages Deep
produce an augmented training set for producing hardened Reinforcement Learning (DRL) to increase the robust-
detectors. In such a way, we obtain more resilient detectors ness of detectors relying on network flow analyses. Our
that can work even against unforeseen evasion attacks with proposal allows an automatic generation of realistic ad-
the great merit of not penalizing their performance in
the absence of specific attacks. We validate our proposal versarial samples that preserve their underlying malicious
through an extensive experimental campaign that considers logic and can evade detection with high probability.
multiple machine learning algorithms and public datasets. The detector is hardened by means of an adversarial
The results highlight the improvements of the proposed training procedure based on automatically generated
solution over the state-of-the-art. Our method paves the samples [10], [12]. To the best of our knowledge, this
way to novel and more robust cybersecurity detectors based
on machine learning applied to network traffic analytics. paper represents the first proposal that exploits deep re-
inforcement learning for the purpose of hardening botnet
Index Terms—Adversarial attack, Machine learning, Net- detectors through adversarial training. In our research
work Intrusion Detection, Deep Reinforcement Learning,
Botnet we consider the real constraints that characterize the
cybersecurity domain. They include the necessity of
creating adversarial samples through small and feasible
I. I NTRODUCTION modifications, but also the implication that the attacker

M ACHINE Learning (ML) approaches are be-

ing increasingly applied to cybersecurity where
data-driven detection algorithms outperform traditional
has limited queries to evade detection. Moreover, adver-
sarial training requires accurate analyses because it may
even decrease detection performance in the absence of
signature-based methods against novel forms of attacks adversarial attacks (e.g., [13]).
[1], [2]. The problem is that defensive systems have The implementation of the proposed approach pro-
to deal with proactive enemies who are turning their duces a framework that can be used to attack state-of-the-
attentions against modern ML detectors. The adopted art botnet detectors and to defend them against known
classifiers are vulnerable to the so called adversarial and novel evasion strategies. With respect to existing
evasion attacks aiming to thwart the ML model through works, our framework protects the detectors against
specific malicious samples that can remain undetected unforeseen evasion attempts without compromising the
(e.g., [3]–[5]). The robustness of cybersecurity detectors detection rate in the absence of adversarial attacks. Our
is a critical issue because few misclassifications can lead proposal is applicable to botnet detectors relying on
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 2

different machine learning classifiers. Its effectiveness malware analysis [8], [28]–[30], while there are few
is demonstrated in realistic scenarios represented by results on network intrusion detection which represents
multiple datasets of enterprise network flows. An exten- the focus of this paper. Attacks against detectors were
sive experimental campaign shows the benefits of our investigated by the same authors and by other researchers
method against previous literature in several terms. The in [4], [5], [10], [31], [32]. Most of these papers highlight
samples generated by our method increase the resilience that even small adversarial perturbations against machine
of botnet classifiers against existing and novel evasion learning detectors can significantly reduce their detection
attacks through adversarial training. The improved detec- rates, while a careless insertion of malicious samples may
tors maintain their performance even in the absence of favor detection.
adversarial attacks. By varying the amount of malicious In Figure 1 we outline the typical scenario consisting
samples to include in the augmented training dataset, we of an enterprise network with many internal hosts, where
also show that our autonomous solution increases the at least one machine has been compromised by a botnet
detection rate by requiring less samples than approaches malware communicating with a Command and Control
entailed by manual adversarial training. In the best cases, system (CnC). The network traffic is inspected by a
by adding just 1% of adversarial samples to the training NIDS that analyzes network flows to identify botnet
set, we are able to increase the detection rate by nearly activities by means of supervised machine learning meth-
25%. ods. Our threat model represents a realistic gray-box
The remainder of this paper is structured as follows. attack where the adversary has partial information about
Section II motivates our paper and describes the threat the defensive system. We consider unrealistic to assume
model. Section III presents the proposed method. Sec- that the attacker knows the precise internal configuration
tion IV details the experimental settings for the evalua- of the model, the full set of features, and the complete
tion. Section V discusses the results of the experiments. training set of the detector as in other papers.
Section VI compares our paper against related work.
Section VII concludes the paper with some final remarks Flow ML-based
Exporter NIDS
and future research directions.
Border
II. M OTIVATION Internal Router
Network Internet

The threats posed by botnets are increasing (e.g., [6],

[7], [14]) and the difficulties of their detection represent
a real menace for modern organizations. Botnet detection
is the topic of a large body of literature where traditional Figure 1: Considered threat scenario.
methods based on full-packet captures are replaced by
recent solutions relying on both supervised and unsu- The attacker can realistically assume that a modern
pervised ML approaches [2], [9] that analyze network network is monitored by a ML-based NIDS that has been
flows [15]–[19]. Unfortunately, the growing popularity of trained on network flows [15], [33]. They do not know
these botnet detectors is arousing the interest of adver- the complete set of features, but they can expect that
saries that plan and produce new evasion attacks [20]. As time- and data-related information are included because
a consequence, it is important to devise novel defensive most flow-based detectors use them as reported by liter-
approaches that can improve the detector robustness ature on botnet detectors (e.g., [4], [16], [17], [34]–[37]).
against these evasion attempts. We consider detectors that analyze the most important
Adversarial attacks to ML-based detectors aim to gen- features captured by network flows, reported in Table I,
erate specific samples that induce the model to produce where the gray background denotes the features on which
an incorrect output [8]. These adversarial samples can be the attacker operates. They can issue commands to the
introduced during the training phase (so called poisoning infected machines through the Command and Control
attacks [21]) or at inference time [22] that is of interest infrastructure. The attacker tries to evade detection by
for this paper. Within this latter category, we consider slightly modifying the botnet communications. These
evasion attempts where the goal of the attacker is to modifications alter the original flow characteristics and
induce a misclassification of malicious samples. Many result in adversarial perturbations that may evade de-
papers tackle this issue in image and speech processing tection. For example, typical adversarial samples may
domains (e.g., [23]–[27]). Surprisingly, the cybersecurity present different durations, different amounts of ex-
context is new although it has inherently to deal against changed bytes or transmitted packets. Modifying these
intentional attackers. Most studies focus on spam and metrics affects multiple attributes in Table I because
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 3

some of them are derived features. We consider these that is recognized by the agent; the botnet detector
alterations because we are interested in modifications Db leverages a ML classifier trained on a dataset T
that do not compromise the underlying logic of the of network flows containing legitimate and malicious
botnet. Similar perturbations can be easily obtained by samples including some pattern of the botnet b. Let D(T b )
inserting short communication delays, or adding random denote this detector.
junk data in the transmitted packets. These operations The Reward depends on the output of the detector
require small changes to the source code of the botnet to the sample produced by the agent. A correct and an
malware variant without compromising its logic [38]. incorrect classification is associated to a positive and a
Metamorphic malware operates in a similar way [39]. null reward, respectively. The agent continues to modify
the sample until it is able to evade detection or after a
Table I: Traffic features of the considered detectors. maximum amount of failures.
# Feature name Type The Action Space includes the set of perturbations that
1,2 Source/Dest IP address type Bool an agent can introduce in a malicious sample. Our focus
3,4 Source/Dest port Num
5 Flow direction Bool
is on flow-based botnet detectors and our goal is to gener-
6 Connection state Cat ate samples that an attacker can realistically reproduce.
7 Duration (seconds) Num As described in Section II, we limit our Action Space
8,9 Source/Dest ToS Num
10,11 Source/Dest bytes Num
to small increments of few essential traffic flow features
12 Total pkts Num (duration, sent bytes, received bytes, transmitted packets)
13 Total bytes Num and correlated features.
14,15 Source/Dest port type Cat
16 Bytes per second Num We consider agents based on two deep reinforcement
17 Bytes per packet Num learning algorithms that have been applied in cybersecu-
18 Packets per second Num rity (e.g., [40], [42], [43]): one is based on the off-policy
19 Ratio of Source/Dest bytes Num
Double Deep Q-Network; the other one is based on the
on-policy Deep State–action–reward–state–action. We
III. P ROPOSED METHOD consider DRL approaches because they address complex
tasks better than basic RL methods. These latter achieve
The proposed approach leverages Deep Reinforcement poor performance when the problem requires to evaluate
Learning to generate realistic adversarial samples that many possibilities in terms of feasible states and related
preserve their malicious logic and are able to evade detec- action-space [44]. The function representing the value
tion. These samples are used as a mean for hardening the of an action can be seen as a table that maps all states
original detector through adversarial training [12]. The and all actions to the expected long-term return. In our
expectation is that the resulting botnet detector achieves case, the dimension of this table is large and compiling it
better detection rates than its initial version. The method requires high computational costs. Approaches based on
consists of three phases that are represented in Figure 2 DRL leverage deep neural networks to estimate (instead
and detailed below. of fully creating) the value function. In such a way, they
avoid the two-dimensional representation of the value
A. Preparation function and devise models with general capabilities that
The goal of the first phase is to create a DRL agent are able to achieve better results [45].
that is capable of autonomously generating evasive ad- To the best of our knowledge, we are the first authors
versarial samples against botnet detectors on the basis of to consider the 2DQN and Sarsa algorithms to both evade
network flows belonging to some botnet b. As anticipated and harden a botnet detector. Other authors in [4] con-
in Section II, modeling a realistic scenario requires to sider Deep Q-Learning but only for offensive purposes.
preserve the malware underlying logic. Moreover, the Double Deep Q-Network (2DQN): was proposed
attackers are also constrained to the number of sam- in [46], and it leverages the synergy of the original Dou-
ples that they can submit to the detector to guess the ble Q-Learning approach with Deep Q-Networks [46]. In
underlying ML logic. This agent learns how to make these methods, the system uses the same values to both
the best decisions through an autonomous trial-and-error choose and evaluate the effects of an action, thus induc-
approach [40], [41] where at each agent’s choice, which ing some over-estimations. In our context, this approach
is chosen among the defined Action Space, corresponds tends to generate samples that deviate significantly from
a Reward provided by the Environment. their initial variant. The aim of 2DQN is to decrease
The Environment includes two elements: the state such over-estimations. The main intuition is to determine
generator transforms the input flow sample in a format the best decision by splitting the optimization into action
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 4

Phase I Phase II Phase III

Botnet Botnet Botnet

Detector Malicious Detector Malicious Detector
Flows Flows Dataset

Trained DRL AUTOMATIC SAMPLE Hardened

PREPARATION Adversarial HARDENING
Agent GENERATION (ATTACK) Detector
Flows

Figure 2: The three phases of the proposed solution.

selection and evaluation by relying on two deep neural differences that can be summarized as follows. On-
networks. The 2DQN algorithm is characterized by the policy techniques such as Sarsa adopt a linear approach.
following formula [46]: They learn to choose the best action by following and
Yt2DQN
≡ Rt+1 +γQ(St+1 , argmax Q(St+1 , a; θt ); θt ) improving one policy, which for each state suggests a
+ −
a single action. On the other hand, off-policy techniques
(1) such as Q-learning (the precursor of 2DQN) use an
where Yt2DQN is the target function at time t, and exploratory policy that suggests multiple actions to play
γ ∈ [0, 1] is the discount factor used for the reward; Rt+1 in each state. These actions are evaluated and chosen by
is the immediate reward and St+1 is the resulting next a separate core policy, and the learning procedure aims to
state (at time t+1); θ+ and θ− denote the two different improve this second core policy [48]. Training on-policy
deep neural networks; Q(·, a; θt ) is the function that methods requires more iterations but they are more robust
regulates the updating procedure at time t: in particular, than off-policy algorithms.
a and θ are used to denote the vector of action values
associated to the θ network. The target network θ− is
used to estimate the value selected by the online network
θ+ . In summary, 2DQN adopts a greedy-policy selection
method, which results in a training process with fast
and low cost iterations. Our expectation is that a similar
approach will generate agents that require few queries to
evade detection.
Deep State-action-reward-state-action (Sarsa): is a
DRL algorithm [47] that fosters a risk-adverse strategy.
Unlike the greedy approach adopted by 2DQN, here the Figure 3: Sarsa and Q-Learning algorithms
learner updates its parameters with the action determined (source: [48]).
by the policy. We consider the deep learning variant of
Sarsa which relies on a deep neural network θ to estimate The final output of this first Preparation phase is the
the action values. The algorithm is based on the following DRL agent A(b) which is trained to generate adversarial
update equation [47]: samples on the basis of network flows belonging to the
botnet b contained in T .
YtSarsa = Rt+1 + γQ(St+1 , at+1 ; θ) − Q(St , at ; θ)
(2)
where the same notation of Eq. 1 is adopted. The function B. Automatic sample generation (Attack)
that regulates the update procedure does not include In this phase, the trained DRL agent A(b) is used
the argmax operator because of the more conservative to produce samples that are able to evade a botnet
strategy characterizing Sarsa. Our expectation is that detector D(T ) similarly to an adversarial attack scenario.
agents based on Sarsa will try to evade detection by It is important to observe that the targeted detector can
preferring smaller modifications, at the expense of an even be different from the one adopted for training the
increased number of iterations. agent [21]. The methodology is outlined in Figure 4.
Our paper investigates the effectiveness of on-policy The system accepts a malicious flow of botnet b as its
and off-policy methods, as they belong to complemen- input which is sent to the state generator (step 1 ) and
tary DRL paradigms. We report in Figure 3 the main then forwarded to the agent A(b) (step 2 ). After that,
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 5

the agent communicates the best action to modify the IV. E XPERIMENTAL SETTINGS
sample to the state generator (step 3 ) that applies the We now describe and motivate the experimental envi-
modification and issues the sample to the detector D(T ) ronment used for our evaluation. We start by presenting
(step 4 ). If the evasion is successful, then the modified the testbed and the specifics of the target detectors, and
sample is saved in a dedicated dataset of adversarial flows then detail the configuration of the proposed framework.
(step 5 ). Otherwise, the process is re-activated and the Finally, we report the characteristics of the considered
sample is further modified until it is able to evade D(T ) adversarial attacks.
or until a maximum amount of attempts Qmax is reached.
The agent A(b) does not receive any reward in this phase
because it has already been trained. A. Datasets
As an example, assume a malicious flow f ∈ b in input: We consider two datasets in the experimental cam-
the system may opt to increase the duration of f thus paign: the CTU [49] and the BOTNET [50], that contain
obtaining the sample f 0 , and submits it to the detector labelled collections of millions of network flows gen-
D(T ). If the evasion is successful, then f 0 is added to the erated by benign and malicious devices in networks of
dataset of the adversarial samples. Otherwise, the agent hundreds of hosts representing modern enterprises. The
further modifies f 0 by increasing another (or the same) heterogeneous environments captured by these datasets
feature and repeats the submission process. are appreciated by related literature in botnet detec-
The output of this procedure is a set of adversarial tion [4], [34], [35], [51]. The malicious flows in these
flows GA D (b) that are perturbed versions of the botnet b datasets belong to different botnet families. As recom-
flows that is, they have been altered by the agent A(b) in mended by the state-of-the-art [4], [5], [8], [52], [53] we
order to evade the detector D(T ). The flows contained use ensembles of classifiers, in which each classifier is
in GAD (b) can then be used to harden the detector. devoted to a specific botnet variant.
We report in Table II some samples of network flows
Botnet b
that are included in these datasets with the corresponding
Action:
Increment Duration
Flows
features. For the sake of readability, we omit the derived
Increment InBytes
Increment OutBytes Malicious
features.
Increment TotPkts Sample
3 1 Environment
Trained
Agent State Generator B. Detectors
A(b) 2
No
5
Adversarial
The botnet detectors are based on two famous ma-
Adversarial
II: AUTOMATIC SAMPLE evasion?
Yes Sample
Flows
G DA(b) chine learning classifiers: Random Forest (RF) and
GENERATION (ATTACK) 4
Detector
D (T)
Wide and Deep (WnD). RF is recognized in literature
as one of the most proficient techniques for cyber detec-
Figure 4: Automatic generation of adversarial samples. tion [4], [12], [53], [54].
WnD is a deep learning-based approach proposed by
Google [55] that, to the best of our knowledge, has
C. Hardening never been evaluated for botnet detection. We consider
The final phase leverages the adversarial training it due to its appreciable results in other classification
paradigm to harden the botnet detector. This goal is contexts [56]. Both algorithms adopt the feature set
achieved by re-training the detector D(T ) through an reported in Table I.
augmented training set GA D (b) that includes the adver- The training procedure of each classifier follows the
sarial samples generated during the previous phase. A best practices in related literature [17], [31]: 80% of
threshold Ψ specifies the percentage of adversarial sam- samples are used for training and 20% for testing; the
ples that are introduced in the initial training dataset to samples are distributed in a legitimate-to-illegitimate
generate the augmented dataset. This latter may contain ratio of 20:1. We report the amount of samples used for
also some additional benign flows to maintain a realistic training and testing in Table III, where five famous botnet
ratio of legitimate-to-illegitimate samples. We are the families (included in the CTU and BOTNET datasets) are
first to evaluate the sensitivity of the detector as a considered. We exclude from the evaluation the botnets
function of different Ψ percentages in Section V. with too few samples in the datasets because they yield
At the end, this phase yields a hardened version of the under-performing detectors. Each detector consists of an
detector D(T ) that is trained on the adversarial samples ensemble of five classifiers, each trained on a specific
A
GA D (b) and that we denote as D (T ). botnet family (see Section IV-A).
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 6

Table II: Example of network flows included in the datasets.

S IP D IP S Port D Port Dir State Dur S ToS D ToS S Bytes D Bytes T Pkts S Port t D Port t
int ext 43458 80 mo RA 0.09 0 0 264 0 4 high known
ext int 22 10005 bi CON 192.62 0 0 4968 1413 40 known high

Table III: Datasets used for training and testing. Total agent relying on the algorithm Sarsa that is trained to
samples: CTU: 3 431 629; BOTNET: 189 352. evade the RF(CTU) detector. In the remainder we omit
Dataset Family
Overall Training set Testing set the considered network because all experiments consider
Malicious Malicious Benign Malicious Benign
Neris 80 097 60 071 1 201 455 20 026 400 485 agents and detectors that operate on the same network.
Rbot 27 509 20 631 412 635 6 878 137 545
CTU

Virut 32 347 24 260 485 205 8 087 161 735

Menti 2 825 2 118 42 375 707 14 125
Murlo 1 106 829 16 590 277 5 530
Neris 3 685 2 763 55 275 922 38 425
The trained agents generate evasive samples against all
BOTNET

Rbot
Virut
3 685
878
2 763
658
55 275
13 170
922
220
38 425
4 390
detectors. The goal is to show that our trained agents are
Menti 3 685 2 763 55 275 922 38 425 not only effective at attacking the same detector used in
Murlo 3 685 2 763 55 275 922 38 425
the preparation phase, but also against different detectors.
For instance, we use 2DQN to attack both RF and WnD
Combining the two network scenarios (CTU and BOT- by considering three values of Qmax =(1, 5, 80). The first
NET) and the two detectors (RF and WnD), we have two values are used to emulate a realistic attack scenario,
four combinations. For example, the detector RF(CTU) compliant with the considered threat model, where the
consists of an ensemble of 5 random forest classifiers adversary cannot see the output of the detector (Qmax =1)
where each is trained on a botnet family of the CTU or can leverage a limited number of queries (Qmax =5).
dataset. The last value is used for hardening purposes because the
defenders can freely access and query the detector. In this
attack phase, the agents modify every available malicious
C. Framework
sample into an evasive network flow. For each value
The training of the agents is based on RF because it of Qmax , the agents generate two sets of adversarial
outperforms other classifiers [5], [51], [54], [57]. When flows for every detector. For example, we obtain the sets
an agent wants to modify a sample, it selects a feature G2DQN and GSarsa for the RF detector.
RF RF
and then increments it by a chosen amount. An important
choice of our approach is to generate novel adversarial
samples that are realistic and remain consistent with traf- We use the sets of adversarial samples that are pro-
fic features. For these reasons, we limit the increment of duced by our agents to create augmented training sets
each selected feature to at most two units. For example, that improve the robustness of the detectors against eva-
an agent can increase the duration feature by 1 or 2 sion attacks. The effects of adversarial training are stud-
seconds or can modify the Total Pkts by increasing it of ied as a function of different amounts of injected sam-
1 or 2 packets. ples. Existing proposals on adversarial training simply
Unlike attacks occurring in the problem space inject a fixed amount of samples and proceed to measure
(e.g., [28]), our agent generates adversarial samples by the results (e.g., [12], [61]). We adopt a more realistic
directly modifying the malicious network flows. As the approach that is necessary for devising cybersecurity
perturbations are applied directly to the features, the solutions in real scenarios. For this reason, we consider to
modified samples must not contain values causing incon- inject different percentages of injected malicious samples
sistencies [58]–[60]. We address this issue by instructing Ψ=(1%, 5%, 10%, 20%, 100%), where Ψ=100% is the
the agent to check and update all the features whenever a baseline as in related literature [61]. For hardening we
manipulation is performed. (For example, an increase to use the adversarial datasets generated with Qmax =1000
the Source bytes determines an increment of the Bytes per which contains the exact amount of samples of the
second). As the magnitude of the modification is small original malicious datasets. We perform the re-training
(at most +2 units), we do not need to update the Duration and subsequent re-testing by following the same ratios
because it is realistic and feasible to transmit more data and splits used for the baseline detectors. After this phase
through packets or bytes in the same timeframe. we have eight hardened detectors for every value of Ψ
At the end of the training process, we obtain four where, for example, the detector RF2DQN is the variant
agents that are denoted by the corresponding reinforce- of RF that is hardened through the samples contained in
ment learning algorithm. For instance, Sarsa(CTU) is the G2DQN
RF .
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 7

D. Implementation of attacks ios is as follows. As the augmented datasets for adversar-

ial training include samples that mimic the corresponding
Solutions leveraging adversarial training are evaluated
attack pattern, we can expect that detectors hardened
against the attack scenarios that resemble the samples
through our framework can achieve better results when
used for producing the augmented dataset [8], [12], [21],
the attack types are in E1, and a manual approach should
[61]. These defense strategies are effective if the attacks
yield better results in the E2 scenario. On the other
can be foreseen that is, by manually crafting samples that
hand, the adversarial samples in E3 represent unforeseen
replicate the predicted evasion attempts. Our proposal
attacks, hence it is of maximum importance that our
aims to overcome a similar limitation and shows that
hardening approach is effective even in this scenario. The
our autonomous approach allows the creation of resilient
experimental evaluation aims to show that the proposed
detectors that are less affected by novel perturbations.
solution is able to mitigate all these attack scenarios.
In order to achieve a comprehensive evaluation, we
consider three gray box evasion attack scenarios that are
compatible with the threat model in Section II and are V. E VALUATION
described below. The experimental campaign has the twofold objective
• E1 represents the attacks performed by our frame- of showing that our framework produces samples that
work. This scenario assumes a powerful attacker are able to evade detectors with high success rate and
that is able to query the detector several times few queries, and that the generated samples can be used
without the risk of triggering other defensive mech- to harden the detectors against evasion attacks without
anisms. We use this scenario to show that our decreasing their performance in non-adversarial settings.
framework is effective even against stronger but less The framework has been implemented in Python3 with
realistic attackers that can see the response of the the scikit-learn, Keras-RL and OpenAI Gym toolkits. For
detector to a malicious flow, by using the machine evaluating the performance in non-adversarial settings
learning model as an oracle [26]. We evaluate the we adopt the usual metrics of machine learning studies:
detectors against attackers that can modify each Precision, Detection Rate (DR, or Recall), and F1-
sample up to five times. In practice, we submit each score [51], [53]; on the other hand, for the attack scenar-
sample in G2DQN (at Qmax = 5) to each hardened ios we consider the Detection Rate of the adversarially
RF
detector in the same network setting. manipulated samples.
• E2 represents an attack strategy that was shown to
effectively evade detection [5]. Here, the detector A. Baseline performance
is evaded by manually modifying combinations of We initially evaluate the baseline detectors RF and
up to four features (duration, source bytes, destina- WnD for the two network scenarios in non-adversarial
tion bytes, total packets) that are altered by fixed contexts. The results in Table IV show that our baseline
amounts. We manually craft adversarial samples detectors achieve values that are comparable to the state
from the original malicious flows that mimic a of the art [5], [36], [51]. RF slightly outperforms WnD as
similar attack pattern. Each flow has the values of anticipated by previous studies (e.g., [5], [51], [54]). We
the considered features that are increased by five observe that the WnD detector in the BOTNET scenario
amounts that is, (+1, +2, +5, +10, +30). Experi- obtains poor Precision in the case of the Virut and Menti
ments show that these perturbations produce adver- families. As these deep learning classifiers are affected
sarial flows that are able to evade the considered by high rates of false positives, signaling that these novel
detectors with high probability. We consider E2 as methods still present margin for improvements. However,
the basis for producing the set of manually crafted these classifiers reach suitable Recall values against these
adversarial flows that are denoted by GM an . botnet families (above 0.95) thus implying that they
• E3 represents attacks that alter the same features are able to detect most malicious samples, and hence
of E2 but with intermediate increments. For ex- represent a valid baseline for the experimental campaign.
ample, if the samples in E2 increase the duration We also present in Figures 5 the ranking of the top-
of the original malicious flows by (1, 2, 5, 10, 30), 5 most significant features for every baseline detector.
then the samples in E3 increase the duration Each detector has minor different rankings, but all of
by (1.5, 3.5, 7.5, 20). The adversarial samples in them have among the most important features those that
E3 represent unforeseen attacks, and the toughest the proposed DRL agent will modify to construct the
testbed for our approach. adversarial samples. The only exception is the destination
The rationale for considering these three attack scenar- port type of the RF(CTU) detector. We also highlight that
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 8

Table IV: Performance of the baseline detectors RF and samples are capable of evading the detection mechanism
WnD in non-adversarial settings. immediately (e.g., over 80% and 50% against RF(CTU)
Detector
Network Family
Random Forest (RF)
F1-score Recall Precision
Wide & Deep (WnD)
F1-score Recall Precision
and WnD(BOTNET), respectively). Moreover, for the
Neris 0.985 0.988 0.983 0.845 0.978 0.743 Sarsa agent, the majority requires few attempts in both
Rbot 0.996 0.993 0.998 0.988 0.983 0.992
Virut 0.987 0.998 0.976 0.934 0.991 0.883 networks. These results confirm that detectors based on
CTU

Menti 0.999 0.998 0.999 0.923 0.960 0.929

Murlo 0.990 0.986 0.994 0.927 0.973 0.878 machine learning are highly vulnerable to adversarial
average 0.991 0.993 0.991 0.935 0.980 0.898
(std. dev.) (0.005) (0.003) (0.008) (0.047) (0.006) (0.082) attacks.
Neris 0.995 0.996 0.994 0.916 0.914 0.919
Rbot 0.999 1.000 0.999 0.986 0.998 0.975 We compare the offensive capabilities of our proposal
BOTNET

Virut 0.993 0.992 0.994 0.358 0.951 0.220

Menti 1.000 1.000 1.000 0.806 0.975 0.686 against other automatic methods for evading ML detec-
Murlo 0.999 0.999 1.000 0.994 0.999 0.989
average 0.998 0.998 0.998 0.902 0.970 0.864 tors [4], [13], [40], [41], [61]–[66]. In Table VII we
(std. dev.) (0.002) (0.001) (0.002) (0.135) (0.034) (0.179)
report the best results for each proposal. Our framework
can leverage agents that achieve a high success rate
these rankings are similar to those obtained by related through few queries, while other approaches generate
researches on botnet detection through supervised ML evasive samples that are either less effective or require
algorithms [34]–[37]. For example, in [34] the Tot bytes many more attempts.
feature is the second most important feature, as in Our proposal is also superior if we limit the compar-
RF(CTU). The detectors in [36] and in [35] focus on ison to the botnet detection scenarios discussed in [4]
the Tot Pkts feature as in WnD(BOTNET). These results that consider only the CTU dataset. Successful evasion
suggest that adversarial attacks aiming to perturb these is reached in 41% of the cases with Qavg =4, while
features are more likely to be effective. our 2DQN agent achieves an evasion rate of 99% with
Qavg =2.4 (Table Va).
The fact that even small and easily achievable alter-
B. Evasion ations affect the most important features of the consid-
We evaluate the offensive capabilities of our proposal ered detection models is a critical issue that demands
by using the trained agents to launch evasion attacks proper countermeasures. The effectiveness of the pro-
against the target detectors. The performance is measured posed attacks is likely due to the high importance that
through the Evasion Rate (ER). The results are reported the modified features have for the baseline detectors.
in Tables V, where the cells show the ER and average Perturbations affecting these features are likely to pro-
amount of queries Qavg (in parentheses) of the samples duce samples that evade detection. A similar observation
generated by our agents with Qmax =80. Gray cells report was in [50]. Other studies on evasion attacks [4] do not
the weighted average across all botnet families of each consider the feature importance of the baseline detectors.
network. From these tables, we observe that the agents
produced by our framework are able to evade the RF
detector with very high probability, and they are also C. Hardened defense
effective against the WnD detector where ER exceeds We now evaluate the effects of adversarial training
90%. We also note that, as anticipated in Section III-A, based on the samples generated by our agents. We report
an agent based on 2DQN requires less attempts to evade the results of the hardened version of RF and WnD that
detection compared to an agent based on Sarsa. are obtained through re-training on datasets including the
We compare the automatic evasion capabilities of the generated malicious samples G2DQN and GSarsa . Let
proposed solution against an attack proposed by the us denote the hardened versions through RF and WnD.
same authors [5], where the adversarial samples were We initially consider non-adversarial settings where Ψ is
generated by manually increasing the same features by set to 100% (see Section IV-C). The results are shown
fixed amounts. Those attacks obtained an average evasion in Tables VIII, which report the average and standard
rate of 35% on the CTU dataset, and of 40% on the deviation of the metrics for each botnet family and
BOTNET dataset, while our evasion rates are above 90% network scenario, and also include the performance of
(Tables V). the baseline detectors in the bottom rows (taken from
In realistic scenarios, attackers cannot arbitrarily query Table IV). By comparing the results of the hardened
the detector or inspect its output without exposing them- versions against those of their corresponding baselines,
selves. For this reason, we consider the amount of we can appreciate that our method does not degrade
generated samples that are able to evade detection on performance in the absence of evasion attacks. This is
the first attempt and in less than five attempts. From an important improvement with respect to the state of
the results in Tables VI, we can observe that many the art.
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 9

RF(CTU) RF(BOTNET) WnD(CTU) WnD(BOTNET)

0.10
Importances

0.05

0.00
c tes pe c t In tes tes Pkt tes c kts tIn c tes ts Dur Sec tes kts
Se y Se Pk ut Se Se Pk
P er otBy rt_T Per sPer y y er
ioO OutB InB esP TotB
y
P er TotP ioOu Per InBy To
t er tBy erP
es T tPo ts e Ra
t kts t
Ra Byte
s tsP Ou tesP
By
t Pk Byt By
t P Pk
Ds By

Figure 5: Top-5 important features for the baseline RF and WnD detectors on each network scenario.

Table V: Attack performance of the 2DQN and Sarsa agents for Qmax = 80.
Network CTU BOTNET
Malware Neris Rbot Virut Menti Murlo average Neris Rbot Virut Menti Murlo average
2DQN 97% (4.72) 99% (1.30) 99% (2.43) 100% (1.59) 100% (2.15) 99% (2.43) 95% (13.93) 99% (11.45) 92% (9.32) 99% (8.68) 99% (14.20) 96% (11.51)
Sarsa 97% (4.00) 99% (1.57) 96% (4.06) 99% (2.43) 100% (2.07) 98% (2.82) 90% (20.16) 95% (10.12) 98% (13.09) 100% (6.99) 92% (10.33) 95% (12.13)
(a) Evasion Rate (and Qavg ) against the baseline RF detectors in both network scenarios.
Network CTU BOTNET
Malware Neris Rbot Virut Menti Murlo average Neris Rbot Virut Menti Murlo average
2DQN 82%(21.27) 96%(11.42) 80%(25.15) 100%(1.07) 100%(23.50) 91%(16.48) 96%(6.52) 100%(1.66) 98%(2.37) 99%(9.78) 100%(4.55) 98%(4.97)
Sarsa 92%(8.07) 95%(12.42) 70%(30.14) 99%(16.21) 100%(30.78) 91%(19.52) 99%(2.50) 99%(1.81) 98%(2.60) 100%(1.47) 72%(25.68) 93%(6.81)
(b) Evasion Rate (and Qavg ) against the baseline WnD detectors in both network scenarios.

Table VI: Attack performance of the 2DQN and Sarsa agents for Qmax = 1 and Qmax = 5.
Network CTU BOTNET
Family Neris Rbot Virut Menti Murlo average Neris Rbot Virut Menti Murlo average
Qmax =1 73.3% 86.2% 93.3% 58.6% 7.9% 80.4% 11.7% 2.1% 43.7% 23.5% 15.1% 15.1%
2DQN
Qmax =5 89.7% 99.7% 96.6% 98.1% 99.3% 93.4% 37.3% 18.7% 72.4% 33.8% 64.9% 40.5%
Qmax =1 78.6% 86.1% 89.9% 31.2% 6.7% 81.2% 10.9% 3.2% 41.4% 9.2% 15.9% 12.3%
Sarsa
Qmax =5 91.8% 98.1% 92.7% 93.8% 100% 93.3% 20.9% 57.8% 57.8% 85.5% 67.7% 57.9%
(a) Evasion Rate against the baseline RF detectors in both network scenarios.
Network CTU BOTNET
Family Neris Rbot Virut Menti Murlo average Neris Rbot Virut Menti Murlo average
Qmax =1 25.9% 2.9% 11.8% 93.4% 12.3% 19.5% 80.8% 86.1% 94.2% 6.7% 43.3% 56.4%
2DQN
Qmax =5 32.8% 13.1% 11.8% 100% 12.3% 25.4% 86.7% 93.4% 97.5% 85.2% 89.2% 90.3%
Qmax =1 85.5% 2.5% 11.1% 8.1% 12.3% 50.2% 90.8% 83.9% 94.6% 90.9% 15.1% 71.5%
Sarsa
Qmax =5 89.7% 12.7% 11.4% 16.3% 15.1% 55.4% 95.5% 98.6% 97.2% 94.4% 64.9% 88.7%
(b) Evasion Rate against the baseline WnD detectors in both network scenarios.

Table VII: Comparison of ER and Qavg of our proposal countering the three considered attack scenarios. We also
with related approaches. compare the effects of training performed on the basis of
Framework ER Qavg the generated samples with samples manually crafted to
[4] 41% 4 replicate the patterns of existing attacks. Moreover, we
[13] 95% 12
[40] 16% 7 carry out a sensitivity analysis by varying the percentages
[41] 46% 7 of adversarial samples (Ψ parameter) introduced in the
[62] 79% > 200
[63] 15% 7 training dataset by evaluating the Detection Rate.
[64] 100% 7
[61] 52% 7 We initially analyze the defensive capabilities against
[65] 100% > 100 the attacks scenario E1. The results are reported in
[66] 100% > 40 000
Ours 97% 9
Tables IX, where cells show the average Detection
Rate for increasing values of Ψ. The cells with a gray
background refer to mechanisms hardened through our
proposal, and numbers in bold denote the results obtained
We now evaluate the efficacy of our proposal at by the best approach for the corresponding value of Ψ.
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 10

Table VIII: Performance of the hardened detectors in Ψ=100%. This means that all malicious samples should
non-adversarial settings. be generated with a considerable effort.
Network
F1-score
CTU
Recall Precision F1-score
BOTNET
Recall Precision
In Table XII we compare the results of the proposed
Metric
(std. dev.) (std. dev.) (std. dev.) (std. dev.) (std. dev.) (std. dev.) approach against those of existing defensive mechanisms
0.988 0.993 0.984 0.995 0.998 0.992
Hardening

2DQN
(0.009) (0.006) (0.017) (0.005) (0.002) (0.008) to evasion attacks. In this table, we report the perfor-
0.989 0.993 0.985 0.996 0.999 0.994
Sarsa
(0.010) (0.005) (0.017) (0.003) (0.001) (0.006) mance of the best classifier as reported by each paper
Baseline
0.991
(0.005)
0.993
(0.003)
0.991
(0.008)
0.998
(0.002)
0.998
(0.001)
0.998
(0.002)
before and after the hardening procedure in adversarial
and non-adversarial scenarios. (The authors in [67] use
(a) Results of the hardened RF detectors. a custom metric denoted as resistance.) We note that
Network
F1-score
CTU
Recall Precision F1-score
BOTNET
Recall Precision
some approaches [13], [61], [68] do not evaluate the
Metric
(std. dev.) (std. dev.) (std. dev.) (std. dev.) (std. dev.) (std. dev.) performance of the hardened detector in non-adversarial
0.928 0.967 0.901 0.896 0.968 0.845
Hardening

2DQN
(0.062) (0.030) (0.112) (0.085) (0.045) (0.129) settings. Addressing this lack is a contribution of this
0.903 0.985 0.848 0.918 0.969 0.883
Sarsa
(0.086) (0.007) (0.149) (0.069) (0.040) (0.118) paper. The method in [69] is used only to enhance the
Baseline
0.935
(0.047)
0.980
(0.006)
0.898
(0.082)
0.902
(0.135)
0.970
(0.034)
0.864
(0.179)
detector but it does not consider adversarial scenarios.
(b) Results of the hardened WnD detectors. Other approaches (denoted with a gray background in
Table XII) are affected by significant performance degra-
dation in the absence of adversarial attacks [29], [67],
[70]–[72]. This problem does not affect our proposal and
The Detection Rate achieved by the baseline detectors is the approach presented in [12]. Nevertheless, its initial
reported in the caption of each table, which corresponds performance in non-adversarial settings is poor (F1-score
to setting Ψ = 0%. From Tables IX, we can appreciate of only 0.69), and the improvement in adversarial sce-
that the detectors hardened through our methods signifi- narios is considerably smaller than the results achieved
cantly improve the capabilities of the baseline detectors. by our method (2% against 30%).
Moreover, they always outperform the results of those
manually trained.
VI. R ELATED WORK
We then test the hardened detectors against E2 that
represents the manually crafted attacks, and report the The literature has demonstrated that even small adver-
results in Tables X. We observe that all proposed ap- sarial perturbations can impact severely the performance
proaches improve the baseline detection rate against of detectors based on machine learning models, but the
existing attacks and that for high values of Ψ the manual solutions are still at an early stage [2], [3], [9], [11], [24].
approach tends to be more effective because these sam- Existing countermeasures conform to either the security-
ples exactly match those used to attack the detectors. by-design or the security-by-obscurity paradigms [8].
Our method, however, requires to inject a significantly Here, we focus on the former group because of the unreli-
smaller amount of samples than the manual approach as ability of security-by-obscurity defensive strategies [73].
shown by the column Ψ=1% in Table Xa for the BOTNET Security-by-design strategies against evasion attacks can
network, and in Table Xb for the CTU network. These be divided into three groups [10]: feature manipulation,
results show that, to be effective, training through human- defensive distillation, and adversarial training.
crafted samples requires not only to predict all the attack Several studies have shown that approaches leveraging
patterns that can be used to evade the detectors, but also altered feature sets may be effective at mitigating [8],
the necessity to craft a significant amount of samples [67] or even nullifying [11] attacks that involve the
with high manual effort. manipulation of the involved features. However, train-
Finally, we show that our framework is also effective ing the model on different sets of features may cause
in protecting the detectors against previously unforeseen significant performance degradation in the absence of
attacks in scenario E3. The results in Tables XI show adversarial attacks [12], [72]. The same drawback also
that in the case of the hardened RF detector in the affects countermeasures based on defensive distillation.
BOTNET network scenario (Table XIa) our proposal As evidenced in [29], [70], these approaches tend to
outperforms the manual approach for the majority of increase the false positive rate. Furthermore, they are
the considered percentages Ψ. The same conclusion is tailored to algorithms based on neural networks that are
valid for the hardened WnD2DQN in the CTU network not the best choice in network intrusion detection [2],
(Table XIb). We can observe that hardening a detector [4], [12], [53], [54].
through manually crafted samples is effective only for Adversarial training aims to harden the detector
a very large number of injected samples in the order of through an augmented dataset containing samples with
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 11

Table IX: Performance against E1 of the hardened detectors for varying Ψ values.
Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.948 0.973 0.991 0.978 0.965 0.845 0.971 0.995 0.984 0.964
Sarsa 0.919 0.954 0.966 0.959 0.953 0.679 0.834 0.965 0.967 0.944
Man 0.523 0.772 0.779 0.958 0.604 0.309 0.761 0.978 0.956 0.807
(a) Average DR of the hardened RF detectors against E1. Baseline DR on E1: RF(CTU)=0.065,
RF(BOTNET)=0.591.
Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.901 0.906 0.908 0.924 0.933 0.786 0.813 0.919 0.921 0.924
Sarsa 0.877 0.908 0.853 0.907 0.911 0.581 0.657 0.921 0.824 0.631
Man 0.718 0.717 0.760 0.877 0.729 0.329 0.812 0.903 0.812 0.830
(b) Average DR of the hardened WnD detectors against E1. Baseline DR on E1: WnD(CTU)=0.456,
WnD(BOTNET)=0.097.

Table X: Performance against E2 of the hardened detectors for varying Ψ values.

Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.681 0.689 0.684 0.692 0.712 0.858 0.886 0.898 0.906 0.914
Sarsa 0.657 0.680 0.689 0.738 0.710 0.848 0.878 0.893 0.901 0.908
Man 0.466 0.589 0.681 0.709 0.789 0.677 0.784 0.821 0.859 0.884
(a) Average DR of the hardened RF detectors against E2. Baseline DR on E2: RF(CTU)=0.327,
RF(BOTNET)=0.679.
Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.748 0.753 0.755 0.756 0.801 0.475 0.570 0.564 0.601 0.761
Sarsa 0.704 0.682 0.694 0.703 0.853 0.531 0.532 0.567 0.582 0.681
Man 0.587 0.656 0.674 0.717 0.937 0.457 0.623 0.672 0.712 0.881
(b) Average DR of the hardened WnD detectors against E2. Baseline DR on E2: WnD(CTU)=0.514,
WnD(BOTNET)=0.413.

Table XI: Performance against E3 of the hardened detectors for varying Ψ values.
Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.717 0.674 0.658 0.671 0.677 0.731 0.766 0.784 0.791 0.851
Sarsa 0.713 0.686 0.669 0.671 0.679 0.714 0.754 0.775 0.788 0.834
Man 0.396 0.545 0.660 0.718 0.765 0.552 0.705 0.756 0.807 0.391
(a) Average DR of the hardened RF detectors against E3. Baseline DR on E3: RF(CTU)=0.253,
RF(BOTNET)=0.526.
Network CTU BOTNET
Ψ 1% 5% 10% 20% 100% 1% 5% 10% 20% 100%
2DQN 0.728 0.738 0.741 0.742 0.805 0.461 0.543 0.544 0.566 0.719
Sarsa 0.664 0.656 0.672 0.676 0.822 0.503 0.504 0.531 0.551 0.648
Man 0.549 0.650 0.671 0.719 0.941 0.422 0.598 0.652 0.698 0.876
(b) Average DR of the hardened WnD detectors against E3. Baseline DR on E3: WnD(CTU)=0.496,
WnD(BOTNET)=0.408.

adversarial perturbations. This approach comes with two To the best of our knowledge, we are the first to evaluate
challenges: obtaining appropriate adversarial samples the impact of adversarial training by varying the amounts
and planning the re-training operations. It may be possi- of injected samples in adversarial and non-adversarial
ble to manually craft samples that reflect realistic attacks, botnet detection scenarios.
but similar methods are time consuming; they can only
protect against predictable attacks that comply to the gen- Reinforcement learning is often associated with adver-
erated samples. Although adversarial training works even sarial machine learning in different contexts and goals.
in non-adversarial contexts, the results in [13] suggest the These mechanisms can be a target of adversarial at-
need of studying the effects of the augmentation process. tacks [74], but also as a means to conceive attacks [4],
[13], [40], [41], [62]–[64], and as a countermeasure to
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 12

Table XII: Performance comparison against evasion at- ative Adversarial Networks (GANs). Such approaches
tacks. require two components: a generator, aiming to gen-
Framework
Non-adversarial settings Adversarial settings erate realistic samples from the original data; and a
Initial Hardened Initial Hardened discriminator, which decides whether the output of the
[4] Acc: 0.99 7 DR: 0.59 7
[13] F1: 0.89 7 DR: 0.56 DR: 0.84
generator appears similar to legitimate traffic [13]. GANs
[41] AUC: 0.96 7 DR: 0.54 7 are similar to DRL approaches, but these latter have
[62] Acc: 0.88 7 DR: 0.31 7 the advantage of controlling more precisely the sample
[63] AUC: 0.96 7 DR: 0.85 7
[64] Acc: 0.99 7 Acc: 0.15 7 generating procedure. In practice, DRL methods allow
[61] AUC: 0.97 7 DR: 0.68 DR: 0.70 to define the detailed action space that is used by the
[29] Acc: 0.98 Acc: 0.94 DR: 0.35 DR: 0.61
[67] Acc: 0.96 Acc: 0.93 Res: 0.36 Res: 0.62
model to create the samples [43]. On the other hand,
[11] F1: 0.96 F1: 0.82 DR: 0.34 DR: 0.61 in GANs the generator network does not consent such
[12] F1: 0.69 F1: 0.71 F1: 0.63 F1: 0.65 a fine-grained control. We consider DRL to be more
[70] Acc: 0.88 Acc: 0.78 DR: 0.02 DR: 0.03
[72] Acc: 0.96 Acc: 0.93 Acc: 0.70 Acc: 0.77 suitable to simulate realistic attacks that involve precise
[68] Acc: 0.99 7 Acc: 0.68 Acc: 0.84 and small modifications because excessive or improper
[69] F1: 68.5 Acc: 70.5 7 7 modifications may trigger detection of defensive mecha-
[71] Acc: 91.9 Acc: 90.8 DR: 0.03 DR: 0.31
Ours F1: 0.99 F1: 0.99 DR: 0.58 DR: 0.88
nisms.

these threats through adversarial training [13], [40], [61]. VII. C ONCLUSIONS
Few papers consider cyber security problems related
to network intrusion detection which is the focus of our Modern detectors based on machine learning classi-
proposal. The authors in [33] and [69], [75] propose fiers are increasingly able at identifying malicious net-
methods to generate flows for training NIDS, but they do work traffic, but they can be exploited by adversarial
not evaluate their performance in adversarial scenarios. samples that allow attackers to evade detection. Existing
The method in [13] based on reinforcement learning solutions are affected by several drawbacks that do
tends to degrade the baseline performance of the detector. not guarantee a reliable defense. We address the issue
The papers in [40], [63], [65] consider reinforcement of evasion attacks against flow-based botnet detectors
learning agents that operate on binary malware detectors, by proposing the first defensive approach that relies
while we operate on network traffic. The proposal in [61] on deep reinforcement learning to mitigate adversarial
focuses on hardening detectors of Domain Generation perturbations against network intrusion detection systems
Algorithms but does not consider the performance in based on machine learning. We consider the character-
non-adversarial scenarios. The approach in [4] does istics of a realistic cybersecurity scenario: small and
not evaluate adversarial training and operates on packet feasible perturbations to the input samples; high degree
captures, while we focus on network flows that nowadays of evasion with a limited number of queries; assessments
are preferred by modern detectors [15]. of several configurations for defensive purposes. The
The primary focus of most proposals related to the implementation of our proposal results in a framework
generation of attack samples is just on the rate of success- that autonomously generates evasive samples against
ful evasions [26], [41], [61], [64] and not on the number a target botnet detector, and then uses these samples
of required queries issued to the target detector. Neglect- for hardening the detector through adversarial training.
ing this characteristic is unrealistic because attackers can An extensive experimental campaign replicating realistic
only perform limited amounts of queries if they want network scenarios of modern organizations shows the
to avoid detection. For example, the methods presented quality of our proposal over state-of-the-art methods for
in [62] and [66] allow their agents to submit hundreds multiple reasons. It increases the detection rate against
or even thousands of queries. Even the proposal in [4], known and novel evasion attacks; it does not degrade
[65] achieves evasion through dozens of attempts against performance in non-adversarial settings; the procedure
the target detector. Unlike these papers, we propose the of malicious sample generation can bypass detection
first framework based on deep reinforcement learning through few queries issued to the detector. Our study may
that hardens existing flow botnet detectors in realistic pave the way to future researches aiming to face evasion
scenarios, even against attackers that are capable of attacks by devising robust detectors that preserve their
issuing some queries to the considered detectors. performance regardless of the presence of adversarial
Other papers on adversarial attacks leverage Gener- perturbations.
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 13

R EFERENCES [22] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov,

G. Giacinto, and F. Roli, “Evasion attacks against machine
[1] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly learning at test time,” in Joint Europ. Conf. Mach. Learn. and
detection and diagnosis from system logs through deep learning,” Knowl. Discov. Databases. Springer, Sept. 2013, pp. 387–402.
in Proc. ACM SIGSAC Conf. Comput. Commun. Secur., 2017, pp. [23] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling
1285–1298. deep neural networks,” IEEE Trans. Evol. Comput., 2019.
[2] A. L. Buczak and E. Guven, “A survey of data mining and [24] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik,
machine learning methods for cyber security intrusion detection,” and A. Swami, “The limitations of deep learning in adversarial
IEEE Commun. Surveys Tuts., vol. 18, no. 2, pp. 1153–1176, settings,” in Proc. IEEE Europ. Symp. Secur. Privacy, Mar. 2016,
2016. pp. 372–387.
[3] N. Papernot, P. McDaniel, A. Sinha, and M. Wellman, “Sok: [25] N. Carlini and D. Wagner, “Audio adversarial examples: Targeted
Security and privacy in machine learning,” in Proc. IEEE Europ. attacks on speech-to-text,” in Proc. IEEE Secur. Privacy Work-
Symp. Secur. Privacy, Apr. 2018, pp. 399–414. shops. IEEE, 2018, pp. 1–7.
[4] D. Wu, B. Fang, J. Wang, Q. Liu, and X. Cui, “Evading [26] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo:
machine learning botnet detection models via deep reinforcement Zeroth order optimization based black-box attacks to deep neural
learning,” in Proc. IEEE Int. Conf. Commun., 2019, pp. 1–6. networks without training substitute models,” in Proc. ACM
[5] G. Apruzzese, M. Colajanni, and M. Marchetti, “Evaluating the Workshop Artif. Intel. Secur., 2017, pp. 15–26.
effectiveness of adversarial attacks against botnet detectors,” in [27] D. Jakubovitz and R. Giryes, “Improving DNN robustness to
Proc. IEEE Int. Symp. Netw. Comput. Appl., Oct. 2019, pp. 1–8. adversarial attacks using jacobian regularization,” in Proc. Europ.
[6] H. Kettani and P. Wainwright, “On the top threats to cyber Conf. Comp. Vision, 2018, pp. 514–529.
systems,” in Proc. IEEE Int. Conf. Inf. Comp. Tech., Mar. 2019,
[28] F. Pierazzi, F. Pendlebury, J. Cortellazzi, and L. Cavallaro, “In-
pp. 175–179.
triguing properties of adversarial ml attacks in the problem space,”
[7] G. Banga, “Why is cybersecurity not a human-scale problem
in IEEE Symp. Secur. Privacy, 2020.
anymore?” Commun. ACM, vol. 63, no. 4, p. 30–34, Mar. 2020.
[29] K. Grosse, N. Papernot, P. Manoharan, M. Backes, and P. Mc-
[8] B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of
Daniel, “Adversarial examples for malware detection,” in Proc.
adversarial machine learning,” Elsevier Pattern Recogn., vol. 84,
Springer Europ. Sympo. Res. Comput. Secur., 2017, pp. 62–79.
pp. 317–331, 2018.
[30] P. Laskov et al., “Practical evasion of a learning-based classifier:
[9] J. Gardiner and S. Nagaraja, “On the security of machine learning
A case study,” in Proc. IEEE Symp. Secur. Privacy, 2014, pp.
in malware C&C detection: A survey,” ACM Comput. Surv.,
197–211.
vol. 49, no. 3, p. 59, 2016.
[10] N. Martins, J. M. Cruz, T. Cruz, and P. H. Abreu, “Adversarial [31] G. Apruzzese and M. Colajanni, “Evading botnet detectors based
machine learning applied to intrusion and malware scenarios: a on flows and random forest with adversarial samples,” in Proc.
systematic review,” IEEE Access, 2020. IEEE Int. Symp. Netw. Comput. Appl., Oct. 2018, pp. 1–8.
[11] G. Apruzzese, M. Colajanni, L. Ferretti, and M. Marchetti, [32] D. J. Miller, Z. Xiang, and G. Kesidis, “Adversarial learning
“Addressing adversarial attacks against security systems based targeting deep neural network classification: A comprehensive
on machine learning,” in Proc. IEEE Int. Conf. Cyber Conflicts, review of defenses against attacks,” Proc. IEEE, vol. 108, pp.
May 2019, pp. 1–18. 402–433, 2020.
[12] S. Calzavara, C. Lucchese, and G. Tolomei, “Adversarial training [33] M. Ring, D. Schlör, D. Landes, and A. Hotho, “Flow-based
of gradient-boosted decision trees,” in Proc. ACM Int. Conf. Inf. network traffic generation using generative adversarial networks,”
Knowledge Manag, 2019, pp. 2429–2432. Computers & Security, vol. 82, pp. 156–172, 2019.
[13] M. Usama, M. Asim, S. Latif, J. Qadir et al., “Generative adver- [34] Z. M. Algelal, E. A. Ghanialdhaher, D. N. Abdul-Wadood et al.,
sarial networks for launching and thwarting adversarial attacks “Botnet detection using ensemble classifiers of network flow,”
on network intrusion detection systems,” in Proc. Int. IEEE Conf. IAES Int. J. Electr. Comput. Eng., vol. 10, no. 3, p. 2543, 2020.
Wireless Commun. Mobile Comput., 2019, pp. 78–83. [35] I. Letteri, G. Della Penna, and P. Caianiello, “Feature selection
[14] “Checkpoint 2020 security report,” https://pages.checkpoint.com/ strategies for http botnet traffic detection,” in Proc. IEEE Europ.
cyber-security-report-2020.html, Accessed in March 2020. Symp. Secur. Priv., 2019, pp. 202–210.
[15] M. F. Umer, M. Sher, and Y. Bi, “Flow-based intrusion detection: [36] B. Abraham, A. Mandya, R. Bapat, F. Alali, D. E. Brown,
Techniques and challenges,” Elsevier Computers & Security, and M. Veeraraghavan, “A comparison of machine learning
vol. 70, pp. 238–254, 2017. approaches to detect botnet traffic,” in Proc. IEEE Int. Conf. Neur.
[16] A. Pektaş and T. Acarman, “Deep learning to detect botnet via Netw., 2018, pp. 1–8.
network flow summaries,” Springer Neural Comput. Appl., pp. [37] M. Alauthaman, N. Aslam, L. Zhang, R. Alasem, and M. A.
1–13, 2018. Hossain, “A p2p botnet detection scheme based on decision
[17] M. Stevanovic and J. M. Pedersen, “An efficient flow-based botnet tree and adaptive multilayer neural networks,” Springer Neural
detection using supervised machine learning,” in Proc. IEEE Int. Computing and Applications, vol. 29, no. 11, pp. 991–1004, 2018.
Conf. Comput., Netw. and Commun., Feb. 2014, pp. 797–801. [38] C. Xiang, F. Binxing, Y. Lihua, L. Xiaoyi, and Z. Tianning,
[18] S. Nõmm and H. Bahşi, “Unsupervised anomaly based botnet “Andbot: towards advanced mobile botnets,” in Proc. USENIX
detection in iot networks,” in Proc. IEEE Int. Conf. Machin. Conf. Large-scale Exploits and Emergent Threats, 2011, pp. 11–
Learn. Appl., 2018, pp. 1048–1053. 11.
[19] S. Lagraa, J. François, A. Lahmadi, M. Miner, C. Hammer- [39] S. Cesare, Y. Xiang, and W. Zhou, “Malwise—an effective
schmidt, and R. State, “Botgm: Unsupervised graph mining to and efficient classification system for packed and polymorphic
detect botnets in traffic flows,” in Proc. IEEE Conf. Cyber Secur. malware,” IEEE T. Comput., vol. 62, no. 6, pp. 1193–1206, 2012.
Netw., 2017, pp. 1–8. [40] H. S. Anderson, A. Kharkar, B. Filar, D. Evans, and P. Roth,
[20] Z. Qiu, D. J. Miller, and G. Kesidis, “Flow based botnet detection “Learning to evade static pe machine learning malware models
through semi-supervised active learning,” in Proc. IEEE Int. Conf. via reinforcement learning,” arXiv:1801.08917, 2018.
Acoustics, Speech, Sign. Process., 2017, pp. 2387–2391. [41] Z. Fang, J. Wang, B. Li, S. Wu, Y. Zhou, and H. Huang, “Evading
[21] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, anti-malware engines with deep reinforcement learning,” IEEE
A. Oprea, C. Nita-Rotaru, and F. Roli, “Why do adversarial at- Access, vol. 7, pp. 48 867–48 879, 2019.
tacks transfer? Explaining transferability of evasion and poisoning [42] T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for
attacks,” in Proc. USENIX Secur. Symp.), 2019, pp. 321–338. cyber security,” arXiv:1906.05799, 2019.
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 14

[43] K. Malialis and D. Kudenko, “Distributed response to network Transactions on Information and Systems, vol. 103, no. 2, pp.
intrusions using multiagent reinforcement learning,” Elsevier Eng. 212–221, 2020.
Appl. Artif. Int., vol. 41, pp. 270–284, 2015. [63] H. S. Anderson, A. Kharkar, B. Filar, and P. Roth, “Evading
[44] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. machine learning malware detection,” Black Hat, 2017.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Os- [64] J. Zhang, Q. Yan, and M. Wang, “Evasion attacks based on
trovski et al., “Human-level control through deep reinforcement wasserstein generative adversarial network,” in Proc. IEEE Conf.
learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015. Comput., Commun. and IoT Appl. IEEE, 2019, pp. 454–459.
[45] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, [65] H. Dang, Y. Huang, and E.-C. Chang, “Evading classifiers by
D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep morphing in the dark,” in Proc. ACM SIGSAC Conf. Comput.
reinforcement learning,” in Int. Conf. Machin. Learn., 2016, pp. Commun. Secur., 2017, pp. 119–133.
1928–1937. [66] Y. Ren, Q. Zhou, Z. Wang, T. Wu, G. Wu, and K.-K. R. Choo,
[46] H. v. Hasselt, A. Guez, and D. Silver, “Deep reinforcement “Query-efficient label-only attacks against black-box machine
learning with double Q-learning,” in Proc. AAAI Conf. Artif. Int., learning models,” Elsevier Computers & Security, vol. 90, p.
2016, pp. 2094–2100. 101698, 2020.
[47] T. Alfakih, M. M. Hassan, A. Gumaei, C. Savaglio, and [67] Q. Wang, W. Guo, K. Zhang, A. G. Ororbia, X. Xing, X. Liu,
G. Fortino, “Task offloading and resource allocation for mobile and C. L. Giles, “Adversary resistant deep neural networks with
edge computing by deep reinforcement learning based on sarsa,” an application to malware detection,” in Proc. ACM SIGKDD Int.
IEEE Access, vol. 8, pp. 54 074–54 084, 2020. Conf. Knowl. Discov. Data Mining, 2017, pp. 1145–1153.
[48] H. Jiang, R. Gui, Z. Chen, L. Wu, J. Dang, and J. Zhou, “An [68] Y. Ji, B. Bowman, and H. H. Huang, “Securing malware cognitive
improved sarsa reinforcement learning algorithm for wireless systems against adversarial attacks,” in Proc. IEEE Int. Conf.
communication systems,” IEEE Access, vol. 7, pp. 115 418– Cognitive Comput., 2019, pp. 1–9.
115 427, 2019. [69] C. Yin, Y. Zhu, S. Liu, J. Fei, and H. Zhang, “An enhancing
[49] S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical framework for botnet detection using generative adversarial net-
comparison of botnet detection methods,” Elsevier Comput. Se- works,” in Proc. IEEE Int. Conf. Artif. Int. and Big Data, 2018,
cur., vol. 45, pp. 100–123, 2014. pp. 228–234.
[50] E. B. Beigi, H. H. Jazi, N. Stakhanova, and A. A. Ghorbani, [70] M. Soll, T. Hinz, S. Magg, and S. Wermter, “Evaluating defensive
“Towards effective feature selection in machine learning-based distillation for defending text processing neural networks against
botnet detection approaches,” in Proc. IEEE Conf. Comm. Netw. adversarial examples,” in Proc. Springer Int. Conf. Artif. Neur.
Secur., 10 2014. Netw., 2019, pp. 685–696.
[51] M. Stevanovic and J. M. Pedersen, “An analysis of network traffic [71] A. Al-Dujaili, A. Huang, E. Hemberg, and U.-M. O’Reilly,
classification for botnet detection,” in Proc. IEEE Int. Conf. Cyber “Adversarial deep learning for robust detection of binary encoded
Situat. Awar., Data Analyt., Assessment, Jun. 2015, pp. 1–8. malware,” in Proc. IEEE Secur. Privacy Workshops, 2018, pp.
76–82.
[52] B. Biggio, I. Corona, Z.-M. He, P. P. Chan, G. Giacinto, D. S.
[72] S. Calzavara, C. Lucchese, F. Marcuzzi, and S. Orlando, “Feature
Yeung, and F. Roli, “One-and-a-half-class multiple classifier
partitioning for robust tree ensembles and their certification in
systems for secure learning against evasion attacks at test time,”
adversarial scenarios,” arXiv:2004.03295, 2020.
in Proc. Springer Int. Workshop Multiple Classifier Syst., 2015,
[73] D. Pavlovic, “Gaming security by obscurity,” in Proc. ACM New
pp. 168–180.
Secur. Paradigms Workshop. ACM, 2011, pp. 125–140.
[53] G. Apruzzese, M. Colajanni, L. Ferretti, A. Guido, and
[74] V. Behzadan and A. Munir, “Vulnerability of deep reinforcement
M. Marchetti, “On the effectiveness of machine and deep learning
learning to policy induction attacks,” in Proc. Springer Int. Conf.
for cybersecurity,” in Proc. IEEE Int. Conf. Cyber Conflicts, May
Machin. Learn. Data Mining Pattern Recogn., 2017, pp. 262–275.
2018, pp. 371–390.
[75] C. Yin, Y. Zhu, S. Liu, J. Fei, and H. Zhang, “Enhancing net-
[54] O. Fajana, G. Owenson, and M. Cocea, “Torbot stalker: Detecting work intrusion detection classifiers using supervised adversarial
tor botnets through intelligent circuit data analysis,” in Proc. IEEE training,” Springer Journal of Supercomputing, pp. 1–30, 2019.
Int. Symp. Netw. Comput. Appl., Oct. 2018, pp. 1–8.
[55] A. Karasaridis, B. Rexroad, D. A. Hoeflin et al., “Wide-scale
botnet detection and characterization.” HotBots, vol. 7, pp. 7–7,
2007.
[56] Z. Li, Z. Qin, and P. Shen, “Intrusion detection via wide and deep
model,” in Proc. Springer Int. Conf. Artif. Neural Netw., 2019,
pp. 717–730.
[57] M. Almseidin, M. Alzubi, S. Kovacs, and M. Alkasassbeh, “Eval-
uation of machine learning algorithms for intrusion detection
system,” in Proc. IEEE Int. Symp. Intel. Syst. Inf., 2017, pp. 277–
282.
[58] D. Han, Z. Wang, Y. Zhong, W. Chen, J. Yang, S. Lu, X. Shi, and
X. Yin, “Practical traffic-space adversarial attacks on learning-
based nidss,” arXiv preprint arXiv:2005.07519, 2020.
[59] A. Chernikova and A. Oprea, “Fence: Feasible evasion attacks
on neural networks in constrained environments,” arXiv preprint
arXiv:1909.10480, 2020.
[60] S. Sen, E. Aydogan, and I. A. Aysan, “Coevolution of mobile
malware and anti-malware,” IEEE Transactions on Information
Forensics and Security, vol. 13, no. 10, pp. 2563–2574, 2018.
[61] H. S. Anderson, J. Woodbridge, and B. Filar, “Deepdga:
Adversarially-tuned domain generation and detection,” in Proc.
ACM Workshop Artif. Intell. Secur., Oct. 2016, pp. 13–21.
[62] Y. Senzaki, S. Ohata, and K. Matsuura, “Simple black-box
adversarial examples generation with very few queries,” IEICE
 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2020 15

Giovanni Apruzzese is a Post-Doctoral re- Andrea Venturi is a PhD student at the

searcher within the Hilti Chair of Data and Department of Engineering “Enzo Ferrari”,
Application Security at the University of University of Modena and Reggio Emilia,
Liechtenstein since 2020. He received the Italy. From the same institution, he received
PhD Degree and the Master’s Degree in the Master’s Degree in Computer Engineering
Computer Engineering (summa cum laude) in summa cum laude in 2020 with a thesis on
2020 and 2016 respectively at the Department cybersecurity analytics, and the Bachelor’s
of Engineering “Enzo Ferrari”, University of Degree in Computer Science in 2017, with
Modena and Reggio Emilia, Italy. In 2019 a thesis on data analysis for scalable and
he spent 6 months as a Visiting Researcher distributed networked systems. His research
at Dartmouth College (Hanover, NH, USA) interests are on machine and deep learning
under the supervision of Prof. VS Subrahmanian. His research interests applications for cybersecurity.
involve all aspects of big data security analytics with a focus on Michele Colajanni is Full Professor in
machine learning, and his main expertise lies in the analysis of Network computer engineering at the University of
Intrusions, Phishing, and Adversarial Attacks. Bologna. He received the Master degree from
Mauro Andreolini is currently an Assis- the University of Pisa, and the Ph.D. de-
tant Professor at the Department of Physics, gree from the University of Rome. He was
Computer Science and Mathematics of the researcher at the University of Rome, and
University of Modena and Reggio Emilia, visiting researcher at the IBM Research Cen-
Italy. He received his Master Degree (summa ter, Yorktown Heights in 1996. From 1998
cum laude) at the University of Roma, Tor to 2020, he was with the Department of
Vergata in January, 2001 and his PhD in May, Engineering “nzo Ferrari” at the University
2005 from the same institution. His research of Modena and Reggio Emilia. He founded
focuses on design, evaluation and security of the Interdepartment Research Center on Security and Safety (CRIS),
distributed and cloud-based systems, malware and the Cyber Academy on cybersecurity training. His research inter-
analysis and secure software design. ests include cybersecurity, performance and prediction models, cloud
systems.
Mirco Marchetti received the Ph.D. degree
in Information and Communication Technolo-
gies in 2009. He is currently an Associate
Professor with the Department of Engineering
“Enzo Ferrari”, University of Modena and
Reggio Emilia, Italy. His research interests
include all aspects of system and network
security, security for cyber physical systems,
automotive security, cryptography applied to
cloud security, and outsourced data and ser-
vices.

View publication stats