2023 10th International Conference on Dependable Systems and Their Applications (DSA)

A
Software
Security
Testing
Model
for
Autonomous
Systems
2023 10th International Conference on Dependable Systems and Their Applications (DSA) | 979-8-3503-0477-0/23/$31.00 ©2023 IEEE | DOI: 10.1109/DSA59317.2023.00019

‡

JinDhang
Hu1,
Kang
Sun2, ,
Sen
Yang3,∗,
Zhanwei
Hui3

and
Song
Huang1,∗
1
College
of
Command
and
Control
Engineering,
Army
Engineering
University
of
PLA,
Nanjing,
China

2
AVIC
The
First
Aircraft
Institute,
Xi’an,
China
3
Academy
of
Military
Science,
China
[email protected],
[email protected]
‡ co-author,
*corresponding
author

Abstract—The autonomous system is an extremely complex through the formal proof method, the security compiler is
and multidisciplinary system, and the overall goal of this paper constructed, which can ensure that the target memory of the
is to study safety testing methods for autonomous systems compiled program conforms to the semantic specifications
to enhance the robustness of autonomous systems. In order of the safe programming language, and can automatically
to realize an autonomous system with high reliability and extract the formal model from the source code using the
robustness, this paper summarizes the security defect detection target language, so as to verify the functional correctness and
methods for autonomous systems from the two dimensions security of the system; (2) For autonomous systems written
of static analysis and dynamic testing and expounds on the in multiple programming languages, by studying program
syntax-level, semantic, and system-level testing methods for analysis methods at the binary code level, it can automatically
autonomous systems. detect software and firmware security defects in the system
and automatically repair the defects found; (3) In view of the
Keywords–autonomous systems; security testing; dynamic test- possible security hazards during the operation of the system,
ing; static analysis by monitoring the behavior of the system, improve the security
ability of the system operation, avoid potential network attacks
1. I NTRODUCTION
and other security defects, and at the same time, evaluate the
An autonomous system is extremely complex and multidis- speed of the dynamic response of the system by monitoring
ciplinary, usually composed of multiple components such as the perception ability of the system under continuous dynamic
hardware platforms, control software, communication com- task conditions, and monitor the abnormalities that may occur
ponents, and sensors. Autonomous systems are oriented to during the operation of the system.
highly complex environments, which need to detect the en-
vironment of the system in real-time, give the internal and 2. AUTONOMOUS SYSTEMS
external state analysis of the system, respond to decisions in As modern systems, especially security or mission-critical
time, and finally execute decision-making actions. However, systems, operate in highly dynamic and uncertain environ-
in actual use, autonomous systems are often threatened by ments becoming more complex, distributed, and ubiquitous,
various security attacks, such as the injection of malware or security threats, attacks, and vulnerabilities[1] are on the rise,
backdoor software into the firmware of the actuator to cause causing massive damage. At the same time, even under ad-
execution failures, forged navigation commands by hacking verse conditions, modern systems increasingly need to provide
into autonomous systems, causing system denial of service continuous services, such as autonomous vehicles such as
failures by interfering with communication components, and land autonomous driving systems, or drones [2][3]. Therefore,
falsify perception data by spoofing GPS sensors. researching and developing new methods and tools to build
The overall goal of this paper is to study safety testing trusted and robust systems that can withstand runtime (even
methods for autonomous systems to enhance the robustness previously unforeseen) security attacks, vulnerabilities, and
of autonomous systems. Figure 1 shows the basic architec- failures is a very urgent and important task.
ture of this method, the test object of this method is the In order to achieve high reliability and robustness of au-
existing autonomous system software developed based on the tonomous systems, the following two challenges need to be
unmanned polymorphic operating system, and the test strategy addressed: First, traditional static security methods [1] tend to
is divided into two dimensions: static analysis and dynamic deal only with specific security issues in a reactive manner,
test. Firstly, aiming at the four key links of the operation requiring service stops and manual changes. They are often too
of the autonomous system under test, this paper specifically slow and costly, making them not practical for safety/mission-
expounds on the security threats that need to be paid attention critical systems. For example, any unforeseen attack on a
to when detecting, analyzing, decision-making, and executing self-driving car is sure to lead to malfunctions or even mal-
the development of the four links in the autonomous system. functions. If the operational failure of self-driving cars is
Next, this paper proposes security defect detection methods not properly predicted and prevented, it can lead to serious
from two dimensions: (1) for the compiler used in the compi- car accidents that lead to personal injury and even death.
lation of the core code section of the autonomous system, Therefore, failure to address such safety issues could hinder

2767-6684/23/$31.00 ©2023 IEEE 70

DOI 10.1109/DSA59317.2023.00019
Authorized licensed use limited to: ISRO - Vikram Sarabhai Space Centre. Downloaded on January 04,2024 at 11:11:47 UTC from IEEE Xplore. Restrictions apply.
 A Software Security Testing Model

Dynamic
y analysis
y of security
y defects

Intelligent navigation system Behavioral safety monitoring

System level

monitoring
Intelligent control system Dynamic response monitoring

Mission
Intelligent decision-making system Simulated attack testing

Ⓠ⸷㓝
... ...

Static analysis of security defects

Semantic
level

Security defect detection Security bug fixes

Syntax
level

Compiler security validation

Autonomous equipment (Unmanned Aerial Vehicle, unmanned vehicles, etc.)

The system under test

Autonomous systems

Monitor Analyse Plan Execute

Sensor Predictive Detection Efficient Verification Actuator

Function
API Behaviour Analysis Trade-off Decisions Degradation

CPU & Memory Resource Analysis Real-Time Planning Security Alarm

Extended Robot Operating System

Figure 1. A software security testing model for autonomous systems

the practical adoption of autonomous vehicle technology [4]. operating environment to collect runtime data, analyzing data
In order to meet the first challenge, some scholars have to assess or detect security threats, attacks, and vulnerabilities,
repeatedly tried to use the self-protection function, which is planning adaptive solutions to mitigate detected security issues
one of the basic self-protection functions of autonomous com- and executing planned solutions to recover from security issues
puting systems [5], by adjusting the structure or behavior of in the system. These four steps form a MAPE (Detect, Ana-
the system, automatically resisting malicious attacks without lyze, Decide, and Act) feedback control loop to continuously
stopping the service [6]. However, most of them only offer adjust safety-critical systems, such as autonomous systems, to
application-specific solutions and do not target autonomous meet the requirements of continuous operation. However, a
vehicle platforms, specifically robotic OS platforms. Basically, common infrastructure for building and self-protecting such
a self-protection system requires monitoring the system and its systems has not yet been developed.

71

Authorized licensed use limited to: ISRO - Vikram Sarabhai Space Centre. Downloaded on January 04,2024 at 11:11:47 UTC from IEEE Xplore. Restrictions apply.
 Second, a great deal of research is currently focused on memory safety for concurrent programs. However, errors in the
building credible (especially using formal methods) or resilient specification of the programming language or in the compiler
systems. However, achieving these two goals in a system is implementation can make the program unsafe. Formal methods
very difficult, resilient systems need to dynamically adapt to have been successfully used to verify the functional and
protect the system, and formal analysis is mostly done offline security correctness of applications and operating systems. In
due to complexity and poor scalability. To address the second this context, our goal is to validate these proven memory-safe
challenge, effective and lightweight validation methods need to languages (e.g., Rust) using formal methods by constructing
be implemented at runtime to support the platform’s ongoing formal models of the language and verifying the security of
assurance. language constructors.
In fact, security validation of language constructors (e.g.,
3. A S OFTWARE S ECURITY T ESTING M ODEL
memory) means that any program that uses these constructors
3.1. Static analysis of security defects is also secure, i.e. security is provided through constructs.
Static detection of security flaws is critical to protecting This speeds up security verification and greatly eases the
software systems from potential threats and enemy attacks. entire system validation process. As far as we know, there is
Firstly, from the syntax level, the correctness of the compiler currently no work using formal methods in memory security
in the autonomous system is formally verified to ensure the verification in any programming language. In the field of
security defects at the syntax level of the system under test. automated extraction of formal models from source code, it
Second, because autonomous systems are often composed of is worth highlighting the C-parser and autocorres[8] tools for
multi-type subsystems, there may be a variety of programming obtaining formal models in Isabelle/HOL from C programs.
languages, such as the underlying detection and execution In addition, in the field of machine-proof systems for parallel
hardware is generally written in C, the upper-level analysis programs, there are Rely-Guarantee proof systems that utilize
and decision-making software is written in C++ or Java, and the Isabelle/HOL theorem prover for machine reasoning [9].
the system is written in Python and other languages, and However, these works prove that the programming language
eventually, there may be multi-type programming languages of the program is too simple to express most of the features
in the system. By directly analyzing the binary code in the found in common programming languages, and it lacks any
executable program generated by the system compilation, the automation to efficiently handle large concurrent programs.
differences in the multilingual coding of the system can be B.Semantic-level analysis
effectively shielded and the security defects at the semantic Security defect detection for binary code is more difficult than
level of the system under test can be efficiently detected. source code detection because analyzing binary code does
A.Syntax-level analysis not accurately see the data types, data structures, function
Ensuring security by building secure system specifications is boundaries, and so on in the program. Even with a well-
the foundation of any system security. However, a correct and known vulnerability, it can be difficult to find where the
secure specification does not guarantee that its implementation vulnerability lies. Also, due to compiler optimizations, the
is secure. Typically, security specifications are translated into compilation process may make two identical programs (at the
unsafe programs, where programming errors are often caused source level) look completely different at the binary code level.
by incorrect manipulation of dynamic memory. In addition, In this approach, our goal is to analyze C/C++ binaries for
the complexity of autonomous systems, where multiple appli- various platforms (such as Windows, Linux, and Android) and
cations can run in parallel, reduces the occurrence of these architectures (such as x86-32, x86-64, and ARM), and we aim
failures and makes them more difficult to detect. In pursuit to build a complete binary vulnerability analysis framework.
of performance, systems programming languages like C or We expect to combine multiple technologies (e.g., program
C++ use very loose memory semantics in which unsafe code analysis, machine learning, and fuzziness) to achieve greater
implementations in complex systems often occur. A new trend vulnerability detection accuracy in a scalable way.
in systems programming languages is to pay more attention The goal of this topic is to build a binary vulnerability
to program security without breaking existing performance. discovery framework based on the semantic mining of core
These programming languages have strict memory operations, modules, which consists of four key modules: semantic patch
and even if the program is incorrect, it is safe, so there are analysis, vulnerability extrapolation, cloning-based decompi-
no security flaws caused by unsafe memory operations. For lation and targeted intelligent fuzzing. The Semantic Patch
example, the Rust language [7] (the first version released Analysis module identifies high-value security patches that are
in May 2015) uses concepts such as ownership, borrowing, easy to collect. The vulnerability extrapolation module finds
lifecycle, and mutability to ensure the security of the system vulnerabilities in the target program (identified from patch
through static analysis performed by the compiler. In addition, analysis), the cloning-based decompilation module tries to
Rust provides mechanisms to prevent memory leaks, and while speculate the ”potential” vulnerability code segment (identified
memory leaks are not considered unsafe operations, it can from vulnerability reasoning) to the source code, and finally,
certainly affect the performance of the system when memory the directed intelligent fuzzing module uses the ”potential”
is not properly freed. In addition, Rust provides primitives vulnerability code as the target, and extracts the type informa-
for implementing concurrent programs, which also guarantees tion from the source code through decompilation to confirm

72

Authorized licensed use limited to: ISRO - Vikram Sarabhai Space Centre. Downloaded on January 04,2024 at 11:11:47 UTC from IEEE Xplore. Restrictions apply.
 Figure 2. A software security testing model

the vulnerability by directed intelligent fuzzing. Finally, we The system-level security defect test model is a cyclic eval-
try to summarize and maintain as many known vulnerabilities uation and monitoring process, the system continuously re-
as possible in a vulnerability database, where vulnerabilities ceives data from each subsystem of the system under test,
are summarized using an adaptive vulnerability model that evaluates the safety status of the automatic unmanned system,
captures the underlying semantics of the vulnerability. This determines the best response, and notifies the corresponding
allows us to generalize vulnerability signatures for each type subsystem to execute the determined response. The system
of vulnerability. under test in this process involves multiple subsystems of
autonomous systems, such as sensors, control and navigation
3.2. Dynamic analysis of security defects systems, and hardware-to-hardware communication systems.
A.System-level navigation and control system safety analysis The entire process consists of three sub-processes, namely
In response to various attacks, the autonomous systems in- System State Assessment, Security State Assessment, and
volved and their components can be divided into two groups. Dynamic Response Assessment. All of these processes use a
One group is reliable and the other is unreliable. The former unified mission-aware system and security model. The unified
includes those encrypted codes, some communication chan- model contains a set of well-coordinated graphical models.
nels, and some interfaces. The latter includes open-source C.System-level security attack testing
code, frequent communication channels, and various inter- Models based on deep neural networks have been widely
faces. In the navigation and control of autonomous driving used, including image processing, natural language processing,
systems, logical-level, and physical-level control has been software engineering, etc., and have played an important role
extensively studied by several methods that assume that all in various scenarios such as assisted decision-making and
autonomous vehicles and their components are reliable. How- intelligent recommendation. However, deep neural networks
ever, this is not the case, and these components can fail are found to have characteristics such as vulnerability, and
because they can be attacked via source code, external com- their decision-making process is fuzzy and unexplainable,
munication channels, and certain interfaces. A well-designed although a variety of deep learning-oriented interpretability
autonomous system should be able to withstand such attacks, systems have a visual analysis of the decision-making process,
meaning that such failed components have minimal harmful this interpretability has large flaws and cannot avoid the
effects on the system. Therefore, it requires that the failure security attacks naturally present in deep neural networks.
of one component can only affect a certain operation of the Common cyberattacks include brutal attacks, privacy attacks,
autonomous vehicle, and the operation of the autonomous fairness attacks, and backdoor attacks. These network attacks
vehicle is strongly dependent on the faulty part. will be extremely destructive to intelligent models and have
B.System-level continuous mission-aware security assessment strong concealment, so system-level intelligent model security

73

Authorized licensed use limited to: ISRO - Vikram Sarabhai Space Centre. Downloaded on January 04,2024 at 11:11:47 UTC from IEEE Xplore. Restrictions apply.
 attacks must adopt a variety of methods to conduct timely
security assessments.
4. C ONCLUSION
In the face of extremely complex and multidisciplinary un-
manned intelligent systems, this paper summarizes the security
defect detection methods for unmanned intelligent systems
from the two dimensions of static analysis and dynamic
testing, puts forward syntax-level, semantic-level, and system-
level testing methods, and gives solutions to ensure the security
of unmanned intelligent systems from the perspective of test-
ing. The novelty of this approach lies in a system-level solution
to address security issues in unmanned intelligent systems: by
building syntax-level security assurances, semantic-level vul-
nerability detection and remediation, system-level task-aware
security assessment, dynamic response, and secure resilient
control and navigation systems.
R EFERENCES
[1] G. Meng, Y. Liu, J. Zhang, A. Pokluda, and R. Boutaba.
Collaborative Security: A Survey and Taxonomy, ACM
Computing Surveys, 48(1), 2015.
[2] J. M. Anderson, K. Nidhi, K. D. Stanley, P. Sorensen,
C. Samaras, and O. A. Oluwatola. Autonomous Vehicle
Technology: A Guide for Policymakers. Rand Corpora-
tion, 2014.
[3] A. Cavoukian. Privacy and Drones: Unmanned Aerial Ve-
hicles. Information and Privacy Commissioner of Ontario,
Canada, 2012.
[4] E. Schoitsch, C. Schmittner, Z. Ma, and T. Gruber. The
Need for Safety and Cyber-Security Co-engineering and
Standardization for Highly Automated Automotive Vehi-
cles. In Advanced Microsystems for Automotive Applica-
tions, pp. 251-261, 2015.
[5] J. O. Kephart and D. M. Chess. The Vision of Autonomic
Computing. Computer, 36(1): 41-50, 2003.
[6] E.Yuan, N. Esfahani, and S. Malek. A Systematic Survey
of Self-Protecting Software Systems. ACM Transactions
on Autonomous and Adaptive Systems, 8(4): 1-41, 2014.
[7] Rust, https://www.rust-lang.org/.
[8] D. Greenaway, J. Andronick, and G. Klein. Bridging the
gap: Automatic verified abstraction of C. In Interactive
Theorem Proving. Springer, pp. 99-115, 2012.
[9] L. P. Nieto. The Rely-Guarantee method in Isabelle/HOL.
In Proceedings of the 12th European Conference on Pro-
gramming, pp. 348-362, 2003.

74

Authorized licensed use limited to: ISRO - Vikram Sarabhai Space Centre. Downloaded on January 04,2024 at 11:11:47 UTC from IEEE Xplore. Restrictions apply.