IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO.

5, MAY 2024 4861

Choose Your Simulator Wisely: A Review on

Open-Source Simulators for Autonomous Driving
Yueyuan Li , Wei Yuan , Songan Zhang , Weihao Yan , Qiyuan Shen , Chunxiang Wang , Member, IEEE,
and Ming Yang , Member, IEEE

Abstract—Simulators play a crucial role in autonomous driving, tested in simulation environments. This trend is particularly pro-
offering significant time, cost, and labor savings. Over the past nounced in driving decision-making tasks, where over 70% of
few years, the number of simulators for autonomous driving has publications underwent simulator-based verification. Compared
grown substantially. However, there is a growing concern about
the validity of algorithms developed and evaluated in simulators, to real-world scenarios, simulators exhibit the unique capability
indicating a need for a thorough analysis of the development status to swiftly synthesize sensor data on a large scale, replicating
of the simulators. To address existing gaps in research, this paper extreme weather conditions, unpredictable behaviors of radical
undertakes a comprehensive review of the history of simulators, drivers, and intricate traffic scenarios. Moreover, simulators
proposes a utility-based taxonomy, and investigates the critical offer the unique advantage of replaying traffic accidents without
issues within open-source simulators. Analysis of the past thirty
years’ development trajectory reveals a trend characterized by posing safety hazards. Kalra et al. highlighted the impracticality
an increase in open-source simulators and an expansion of their of achieving statistical significance in autonomous vehicle per-
functionality scope. The categorization of simulators based on formance validation through real-world testing alone, estimating
feature functionalities delineates five primary classes: traffic flow, a staggering 215 billion miles of testing would be required [1].
sensory data, driving policy, vehicle dynamics, and comprehensive Addressing this challenge, Shuo et al. presented compelling
simulators. Furthermore, the paper identifies critical unresolved
issues in open-source simulators, including concerns regarding evidence that introducing proper simulation methodologies can
the fidelity of sensory data, representation of traffic scenarios, accelerate the evaluation process 103 to 105 times faster [2].
and accuracy in vehicle dynamics simulation, all of which have Leading technology companies in the autonomous driving in-
the potential to undermine experimental confidence. Additionally, dustry, including Waymo, Pony.ai, and Didi, emphasize the
challenges in data format inconsistency, labor-intensive map con- development and application of simulators [3], [4], [5]. Waymo’s
struction processes, sluggish step updating, and insufficient sup-
port for Hardware-In-the-Loop testing are discussed as hindrances remarkable performance, reflected in the Disengagement Report
to experimental efficiency. In light of these findings, the survey from California between 2020 and 2023 [6], serves as com-
furnishes task-oriented recommendations to aid in the selection pelling evidence of the critical role that realistic simulators play
of simulators, taking into account factors such as accessibility, in achieving outstanding results [3].
maintenance status, and quality, while highlighting the inherent Open-source simulators are particularly meaningful in fos-
limitations of existing open-source simulators in validating algo-
rithms and facilitating real-world experimentation. tering collaboration within the researcher community. They
significantly reduce entry barriers for individual researchers and
Index Terms—Autonomous vehicles, simulator, performance small institutions, especially for those where hardware-based
evaluation.
development is financially challenging and creating a new sim-
ulation platform is time-intensive [7]. By doing so, open-source
I. INTRODUCTION simulators alleviate researchers from the burden of repeatedly
N THE realm of autonomous driving, the importance of setting up training workflows and evaluation benchmarks, al-
I simulators has been widely acknowledged within academic
and industrial communities. From 2022 to 2023, over 50% of
lowing them to concentrate more on algorithm development.
Furthermore, open-source simulators facilitate resource-sharing
the methods published in this domain were either trained or by establishing standardized data formats and encouraging the
sharing of sensor data on a common platform. This collaborative
Manuscript received 26 December 2023; revised 18 February 2024; accepted approach will result in significant time and cost savings in data
2 March 2024. Date of publication 6 March 2024; date of current version 26 collection efforts [8]. Influential institutions such as the Com-
July 2024. This work was supported by the National Natural Science Foundation puter Vision Center and Waymo have recognized the importance
of China under Grant 62173228, Grant 62203301, Grant 62373250, and Grant
U22A20100. (Corresponding author: Ming Yang.) of open-source simulators, and thus, they are devoted to devel-
Yueyuan Li, Weihao Yan, Qiyuan Shen, Chunxiang Wang, and Ming Yang are oping Carla and Wayax [7], [9]. These endeavors underscore the
with the Department of Automation, Shanghai Jiao Tong University, Key Lab- growing acknowledgment of the value of open-source simulators
oratory of System Control and Information Processing, Ministry of Education
of China, Shanghai 200240, China (e-mail: [email protected]). in advancing research within the field of autonomous driving.
Wei Yuan and Songan Zhang are with the Global Institute of Future Technol- While several surveys have covered various aspects of
ogy, Shanghai Jiao Tong University, Shanghai 200240, China. autonomous driving, there is a distinct gap in the literature
Color versions of one or more figures in this article are available at
https://doi.org/10.1109/TIV.2024.3374044. regarding a comprehensive exploration of simulators. Despite a
Digital Object Identifier 10.1109/TIV.2024.3374044 considerable increase in open-source simulators in recent years,

2379-8858 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4862 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

existing surveys in this domain primarily focus on available to improve their decision-making capabilities [16]. This view-
alternatives. Works such as [10], [11], [12] investigate the func- point is echoed in other reviews focusing on decision-making
tionalities of a few well-known simulators, listing the specific and motion-planning algorithms [22], [23], [24], [25]. What’s
traffic scenarios they support. There remains a critical need for more, Tampuu et al. expressed concerns about the fidelity of
an in-depth and systematic analysis of the evolution, taxonomy, simulators, highlighting the challenges of transferring decision
and technical challenges autonomous driving simulators face. models trained in virtual environments to the real world [20].
This paper is presented to fill this gap. Readers can regard As simulators are not the primary focus of these surveys and
this survey as a guide for selecting an appropriate open-source reviews, most works merely provide a brief list of available
simulator tailored to their specific needs while being alerted simulation platforms related to their respective themes [26], [27].
to the deficiencies threatening simulation-based experiments’ It was not until 2020, with the increasing number of open-
validity. Concurrently, developers can utilize this survey as source simulators, that reviews and discussions focusing on
a roadmap that highlights the implementation priority of the simulators began to emerge. Zhou et al. conducted a detailed
urgently needed functionalities and derives inspiration for po- survey of seven popular simulators, including CARLA, LGSVL,
tential solutions. CarSim, AirSim, Prescan, Matlab/Simulink, and CarMaker.
The subsequent sections are organized as follows: Their work explored the functionalities of these simulators and
Section II reviews existing literature in the autonomous driv- summarized their respective application scopes [10]. Holen et al.
ing field, covering surveys that mention simulators, reviews spe- conducted a similar study but expanded the range of simulators,
cific to simulators for autonomous driving, and works discussing offering a broader perspective on the available options [11].
issues in simulators for autonomous driving. The simulators mentioned in Kaur et al.’s research were mostly
Section III overviews the evolution of simulators over the past invariant with Zhou’s work, with the main difference being the
three decades, proposing a split into the incipient, dormant, and replacement of AirSim with Gazebo [12]. This survey qualita-
outbreak periods based on development patterns. This section tively describes the performance of the simulators, along with a
identifies a tendency to develop open-source simulators and summary of the desired functionalities and characteristics of
expand the simulation task scope. simulators. The existing reviews intend to assist researchers
Section IV introduces a taxonomy categorizing simulators in choosing a suitable simulator for their task, so their main
based on their suitability for specific tasks. The influential sim- content is to investigate the alternatives and check the available
ulators are classified into five categories: traffic flow simulator, functionalities.
sensory data simulator, driving policy simulator, vehicle dynam- Recently, the discussion about the limitations of simulators
ics simulator, and comprehensive simulator. Each category is is gaining attention. Hu et al. stressed the need to narrow the
explained, typical functionalities are detailed, and guidelines gap between simulation and reality by automating the recon-
for selecting appropriate tools are provided. struction of simulation scenarios and improving the sim2real
Section V identifies critical issues in open-source simulators, model transfer efficiency [28]. [29] holds the same opinion and
justifies the necessity to address them, and investigates potential thoroughly reviews the potential solutions for generating driving
methodologies for improvement. The identified problems, which scenarios. Meanwhile, Zhang et al. identified that enhancing
hazard the validity of experiments, include the degradation of fi- the intelligence of traffic participants’ behaviors is a promising
delity in sensory data, behavior of traffic participants, and vehicle avenue to boost simulator performance [30]. Similarly, Siebke
dynamics. Alongside these are obstacles hindering the effective et al. emphasized the importance of including human errors in
setup and execution of experiments, including data format incon- the traffic simulation process to enhance realism [31]. While
sistency, labor-intensive map construction, slow step updating, these existing works successfully pinpoint critical issues within
and the absence of an in-built interface for hardware-in-the-loop simulators, they tend to focus on isolated specific challenges,
(HIL) testing. creating a void in systematically cataloging the urgent problems
that hinder the ongoing development of simulators.
This paper differs from previous works in that it clarifies
the relationship between the research tasks and the simulator
II. RELATED WORKS categories in the ADS, offering task-specific recommendations
Most surveys and reviews related to autonomous driving fo- for open-source simulators. Moreover, the paper conducts a
cused on specific tasks [13], [14], [15], algorithm categories [16], comprehensive analysis of critical issues in existing simulators,
[17], [18], and overall system architecture [19], [20]. These accompanied by a compelling rationale emphasizing the urgency
reviews admitted that simulators have a wide application in for their resolution. In addition to identifying these issues, the
various scenarios. Yurtsever et al. demonstrated the capability paper summarizes the potential solutions, providing valuable
of simulators to generate diverse data that captures different insights for addressing the identified challenges.
weather conditions [19]. The simulator is regarded as a data
augmentation approach to enhance the robustness of perception
models, enabling them to perform effectively in various scenar- III. HISTORY OF SIMULATORS
ios [21]. Similarly, Grigorescu et al. emphasized the importance The history of simulators for autonomous driving provides
of simulation for deep learning-based driving decision-making valuable insights into the technological evolution within this
models, as these models require diverse interaction behavior domain. By observing the number of simulators released every

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4863

Fig. 1. Number of simulators released by year.

year (Fig. 1), this paper suggests categorizing the development resulting in a lack of urgent demand for generating extensive
of simulators into three distinct periods: the incipient period, the high-fidelity data. These technological barriers and a dearth
dormant period, and the outbreak period. of compelling motivations led to a decade-long gap. Only a
The era before 2000 is characterized as the incipient period, select few simulators survived and left a lasting impact. Among
during which simulators began to be deployed to improve the them were PreScan, supported by TASS International with
intelligence of vehicles and the transportation system. Notable funding and hardware assistance [38]; rFpro, which benefited
commercial entries during this time included PTV Vissim by from financial backing and data support from Formula 1 (F1)
PTV Group in 1992, concentrating on simulations related to racing groups [37]; and VI-grade, a comprehensive simulation
traffic flow and logistics system [32], and Paramics in 1994, project facilitated by HBK Investment [39]. The survival of these
offering comparable functionalities [33]. Complementing these simulators relied on substantial investments from world-leading
commercial options, the open-source simulator SUMO emerged profit organizations.
as a pivotal force in traffic flow simulation [34]. Simultane- A significant growth in the number of simulators began in
ously, in the realm of vehicle dynamics, CarSim and IPG Car- 2016. Thus, this paper named this the outbreak period. The
Maker, established in the 1990 s, forged enduring partnerships advent of Neural Networks (NN) boosted the performance of
with various vehicle manufacturers [35], [36]. Their reliance models in many ADS tasks. Learning-based algorithms depend
on extensive real-world data contributed to their dominance heavily on large-scale data. An image dataset sourced from
in this domain. RFpro, entering the scene in 2007, stood out Grand Theft Auto V (GTA V) underscored the simulators’
as one of the few simulators matching the vehicle dynamics advantage in synthesizing diverse sensory data [40]. Moreover,
fidelity of CarSim and IPG CarMaker, thanks to real-world data simulators provided an ideal environment for training and testing
collection from racing cars [37]. During the incipient period, end-to-end ADS, where the costs for collisions and failures are
the primary focus of simulators was on two topics: traffic negligible compared to real-world scenarios [41]. Despite the
flow and vehicle dynamics. These simulators were developed termination of the interface for ADS development by GTA V’s
under well-developed theories, operated on explicit rules and manufacturer, the research community embarked on developing
physics models, and demanded relatively modest computational independent simulators striving to approximate the real world
resources. as much as possible. Microsoft’s Airsim, LG’s SVL, and CVC’s
Between 2000 and 2015, the period witnessed a dormant CARLA emerged, designed to generate diverse sensory data
phase in simulator development, primarily attributed to tech- and simulate the movement of self-driving vehicles [7], [42],
nological bottlenecks. Shortages in computational resources [43]. With an awareness of the computation and time cost of
hindered support for high-performance and large-scale compu- high-fidelity simulators, a trend emerged favoring only provid-
tations. Essential software techniques, including rendering, 3D ing 2D BEV semantic information to train and verify driving
modeling, and physics engines, were still under development. policies [8], [44], [45].
These factors collectively made the creation of high-fidelity The evolution history of simulators for autonomous driving
simulators a costly endeavor. Furthermore, the potential appli- reveals some trends:
cations of simulators remained underexplored during that time. r Rise of open-source simulators: The increasing prevalence
Learning-based methods had yet to demonstrate their potential, of open-source simulators might be explained by two

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4864 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

Fig. 2. Relationship between the simulator categories, tasks in the ADS, and the compositions of the ADS [35], [42], [45], [46].

factors. Firstly, technological advancements have made Table I presents a comprehensive overview of simulators,
the development of simulators more accessible. The de- including their release year, maintenance status, open status,
creasing costs and enhanced availability of computing support for HIL testing, and related ADS tasks. This paper
resources and software tools have made it easier for de- identifies open-source simulators as those with source codes
velopers to create and distribute open-source simulators. openly available online. Simulators that distribute their core
Secondly, open-source simulators would benefit the entire for free but may charge for technical support or customized
autonomous driving industry. These platforms might con- functionalities are also considered open-source [7], [69]. Com-
tribute to establishing standardized data formats and eval- mercial simulators are covered when discussing the history and
uation benchmarks. This, in turn, fosters a more cohesive taxonomy of the simulators, given their dominant position in
and efficient research environment, facilitating the sharing certain categories. In the Active Maintenance column, those
and validation of findings across the industry. simulators without an update in the past year are annotated with
r Diversification of task scope: In the incipient and dor- a question mark. A simulator is considered no longer actively
mant periods, simulators primarily focused on examining maintained if it has been out of maintenance for over one year.
traffic control designs and vehicle structures. However, as Considering the proliferation of simulators in recent years, this
the focus in autonomous driving research shifted towards paper strives to cover relatively influential simulators, defined
vehicle-centric exploration, simulators assumed a crucial as those with over 100 citations on Google Scholar or over
role in the training and testing of algorithms. The expan- 100 stars on code-sharing platforms like Github. The table is
sion in application range has engendered a higher demand organized according to the proposed taxonomy. Support for HIL
for simulation quality, especially in terms of fidelity and testing is contingent upon the explicit statement in the official
diversity. documentation and the implication by the ecosystem, in which
case there is an integration with HIL-supported autonomous
driving platforms such as Autoware and Baidu Apollo [77],
IV. CATEGORIZATION OF SIMULATORS [78]. The subsequent subsections will present a detailed
This section classifies simulators based on their compatibility exploration of the functionalities of actively maintained open-
with distinct ADS tasks. Most simulators can be categorized into source simulators.
five groups: traffic flow simulator, vehicle dynamics simulator,
sensory data simulator, driving policy simulator, and compre-
A. Traffic Flow Simulator
hensive simulator. It is important to note that specific simulators
may exhibit some overlap in functionalities. Each subsection Traffic flow simulators are software that can simulate the
in this section will delineate a specific category and provide movement of vehicles and other dynamic traffic participants
practical recommendations for selecting actively maintained within a transportation system. Typically, a traffic flow simulator
open-source simulators for associated tasks. may include the following features:

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4865

TABLE I
BASIC INFORMATION OF CATEGORIZED SIMULATORS. FUNCTIONS: 
1 TRAFFIC CONTROL DESIGN; 
2 V2X COMMUNICATION;  3 SENSOR DATA PROCESSING; 
4
DRIVING POLICY DESIGN; 5 END-TO-END DRIVING POLICY DESIGN; 
6 VEHICLE DYNAMICS OPTIMIZATION; 
7 VEHICLE CONTROL

r Road network customization: A traffic flow simulator en- illustrating the location and status of traffic participants
ables users to customize the road network by providing across the map.
detailed information on traffic rules, including speed limits, Initially, traffic flow simulators were employed for urban
traffic lights, and road markings. Users can edit lanes, mobility, road infrastructure planning, and traffic control strat-
junctions, and traffic rules to tailor the simulation to specific egy design [32], [33], [49], [50]. Recent advancements include
scenarios. expanding their support to the development of Connected Au-
r Microscopic traffic simulation: This aspect involves mod- tonomous Vehicles (CAV) and Vehicle-to-Everything (V2X)
eling individual vehicles and their interactions. Each vehi- technologies [46], [47], [48], reflecting a growing attention to
cle is simulated independently, considering factors such as roadside cooperation perception for autonomous driving [79].
acceleration, lane following, and lane changing. To accom- SUMO stands out as the only actively maintained open-source
modate city-level traffic flow simulation, these simulators option among the available traffic flow simulators. It supports
simplify the behavior models of vehicles. a wide range of map formats and keeps improving its algo-
r Visualization: Traffic flow simulators tend to provide a rithms and hardware efficiency. Together, these make it the
simplified bird-eye-view (BEV) graphical visualization, recommended choice when working on tasks like transportation

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4866 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

TABLE II TABLE V
COMPARISON OF THE SENSORS THAT ACTIVELY MAINTAINED OPEN-SOURCE COMPARISON OF THE ACTIVELY MAINTAINED OPEN-SOURCE DRIVING POLICY
SIMULATORS CAN SYNTHESIZE SIMULATORS

TABLE III
COMPARISON OF THE GROUND TRUTH ANNOTATIONS THAT ACTIVELY
MAINTAINED OPEN-SOURCE SIMULATORS CAN SYNTHESIZE

researchers engaged in ADS tasks like object detection, se-

mantic segmentation, instance segmentation, depth prediction,
and trajectory tracking are advised to avoid complete reliance
on simulator-generated data for training and testing purposes.
TABLE IV Among the available simulators, CARLA stands out as the most
COMPARISON OF THE ADVERSE WEATHERS THAT ACTIVELY MAINTAINED
OPEN-SOURCE SIMULATORS CAN SYNTHESIZE suitable option for synthesizing data [7].

C. Driving Policy Simulator

Driving policy simulators are software that offers executable
traffic scenarios for the development and evaluation of driving
policy. The typical functionalities of a driving policy simulator
may include:
r Traffic scenario running: These simulators enable the ex-
planning, traffic control design, road infrastructure design, and
ecution and visualization of driving policy models within
V2X and CAV communication.
specific traffic scenarios.
r Traffic status checking: Typically, driving policy simulators
B. Sensory Data Simulator feature a traffic status checking capability, facilitating the
Sensory data simulators can generate highly realistic sensory validation of driving policy models.
data and ground truth for perception tasks. To support multi- r Real-world trajectory data import: Some driving policy
modal perception tasks, a sensory data simulator must be capable simulator offers an interface for importing real-world tra-
of providing synthesis results from various sensors. One distinct jectory data to enhance the fidelity of traffic scenarios.
advantage of sensory data simulators is their capacity to expe- r Traffic scenario customization: This function allows users
dite data collection under challenging conditions, particularly to customize road layouts and vehicle behavior models to
adverse weather scenarios, thereby saving valuable time [1]. improve the diversity of traffic scenarios.
Table II–IV outlines the sensors, ground truth annotations, and Table V overviews the most concerned features in actively
adverse weathers each sensory data simulator supports. CARLA maintained open-source simulators. Realistic perception de-
is included as it is capable of simulating high-fidelity sensory notes simulators capable of producing photo-realistic RGB im-
data. age output; otherwise, they provide a BEV semantic image.
As is shown in Table I, comprehensive simulators usually Custom map represents that the simulator supports users to
cover the functionalities of sensory data simulators. The key import maps in some popular data format like OpenStreetMap
difference lies in the absence of quality vehicle dynamics sim- (OSM) and OpenDRIVE [83], [84]. It is essential to note that
ulation in sensory data simulators, making them unsuitable in most driving policy simulators lack a built-in scenario editor.
end-to-end driving tasks [43], [51], [52]. If the users need to customize their traffic scenario, a third-
Notably, most sensory data simulators are no longer actively party scenario editor is necessary. Trajectory importing indicates
maintained. This trend may be attributed to the inherent technical whether the simulator is able to import trajectory data from at
challenges and costs of developing high-fidelity sensory data least one of the popular open trajectory datasets: HighD and its
synthesis. As indicated in Table IV, the creation of photo- successors, INTERACTION, Nuplan, and Waymo Open Motion
realistic snowy scenarios remains a considerable challenge, Dataset [8], [85], [86], [87], [88], [89], [90], [91]. Multi-agent
and progress in generating high-fidelity adverse weather con- shows whether the simulator supports multi-agent driving policy
ditions is relatively slow [80], [81], [82]. Given this situation, models.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4867

Fig. 3. Critical issues of simulators and their relationship to the compositions of ADS.

There are numerous available driving policy simulators. Read- The demand for the ability to generate high-fidelity data for
ers can refer to Table V and select the most suitable ones based on every task calls for a substantial commitment of human re-
their requirements. For researchers working on Imitation Learn- sources and financial support to the development process. Con-
ing (IL)-based methodologies, there is no specific requirement sequently, most of the qualified comprehensive simulators are
to set up the simulator, so they can choose whatever simulator for commercial use. The Virtual Test Drive (VTD) is a widely
they prefer. Except for VDrift and CarRacing, all the other sim- acclaimed comprehensive simulator offering highly integrated
ulators in Table V have built-in support for RL-based planning functionalities that satisfy diverse needs in developing an au-
algorithms. Simulators offering photo-realistic outputs, such as tonomous driving system [73]. A similar product, SCAnER,
MetaDrive, are recommended for the end-to-end architecture. presents comparable attributes, yet its popularity remains rel-
atively limited due to a lack of promotion [72]. The only
D. Vehicle Dynamics Simulator open-source, comprehensive simulator confirmed to be under
maintenance is CARLA [7]. At present, there is no substitute
Vehicle dynamics simulators are software that can replicate
for CARLA in some specific tasks among the open-source
the dynamic behavior of vehicles based on physics principles.
simulators.
These simulators consider both internal factors, such as the ve-
hicle’s powertrain, suspension, and aerodynamics, and external
conditions, including road friction, wind resistance, and slope. V. CRITICAL ISSUES AND IMPROVE SUGGESTIONS
By considering these variables, the simulator accurately pre- This section will address the unresolved critical issues within
dicts the dynamic characteristics of vehicles, like acceleration, simulators. The challenges faced by simulators can be sum-
braking, and steering response to different control commands. marized as two general aspects: fidelity and efficiency. Fig. 3
Due to the well-established theories in vehicle dynamics, illustrates the relationship between these critical issues and the
high-quality vehicle dynamics simulators have existed for a modules of the ADS, emphasizing their widespread impact on
considerable period. However, owing to their reliance on real- the training and testing processes across various tasks within the
world experimental data, the most reliable vehicle dynamics ADS framework.
simulators are typically developed for commercial use in close The following subsections will explore the critical issues of
collaboration with car manufacturers [35], [36]. the simulators, providing in-depth descriptions and justifications
Vehicle dynamics simulators find application in optimizing for their significance. Furthermore, the subsections will propose
vehicle design and testing vehicle control algorithms. Among suggestions for improvement by drawing insights from existing
open-source options, Gazebo stands out as a reliable choice literature.
due to its rich experience in physics simulation in the robotics
domain and its active community support [69]. Additionally, A. Fidelity of Sensory Data
for academic institutions collaborating with MathWorks, Matlab
1) Description: Optical sensors, including cameras, LiDAR,
proves to be a considerable choice for its integration capabilities
and radar, play a crucial role in interpreting the dynamic traffic
with various simulators [71].
environment. Unfortunately, these sensors are vulnerable to
adverse light and weather conditions [92]. Factors like illumi-
E. Comprehensive Simulator nation, visibility, and humidity fluctuate in different weather,
A comprehensive simulator is the integration of the function- introducing noise into sensor data, so the detection range and
alities discussed earlier. A comprehensive simulator can support precision are compromised [93]. Adverse weather conditions
multiple tasks across the perception, planning, and control parts also alter the reflectance of roads, further perturbing the efficacy
of the ADS (see Table I). Fig. 2 illustrates how a comprehensive of perception algorithms [94]. Hence, enhancing the robustness
simulator performs Software-In-the-Loop simulation for various of perception algorithms in diverse light and weather conditions
ADS tasks. has been a longstanding objective.
The development of a reliable, comprehensive simulator The process of collecting sensory data from reality is costly
necessitates proficiency across multidisciplinary technologies. and time-consuming. Given the unpredictable nature of weather,

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4868 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

Fig. 4. Comparison of real-world and simulated RGB images in day, night, rainy, and foggy weather. The first row is from BDD100 K dataset, and the second
row is from CARLA [7], [96].

it is common for sensory data to be unavailable under demanded perception results must be alerted to this simulation-to-reality
conditions. In contrast, simulators offer precise control over gap, which poses a hazard to the validity and threatens safety
weather conditions through adjustable parameters, eliminating when conducting real-world tests.
the need for extended waits to encounter specific weather sce- 2) Improvement Suggestions: To improve the fidelity of sen-
narios [1]. Moreover, the labor-intensive task of manually anno- sory data simulation, an intuitive idea is to study the realistic
tating ground truth data is obviated. These inherent advantages noise pattern. In most synthesis processes, the distribution pat-
suggest that if simulators can proficiently generate high-fidelity tern of noise is first extracted, then a mask is generated based
data in adverse weather conditions, they have the potential to on the distribution, and the original data merges with the mask
substitute real-world data and promote relevant research [93], through post-processing. One of the earliest practices involved
[95]. modeling the noise distribution pattern with particle systems and
In the context of synthesized sensory data, fidelity describes physical properties [80], [98], [99], which was initially applied
how faithful the synthesized data is to reality. As indicated to image generation and extended to enhance LiDAR data [82].
in Table II, only a few simulators can generate sensory data In recent years, there has been an increase in learning-based
under complex conditions. Due to technology limitations, the data synthesis methods. This evolution began by employing
simulators still struggle to present the functionality of adverse Deep Neural Networks (DNN) to introduce filter variations,
weather simulation, with the quality of implementation often addressing rain, fog, and camera-blurring situations [100], [101],
overlooked. Fig. 4 shows the fidelity gap between the RGB [102]. K-Radar proposed a Neural Network (NN)-based ap-
images obtained from the leading simulator CARLA and the proach to generate adverse weather data specifically for Radar
real-world dataset BDD100K [7], [96]. This gap explains why applications [103]. As the quest for improved fidelity and com-
the existing simulators may not be qualified for robust perception plexity intensified, Adversarial Generative Networks (GAN)
model training and testing. were introduced to the synthesis task. GANs enabled the genera-
The loss of fidelity of sensory data is closely related to the tion of artificial instances of snowing occasions [104], [105]. The
architecture of simulators. Since these simulators commonly study by [106] underscored the potential of GANs in generating
leverage game engines for development, a prevalent practice is to RGB images under various lighting conditions, including day,
utilize default particle systems within these engines to simulate night, and dawn. Tremblay et al. sought to combine physics rules
adverse weather conditions [7], [42], [43]. In a particle system, with GAN techniques, allowing for control over precipitation in
rain and snow are treated as particles with physics properties. synthesized images [81]. This synthesis approach successfully
During the simulation process, the emphasis is on the movement maintains visual realism while adhering to fundamental physical
trajectory of particles influenced by wind speed and gravity, laws. ClimateNerf serves as the first attempt to apply NeRF to
but less attention is paid to the texture pattern [80], [81], [97]. simulate adverse weather conditions, achieving realistic 3D ren-
Furthermore, existing weather simulations often only impact dering results, particularly in simulating flood and snow [107].
rendering, meaning they can only generate image data for ad- Sensory data simulation algorithms are often available as
verse weather without adding corresponding noise interference open-source projects online [95], [103], [107]. However, these
to LiDAR, radar, etc. algorithms exhibit variations in environment configurations and
One consequence of the fidelity issue in sensory data is that hardware requirements, presenting challenges for users seeking
simulators are limited to providing only ground truth represen- quick adoption. If the simulators could establish a standardized
tations of the surrounding and dynamic objects for downstream interface that defines the input and output of algorithms, it
ADS tasks. Due to insufficient exploration of sensor noise would simplify the process for users and foster a more seamless
patterns, simulators cannot realistically mimic detection er- incorporation of diverse synthesis models into the simulation
rors. Researchers who evaluate their algorithms using simulated framework.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4869

B. Fidelity of Traffic Scenario TABLE VI

COMPARISON OF THE PHYSICS MODEL/DYNAMICS MODEL OF THE ACTIVELY
1) Description: The main components of a traffic scenario MAINTAINED OPEN-SOURCE SIMULATORS
are the geometry map, traffic rubrics, and traffic participants.
Concerning the static parts, the geometry map, and traffic
rubrics, fidelity is not the primary challenge since manual anno-
tation can effectively capture information, including temporary
changes. When handling this static data, the efficiency issue is a
topic to explore in subsection V-E. The dynamic part of the traffic
scenario faces the fidelity problem. Intricate movements and
interactions among traffic participants resist explicit depiction.
Defined by inherent randomness and individual variations, the
dynamic behavior of traffic participants becomes a principal
source of complexity of traffic scenarios.
The traffic participants refer to the dynamic objects in a traffic
scenario, such as vehicles, cyclists, and pedestrians. Ideally,
these participants should be able to perceive their surround-
ings, comprehend the actions of other participants, and navigate
within the scenario [108]. architectures to elevate the sophistication and realism of the
A significant limitation of the ADS simulators lies in the generated behaviors. In this context, empowering open-source
lack of fidelity in traffic participants. Currently, two mainstream simulators to provide a user-friendly interface for customizing
methods exist to generate a traffic scenario in simulators. One traffic participants could prove beneficial. By equipping simu-
approach is replicating actual trajectories, resetting the envi- lators with a reliable traffic event detector and a framework for
ronment when a tested agent violates a rule or experiences a tailoring behavior models, the research community can facilitate
collision [8]. However, this method assumes that the replayed realizing more interactive traffic participant controllers through
trajectory is an optimal solution, potentially limiting the planners collaborative contributions.
to human-level behavior. Another way is to rely on rule-based be-
havioral models, which exhibit simplistic behavior patterns [7],
[109], [110]. These models are constrained to actions like main- C. Fidelity of Vehicle Dynamics
taining a specific distance and stopping, lacking the flexibility to 1) Description: A reliable vehicle dynamics model is critical
circumvent obstacles or accelerate through scenes [111], [112], to guarantee that the planning result of the ADS is executed
[113]. The notable disparity between the behavior of simulated as expected. The fidelity of vehicle dynamics is a particularly
traffic participants and their real-world counterparts undermines prevalent problem in open-source simulators, while it has been
the credibility of simulation experiments in planning tasks [31]. well resolved in commercial vehicle dynamics simulators. This
2) Improvement Suggestions: The key to enhancing the fi- discrepancy may arise from the demanding nature of construct-
delity of dynamic object movements is to refine behavior model- ing an accurate vehicle dynamics model, which necessitates
ing with realistic random events [23], [114]. Lee et al. proposed substantial measurement data, particularly under extreme con-
to treat the modeling process as a trajectory prediction task, ditions such as sharp turns and sudden stops at high speeds com-
leveraging Graphical Neural Networks (GNN) to learn patterns bined with diverse weather and road conditions [121]. Under-
from real-world datasets [115]. [116] followed this idea and taking this endeavor is costly and raises safety concerns, making
implemented TrafficGraphNet, capable of temporal interactions it challenging to accomplish without collaborative support from
in a sparse traffic scenario. Similarly, VectorNet and InterSim vehicle manufacturers.
employ deeper and customized NN to extract motion patterns Table VI investigates the implementation status of vehi-
from datasets [66], [117]. cle physics models in actively maintained open-source simu-
To alleviate a dependency on an initial dataset, researchers lators. Half of the simulators integrate a third-party physics
have explored the concept of adversarial to interaction genera- engine [122], [123], [124], [125], [126]. The rest choose to
tion. RouteGAN applied GAN to produce styled behavior for implement a dynamics/kinematics bicycle model independently.
individual vehicles [118]. Soon, the researchers realized that Most of these physics models are faithful under conditions where
inverse Reinforcement Learning (RL) structure could adopt a velocity, acceleration, and steering angle values are small [127].
similar idea while better fitting in the time-sequential decision- In extreme scenarios, including high-speed maneuvers, sharp
making [119]. A deep deterministic policy gradient was tried to turns, and emergency braking, the prediction result of bicy-
introduce complexity in lane-changing scenarios [120]. Trajgen cle models often diverges significantly from the real world.
pointed out that the combination of trajectory prediction guar- Therefore, researchers seeking to implement their models in
antees fidelity, while the RL method adds flexibility to achieve actual vehicles should not rely solely on open-source simulators.
a realistic and diverse trajectory output [30]. Instead, simulations conducted within commercial vehicle dy-
Generating interactive traffic participants remains a prevalent namics simulators, such as CarMaker, remain indispensable for
research topic [29]. The researchers are developing various ensuring robustness and accuracy. The identified limitations in

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4870 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

TABLE VII
COMPARISON OF THE PUBLIC MAP AND SCENARIO DATA FORMATS THAT THE ACTIVELY MAINTAINED OPEN-SOURCE SIMULATORS SUPPORT

vehicle dynamics within existing open-source simulators under- the fundamental structure of OSM draws inspiration from
score the urgent need for continued improvement and increased road network representation. Consequently, it continues to
collaboration in sharing vehicle testing data. employ a centerline to represent lanes, thereby falling short
2) Improvement Suggestions: Creating a realistic vehicle dy- in high-resolution geometry descriptions of lanes.
namics model relies on real-world data fraught with cost and r OpenDRIVE has gained popularity due to its rich param-
safety concerns, so it is hard for open-source simulators to eters and clear documentation [84]. OpenDRIVE excels
achieve high-fidelity representations. A possible solution lies in in recording high-resolution map geometry but overlooks
establishing a platform where researchers and institutions can traffic control and road semantic information.
contribute locally collected real-vehicle dynamics data. A simi- r Lanelet2 tempts to integrate the strong points of the data
lar data-sharing endeavor has successfully occurred in structured formats mentioned above together [129]. It accurately
maps and vehicle 3D models, which suggests the feasibility of describes geometry structures and versatile, customizable
such an approach [70], [83]. semantic traffic contexts. However, it has not attracted
as much support as OpenDRIVE, potentially because
D. Inconsistency of Data Format Lanelet2 lacks a corresponding scenario data format to
articulate the dynamic objects.
1) Description: Most open-source simulators employ a self- For scenario data formats, OpenSCENARIO provides a pre-
defined data format to record maps and scenarios. Suppose the cise though intricate description of dynamic objects [130]. Each
publicly available data format is defined as one that has an open traffic participant strictly runs on the assigned trajectory in
document that describes all labels and annotations clearly. In that this format, ensuring a detailed representation of scenarios.
case, it becomes evident from Table VII that actively maintained In contrast, Scenic emphasizes introducing randomness into
open-source simulators provided limited support for such data scenarios [131]. It enables the generation of traffic scenarios
formats [83], [84], [128], [129], [130], [131]. The inconsistency by illustrating the probability distribution of behaviors.
in map and scenario formats is problematic, as it necessitates Upon examining the current landscape of data formats, it
redundant efforts in manually constructing traffic scenarios and becomes evident that the roadmap for enhancing map data
maps, even when the data is already accessible. The absence of formats involves combining lane-level representation accuracy
an official data conversion interface also introduces errors during with comprehensive traffic control and semantic information.
data transition. Simultaneously, improvements in scenario data formats should
2) Improvement Suggestions: The underlying problem be- accommodate both accurate scenario replay and flexible random
hind the inconsistency of data format is that the existing data movements. A seamless integration between map data and sce-
structures fail to fulfill the need to express the scenarios com- nario representation demands further attention and dedication.
prehensively and efficiently. The advantages and shortcomings Furthermore, developing an official implementation for a data
of the map data format are as follows: parser is deemed necessary.
r SUMO Road Network is one of the earliest publicly avail-
able map data formats [128]. It emphasizes the relationship
between roads and junctions and provides rich annotation E. Efficiency of Map Construction
for traffic rule restrictions. However, its utility focuses on 1) Description: The simulators for autonomous driving are
traffic flow simulation tasks, leading to a lack of detailed expected to have the ability to quickly rebuild or synthesize
geometry information concerning roads and irregular static diverse traffic scenarios for training and testing purposes [29].
obstacles. While the dynamic objects in traffic scenarios struggle with the
r OSM originates from a free mapping project [83]. Despite fidelity issue (as discussed in Section V-B), the static maps’ con-
its inclusion of support for static obstacles and regions, struction also faces efficiency challenges. Fig. 5 demonstrates

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4871

Fig. 5. Graphical User Interface (GUI) of the scenario editors.

the popular scenario editors [132], [133], [134]. Currently, the on the NeRF algorithm, demonstrating its great potential for
preferred map reconstruction process requires manually extract- automated 3D map reconstruction [146].
ing road structures, labeling road network relationships, and Another avenue focuses on reconstructing the road structure
annotating traffic signals from LiDAR points and images [135]. and traffic information, exhibiting a greater tolerance for de-
This approach heavily relies on handcraft and is inherently viation in the map’s geometry details [147]. These techniques
time-consuming. prioritize comprehensively depicting the traffic environment,
Another challenge in efficiently creating high-quality maps encompassing traffic elements such as the road network, traffic
is related to data formats. The common output of 3D map signals, and dynamic objects [108], [148]. With the overarching
reconstruction, a 3D mesh model, often lacks detailed structural structure and interplay dynamics of the traffic scenario, the
information about roads and buildings [136], [137]. Moreover, 2D maps generated by these approaches become preferable for
semantic annotations for elements like signs and signals are driving decision-making algorithms [117], [149].
not inherently present in 3D mesh models. Meanwhile, though These two categories of automated map reconstruction meth-
2D map reconstruction approaches provide valuable data on ods can reduce the need for manual intervention in constructing
road structure, seamlessly integrating the outputs of 2D recon- simulation scenarios for autonomous driving [65]. However,
struction with 3D map reconstruction poses challenges. This is they are unsuitable for practical implementation at present.
primarily because the 2D map format typically uses polygons to The 3D map construction techniques exhibit sensitivity to data
depict geometric structures, while 3D models utilize triangular distribution and susceptibility to noise interference [148]. The
meshes [83], [129], [138]. Consequently, aligning the 2D road precision and speed of these methods fall short of the standard
structure with the 3D mesh model becomes complex. for industrial application [144].
2) Improvement Suggestions: Automated map reconstruc-
tion utilizes algorithms to process 2D pixels and 3D points
and generate structured maps. Achieving partial or complete F. Efficiency of Simulator
automated map construction can significantly reduce manual 1) Description: Training and testing data-driven ADS tasks
effort and improve the efficiency of building scenarios. are time-consuming due to the need for algorithms to cover a
Automated map reconstruction is advancing along two dis- variety of scenarios, ensuring that challenging corner cases are
tinct trajectories. One avenue is oriented towards 3D recon- addressed [150]. The estimated 215 billion miles for validating
struction, characterized by an emphasis on achieving high-level an ADS highlights the necessity to apply a simulator to speed up
realism and details. The general idea is to extract features from this process [1]. There are several common strategies to improve
2D images and match them with 3D points and pose informa- simulation efficiency:
tion. By establishing correlations between object models and r Hardware acceleration accelerates the simulation process
textures, this method enables effective automatic map recon- by providing more powerful computational resources. An
struction [139], [140], [141]. Traditional 3D map reconstruction upgrade to the CPU, GPU, and memory devices could
methods use explicit priors to process large-scale raw data expedite the computational operations and strengthen the
rapidly [141], [142]. The new rising learning-based methods capability to handle data batches [9].
extract data-driven priors by neural networks to handle noise r Computation optimization is to utilize the computational
and data sparsity [137], [143]. A recent breakthrough in this resource better. It streamlines the simulation workflow
area is the NeRF algorithm. It learns the implicit features of by fine-tuning the algorithm, code structure, and cache
space from images captured from different angles, representing management [151].
3D space as a continuous radiance field [144], [145]. NVIDIA r Parallelization maximize the utility of the computational
has implemented 3D surface reconstruction and rendering based resource at a lower level. It enables simultaneous execution

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4872 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

TABLE VIII
COMPARISON OF THE HARDWARE REQUIREMENTS AND PERFORMANCE OF THE ACTIVELY MAINTAINED OPEN-SOURCE-SOURCE SIMULATORS

of multiple threads. This approach would significantly HIL testing enables meticulous checks of the autonomous vehi-
reduce the simulation time by distributing the workload cle stack’s readiness for real-world deployment by interfacing
across multiple cores or processors [152]. sensors and vehicle controllers with the simulator. This approach
r Distributed computation deploys large-scale computation boosts development efficiency and ensures safety when imple-
resources over different machines. Running multiple simu- menting autonomous driving technologies on actual vehicles.
lators concurrently for training or testing individual models Regrettably, as is shown in Table I, the investigation of HIL
minimizes the overall running time, enhancing the effi- testing support within open-source simulators shows a dismal
ciency of the process [152]. status. Most simulators lack built-in interfaces to receive input
Additionally, headless simulation, also known as off-screen from real-time sensors or send output to vehicle controllers.
simulation, may become an unavoidable consequence for 2) Improvement Suggestions: Developing an interface to fa-
achieving efficient simulation, as powerful hardware and dis- cilitate HIL support presents a formidable challenge for develop-
tributed computation often depend on servers without display. ers of open-source simulators. This challenge arises partly due
A field investigation of the actively maintained open-source to the proprietary nature of vehicle control protocols, which are
simulator shows that most simulators lack the necessary features closely guarded by manufacturers for commercial and safety
for high-efficiency simulation. Table VIII provides insights into reasons. Moreover, the rapid evolution of sensor technologies
the minimal hardware requirements for installation disk space, necessitates a considerable cost to maintain an interface with
the memory occupancy during tutorial running, and the GPU high compatibility across a diverse range of sensor inputs. Al-
memory occupancy, which reflects how well the simulator ar- though some attempts have been made to establish third-party
chitecture has been optimized. According to the official docu- HIL co-simulation for existing open-source simulators [153],
mentation, most simulators are now cross-platform compatible, [154], alongside the ongoing development of simulation plat-
but few of them support hardware acceleration, distributed com- forms equipped with HIL support [155], [156], it is evident that
putation, and headless simulation. this domain remains in its nascent stages. The progress of this
2) Improvement Suggestions: The efficiency of the simula- field still awaits enhanced collaboration between open-source
tors is closely related to the software architecture and code simulator developers and hardware providers in the future.
quality. It is crucial to make thoughtful choices of the back-end
physics engine and programming language at the design stage. VI. CONCLUSION
Some physics engines have rendering library dependencies
This paper presents a comprehensive overview of the his-
incompatible with headless and computational functionalities.
tory of simulators for autonomous driving over the past three
Moreover, choosing a high-level programming language could
decades. According to the increment pattern of simulators, the
slow down computational speed. Implementing a Continuous
timeline is divided into three distinct phases: the incipient pe-
Integration (CI) workflow is imperative during the development
riod (1990s-2000 s), the dormant period (2000s-2015), and the
process, which automatically checks performance and promptly
outbreak period (2015-present). A tendency toward open-source
identifies coding issues, ensuring a smoother and more reliable
accessibility and comprehensive functionality is revealed in the
development pipeline.
evolution process.
After investigating the influential commercial and open-
source simulators, they are classified into four task-specific
G. Support for HIL Testing classes: traffic flow simulator, sensory data simulator, driving
1) Description: HIL testing is a vital testing methodology policy simulator, and vehicle dynamics simulator. Additionally,
that integrates physical hardware with a simulated environment simulators that enclose multiple functionalities are grouped as
to test and validate systems. In the context of ADS development, the fifth type, comprehensive simulators.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4873

This paper systematically explores simulators and identifies [17] A. Haydari and Y. Yılmaz, “Deep reinforcement learning for intelligent
critical issues focusing on open-source variants. Key concerns transportation systems: A survey,” IEEE Trans. Intell. Transp. Syst.,
vol. 23, no. 1, pp. 11–32, Jan. 2022.
include simulation fidelity and efficiency. Fidelity issues ex- [18] Y. Cui et al., “Deep learning for image and point cloud fusion in au-
tend to synthesizing sensory data, traffic scenarios, and vehicle tonomous driving: A review,” IEEE Trans. Intell. Transp. Syst., vol. 23,
dynamics. In generating traffic scenarios, efficiency challenges no. 2, pp. 722–739, Feb. 2022.
[19] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of
arise in constructing detailed maps with rich traffic information autonomous driving: Common practices and emerging technologies,”
with minimal human labor, intricately linked to data format IEEE Access, vol. 8, pp. 58443–58469, 2020.
inconsistency. Simulation efficiency is a global concern while [20] A. Tampuu, T. Matiisen, M. Semikin, D. Fishman, and N. Muhammad,
“A survey of end-to-end driving: Architectures and training methods,”
supporting HIL testing is a difficult challenge awaiting future IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 4, pp. 1364–1384,
opportunities and resources to address effectively. The potential Apr. 2022.
hazards posed by these issues to the validity of ADS models [21] M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and
R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-
developed in simulators are justified, serving as a reminder to generated annotations for real world tasks?,” 2016, arXiv:1610.01983.
simulator users and developers. The paper also reviews potential [22] C. Katrakazas, M. Quddus, W.-H. Chen, and L. Deka, “Real-time motion
mitigation methods for these impacts. planning methods for autonomous on-road driving: State-of-the-art and
future research directions,” Transp. Res. Part C: Emerg. Technol., vol. 60,
This paper is a comprehensive guide for selecting an ap- pp. 416–442, 2015.
propriate simulator for ADS research tasks. Simultaneously, it [23] W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and decision-
calls to address the pressing limitations of actively maintained making for autonomous vehicles,” Ann. Rev. Control, Robot., Auton. Syst.,
vol. 1, pp. 187–210, 2018.
open-source simulators. Simulation technology advancements [24] L. Claussmann, M. Revilloud, D. Gruyer, and S. Glaser, “A review of
are believed to accelerate the implementation and deployment motion planning for highway autonomous driving,” IEEE Trans. Intell.
of autonomous driving technology. Transp. Syst., vol. 21, no. 5, pp. 1826–1848, May 2020.
[25] U. M. Gidado, H. Chiroma, N. Aljojo, S. Abubakar, S. I. Popoola, and M.
A. Al-Garadi, “A survey on deep learning for steering angle prediction in
REFERENCES autonomous vehicles,” IEEE Access, vol. 8, pp. 163797–163817, 2020.
[26] Y. Kang, H. Yin, and C. Berger, “Test your self-driving algorithm:
[1] N. Kalra and S. M. Paddock, “Driving to safety: How many miles of An overview of publicly available driving datasets and virtual testing
driving would it take to demonstrate autonomous vehicle reliability?,” environments,” IEEE Trans. Intell. Veh., vol. 4, no. 2, pp. 171–185,
Transp. Res. Part A: Policy Pract., vol. 94, pp. 182–193, 2016. Jun. 2019.
[2] S. Feng et al., “Dense reinforcement learning for safety validation of [27] L. Wei, Z. Li, J. Gong, C. Gong, and J. Li, “Autonomous driving strategies
autonomous vehicles,” Nature, vol. 615, no. 7953, pp. 620–627, 2023. at intersections: Scenarios, state-of-the-art, and future outlooks,” in Proc.
[3] The Waymo Team, “Simulation city: Introducing waymo’s most ad- IEEE Int. Intell. Transp. Syst. Conf., 2021, pp. 44–51.
vanced simulation system yet for autonomous driving,” 2021. Accessed: [28] X. Hu, S. Li, T. Huang, B. Tang, and L. Chen, “Sim2real and digital twins
Mar. 12, 2024. [Online]. Available: https://waymo.com/blog/2021/06/ in autonomous driving: A survey,” 2023, arXiv:2305.01263.
SimulationCity.html [29] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on
[4] Pony.ai, “Technology,” 2024. Accessed: Mar. 12, 2024. [Online]. Avail- safety-critical driving scenario generation–A methodological perspec-
able: https://pony.ai/tech?lang=en tive,” IEEE Trans. Intell. Transp. Syst., vol. 24, no. 7, pp. 6971–6988,
[5] DiDi, “Didi’s autonomous driving,” 2024. Accessed: Mar. 12, 2024. [On- Jul. 2023.
line]. Available: https://www.didiglobal.com/science/intelligent-driving [30] Q. Zhang et al., “TrajGEN: Generating realistic and diverse trajectories
[6] A. Sinha, S. Chand, V. Vu, H. Chen, and V. Dixit, “Crash and disengage- with reactive and feasible agent behaviors for autonomous driving,” IEEE
ment data of autonomous vehicles on public roads in California,” Sci. Trans. Intell. Transp. Syst., vol. 23, no. 12, pp. 24474–24487, Dec. 2022.
Data, vol. 8, no. 1, 2021, Art. no. 298. [31] C. Siebke, M. Mai, and G. Prokop, “What do traffic simulations have
[7] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: to provide for virtual road safety assessment? human error modeling
An open urban driving simulator,” in Proc. Conf. Robot Learn., 2017, in traffic simulations,” IEEE Trans. Intell. Transp. Syst., vol. 24, no. 2,
pp. 1–16. pp. 1419–1436, Feb. 2023.
[8] H. Caesar et al., “NuPlan: A closed-loop ML-based planning benchmark [32] PTV Group, “PTV vissim: Multimodel traffic simulation software,” 2024.
for autonomous vehicles,” 2021, arXiv:2106.11810. Accessed: Mar. 12, 2024. [Online]. Available: https://www.myptv.com/
[9] C. Gulino et al., “Waymax: An accelerated, data-driven simulator for en/mobility-software/ptv-vissim
large-scale autonomous driving research,” in Proc. Adv. Neural Inf. [33] M. Smith, G. Duncan, and S. Druitt, “PARAMICS: Microscopic traffic
Process. Syst., 2024, vol. 36. simulation for congestion management,” in Proc. IEE Colloquium Dy-
[10] J. Zhou, Y. Zhang, S. Guo, and Y. Guo, “A survey on autonomous driving namic Contr. Strategic Inter-Urban Road Networks, 1995.
system simulators,” in Proc. IEEE Int. Symp. Softw. Rel. Eng. Workshops, [34] K. G. Lim, C. H. Lee, R. K. Y. Chin, K. B. Yeo, and K. T. K. Teo, “SUMO
2022, pp. 301–306. enhancement for vehicular ad hoc network (VANET) simulation,” in
[11] M. Holen, K. Knausgård, and M. Goodwin, “An evaluation of au- Proc. IEEE 2nd Int. Conf. Autom. Control Intell. Syst., 2017, pp. 86–91.
tonomous car simulators and their applicability for supervised and re- [35] Mechanical Simulation, “Carsim,” 2024. Accessed: Mar. 12, 2024. [On-
inforcement learning,” in Proc. Int. Conf. Intell. Technol. Appl., 2021, line]. Available: https://www.carsim.com
pp. 367–379. [36] IPG Automotive, “Solutions for virtual test driving,” 2024. Accessed:
[12] P. Kaur, S. Taghavi, Z. Tian, and W. Shi, “A survey on simulators for Mar. 12, 2024. [Online]. Available: https://ipg-automotive.com/en
testing self-driving cars,” in Proc. IEEE 4th Int. Conf. Connected Auton. [37] rFpro, “The world’s most accurate simulation environment,” 2023. Ac-
Driving, 2021, pp. 62–70. cessed: Mar. 12, 2024. [Online]. Available: https://rfpro.com
[13] D. González, J. Pérez, V. Milanés, and F. Nashashibi, “A review of motion [38] Siemens, “Simcenter prescan software,” 2024. Accessed: Mar.
planning techniques for automated vehicles,” IEEE Trans. Intell. Transp. 12, 2024. [Online]. Available: https://plm.sw.siemens.com/en-US/
Syst., vol. 17, no. 4, pp. 1135–1145, Apr. 2016. simcenter/autonomous-vehicle-solutions/prescan
[14] B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A survey of [39] VI-grade, “Driving simulators: An invaluable set of tools to suc-
motion planning and control techniques for self-driving urban vehicles,” cessfully accelerate your development process,” 2023. Accessed: Mar.
IEEE Trans. Intell. Veh., vol. 1, no. 1, pp. 33–55, Mar. 2016. 12, 2024. [Online]. Available: https://www.vi-grade.com/en/solutions/
[15] D. Feng, A. Harakeh, S. L. Waslander, and K. Dietmayer, “A review driving_simulators
and comparative study on probabilistic object detection in autonomous [40] M. Martinez, C. Sitawarin, K. Finch, L. Meincke, A. Yablonski, and
driving,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 8, pp. 9961–9980, A. Kornhauser, “Beyond grand theft auto v for training, testing and
Aug. 2022. enhancing deep learning in self driving cars,” 2017, arXiv:1712.01397.
[16] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep [41] D. Wang, J. Wen, Y. Wang, X. Huang, and F. Pei, “End-to-end self-driving
learning techniques for autonomous driving,” J. Field Robot., vol. 37, using deep neural networks with multi-auxiliary tasks,” Automot. Innov.,
no. 3, pp. 362–386, 2020. vol. 2, pp. 127–136, 2019.
Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4874 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

[42] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “AirSim: High-fidelity visual [67] E. Vinitsky, N. Lichtlé, X. Yang, B. Amos, and J. Foerster, “Nocturne:
and physical simulation for autonomous vehicles,” in Proc. Field Serv. A scalable driving benchmark for bringing multi-agent learning one step
Robot.: Results 11th Int. Conf., 2018, pp. 621–635. closer to the real world,” in Proc. Adv. Neural Inf. Process. Syst., 2022,
[43] G. Rong et al., “LGSVL simulator: A high fidelity simulator for au- pp. 3962–3974.
tonomous driving,” in Proc. IEEE 23rd Int. Conf. Intell. Transp. Syst., [68] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone, “BITS: Bi-level imitation
2020, pp. 1–6. for traffic simulation,” in Proc. IEEE Int. Conf. Robot. Automat., 2023,
[44] E. Leurent, “An environment for autonomous driving decision-making,” pp. 2929–2936.
GitHub Repository, 2018. Accessed: Mar. 12, 2024. [Online]. Available: [69] O. Michel, “Cyberbotics ltd. webots: Professional mobile robot simula-
https://github.com/eleurent/highway-env tion,” Int. J. Adv. Robot. Syst., vol. 1, no. 1, p. 5, 2004.. [Online]. Available:
[45] M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Composable https://journals.sagepub.com/doi/full/10.5772/5618
benchmarks for motion planning on roads,” in Proc. IEEE Intell. Veh. [70] N. Koenig and A. Howard, “Design and use paradigms for gazebo, an
Symp., 2017, pp. 719–726. open-source multi-robot simulator,” in Proc. IEEE/RSJ Int. Conf. Intell.
[46] P. A. Lopez et al., “Microscopic traffic simulation using SUMO,” in Proc. Robots Syst., 2004, pp. 2149–2154.
IEEE 21st Int. Conf. Intell. Transp. Syst., 2018, pp. 2575–2582. [Online]. [71] The MathWorks, “Vehicle dynamics blockset: Model and simulate ve-
Available: https://elib.dlr.de/124092/ hicle dynamics in a virtual 3D environment,” 2024. Accessed: Mar.
[47] J. Barceló and J. Casas, “Dynamic network simulation with AIMSUN,” 12, 2024. [Online]. Available: https://www.mathworks.com/products/
in Simulation Approaches in Transportation Analysis: Recent Advances vehicle-dynamics.html
and Challenges. Berlin, Germany: Springer, 2005, pp. 57–98. [72] AVSimulation, “SCANeR 2023.2 release note,” 2023. Accessed: Mar.
[48] J. Auld, M. Hope, H. Ley, V. Sokolov, B. Xu, and K. Zhang, “Po- 12, 2024. [Online]. Available: https://www.avsimulation.com/scaner-
laris: Agent-based modeling framework development and implemen- 2023-2-release-note
tation for integrated travel demand and network and operations simu- [73] Hexagon, “Virtual test drive: Complete tool-chain for driving simulation
lations,” Transp. Res. Part C: Emerg. Technol., vol. 64, pp. 101–116, applications,” 2024. Accessed: Mar. 12, 2024. [Online]. Available: https:
2016. //hexagon.com/products/virtual-test-drive
[49] C. Wu, A. Kreidieh, K. Parvate, E. Vinitsky, and A. M. Bayen, “Flow: [74] Deepdrive Team, “Deepdrive: A simulator that allows anyone with a pc to
Architecture and benchmarking for reinforcement learning in traffic push the state-of-the-art in self-driving,” 2019. Accessed: Mar. 12, 2024.
control,” 2017, arXiv:1710.05465. [Online]. Available: https://github.com/deepdrive/deepdrive
[50] H. Zhang et al., “Cityflow: A multi-agent reinforcement learning envi- [75] Nvidia, “Nvidia drive end-to-end platform for software-defined vehicles,”
ronment for large scale city traffic scenario,” in Proc. World Wide Web 2024. Accessed: Mar. 12, 2024. [Online]. Available: https://www.nvidia.
Conf., 2019, pp. 3620–3624. com/en-us/self-driving-cars/
[51] M. Müller, V. Casser, J. Lahoud, N. Smith, and B. Ghanem, “Sim4cv: [76] A. Amini et al., “Vista 2.0: An open, data-driven simulator for multimodal
A photo-realistic simulator for computer vision applications,” Int. J. sensing and policy learning for autonomous vehicles,” in Proc. IEEE Int.
Comput. Vis., vol. 126, pp. 902–919, 2018. Conf. Robot. Automat., 2022, pp. 2419–2426.
[52] Parallel Domain, “Parallel domain,” 2023. Accessed: Mar. 12, 2024. [77] S. Kato et al., “Autoware on board: Enabling autonomous vehicles with
[Online]. Available: https://paralleldomain.com/ embedded systems,” in Proc. ACM/IEEE 9th Int. Conf. Cyber- Phys.
[53] Z. Yang et al., “UniSim: A neural closed-loop sensor simulator,” in Proc. Syst., 2018, pp. 287–296.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 1389–1399. [78] Baidu, “Baidu apollo,” 2023. Accessed: Mar. 12, 2024. [Online]. Avail-
[54] B. Wymann, E. Espié, C. Guionneau, C. Dimitrakakis, R. Coulom, and able: https://github.com/ApolloAuto/apollo
A. Sumner, “Torcs, the open racing car simulator,” 2020. Accessed: Mar. [79] C. Olaverri-Monreal, J. Errea-Moreno, A. Díaz-Álvarez, C. Biurrun-
12, 2024. [Online]. Available: https://sourceforge.net/projects/torcs/ Quel, L. Serrano-Arriezu, and M. Kuba, “Connection of the SUMO
[55] F. Kehrle, J. V. Frasch, C. Kirches, and S. Sager, “Optimal control of microscopic traffic simulator and the unity 3D game engine to evaluate
formula 1 race cars in a vdrift based virtual environment,” IFAC Proc. v2x communication-based systems,” Sensors, vol. 18, no. 12, 2018,
Volumes, vol. 44, no. 1, pp. 11907–11912, 2011. Art. no. 4399.
[56] C. Campbell, “Box2D C++ tutorials - top-down car physics,” 2014. [80] A. Von Bernuth, G. Volk, and O. Bringmann, “Simulating photo-realistic
Accessed: Mar. 12, 2024. [Online]. Available: http://www.iforce2d.net/ snow and fog on existing images for enhanced CNN training and evalu-
b2dtut/top-down-car ation,” in Proc. IEEE Intell. Transp. Syst. Conf., 2019, pp. 41–46.
[57] Udacity, “Udacity self-driving platform,” 2017. Accessed: Mar. 12, 2024. [81] M. Tremblay, S. S. Halder, R. De Charette, and J.-F. Lalonde, “Rain
[Online]. Available: https://github.com/udacity/self-driving-car-sim rendering for evaluating and improving robustness to bad weather,” Int.
[58] P. Palanisamy, “Multi-agent connected autonomous driving using deep J. Comput. Vis., vol. 129, pp. 341–360, 2021.
reinforcement learning,” in Proc. IEEE Int. Joint Conf. Neural Netw., [82] M. Hahner et al., “LiDAR snowfall simulation for robust 3D object
2020, pp. 1–7. detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
[59] J. Bernhard, K. Esterle, P. Hart, and T. Kessler, “BARK: Open behavior 2022, pp. 16364–16374.
benchmarking in multi-agent environments,” in Proc. IEEE/RSJ Int. [83] M. Haklay and P. Weber, “OpenStreetMap: User-generated
Conf. Intell. Robots Syst., 2020, pp. 6201–6208. street maps,” IEEE Pervasive Comput., vol. 7, no. 4, pp. 12–18,
[60] P. Kothari, C. Perone, L. Bergamini, A. Alahi, and P. Ondruska, “Driver- Oct.–Dec. 2008.
Gym: Democratising reinforcement learning for autonomous driving,” in [84] M. Dupuis, M. Strobl, and H. Grezlikowski, “Opendrive 2010 and
Proc. NeurIPS Workshop Mach. Learn. Autonom. Driving, 2021. beyond–status and future of the de facto standard for the descrip-
[61] M. Zhou et al., “Smarts: An open-source scalable multi-agent rl training tion of road networks,” in Proc. Driving Simul. Conf. Eur., 2010,
school for autonomous driving,” in Proc. Conf. Robot Learn., 2021, pp. 231–242.
pp. 264–285. [85] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highD dataset:
[62] P. Cai, Y. Lee, Y. Luo, and D. Hsu, “SUMMIT: A simulator for urban driv- A drone dataset of naturalistic vehicle trajectories on german highways
ing in massive mixed traffic,” in Proc. IEEE Int. Conf. Robot. Automat., for validation of highly automated driving systems,” in Proc. IEEE 21st
2020, pp. 4023–4029. Int. Conf. Intell. Transp. Syst., 2018, pp. 2118–2125.
[63] DI-drive Contributors, “DI-drive: OpenDILab decision intelligence plat- [86] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein,
form for autonomous driving simulation,” 2021. Accessed: Mar. 12, 2024. “The inD dataset: A drone dataset of naturalistic road user trajecto-
[Online]. Available:https://github.com/opendilab/DI-drive ries at german intersections,” in Proc. IEEE Intell. Veh. Symp., 2020,
[64] J. Herman et al., “Learn-to-race: A multimodal control environment for pp. 1929–1934.
autonomous racing,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, [87] R. Krajewski, T. Moers, J. Bock, L. Vater, and L. Eckstein, “The round
pp. 9793–9802. dataset: A drone dataset of road user trajectories at roundabouts in
[65] Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “MetaDrive: Germany,” in Proc. IEEE 23rd Int. Conf. Intell. Transp. Syst., 2020,
Composing diverse driving scenarios for generalizable reinforcement pp. 1–6.
learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 3, [88] J. Bock, L. Vater, R. Krajewski, and T. Moers, “Highly accurate scenario
pp. 3461–3475, Mar. 2023. and reference data for automated driving,” ATZ Worldwide, vol. 123,
[66] Q. Sun, X. Huang, B. C. Williams, and H. Zhao, “Intersim: Interactive no. 5, pp. 50–55, 2021.
traffic simulation via explicit relation modeling,” in Proc. IEEE/RSJ Int. [89] Y. Xu et al., “SIND: A drone dataset at signalized intersection in China,”
Conf. Intell. Robots Syst., 2022, pp. 11416–11423. in Proc. IEEE 25th Int. Conf. Intell. Transp. Syst., 2022, pp. 2471–2478.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 LI et al.: CHOOSE YOUR SIMULATOR WISELY: A REVIEW ON OPEN-SOURCE SIMULATORS FOR AUTONOMOUS DRIVING 4875

[90] W. Zhan et al., “Interaction dataset: An international, adversarial and [116] S. Kumar, Y. Gu, J. Hoang, G. C. Haynes, and M. Marchetti-
cooperative motion dataset in interactive driving scenarios with semantic Bowick, “Interaction-based trajectory prediction over a hybrid traf-
maps,” 2019, arXiv:1910.03088. fic graph,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2021,
[91] S. Ettinger et al., “Large scale interactive motion forecasting for au- pp. 5530–5535.
tonomous driving: The waymo open motion dataset,” in Proc. IEEE/CVF [117] J. Gao et al., “VectorNet: Encoding HD maps and agent dynamics
Int. Conf. Comput. Vis., 2021, pp. 9710–9719. from vectorized representation,” in Proc. IEEE/CVF Conf. Comput. Vis.
[92] J. Vargas, S. Alsweiss, O. Toker, R. Razdan, and J. Santos, “An overview Pattern Recognit., 2020, pp. 11525–11533.
of autonomous vehicles sensors and their vulnerability to weather con- [118] Z.-H. Yin, L. Sun, L. Sun, M. Tomizuka, and W. Zhan, “Diverse critical
ditions,” Sensors, vol. 21, no. 16, 2021, Art. no. 5397. interaction generation for planning and planner evaluation,” in Proc.
[93] C. Goodin, D. Carruth, M. Doude, and C. Hudson, “Predicting the IEEE/RSJ Int. Conf. Intell. Robots Syst., 2021, pp. 7036–7043.
influence of Rain on LiDAR in Adas,” Electronics, vol. 8, no. 1, p. 89, [119] S. Shiroshita et al., “Behaviorally diverse traffic simulation via reinforce-
2019. [Online]. Available: https://www.mdpi.com/2079-9292/8/1/89 ment learning,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2020,
[94] S. Zang, M. Ding, D. Smith, P. Tyler, T. Rakotoarivelo, and M. A. Kaafar, pp. 2103–2110.
“The impact of adverse weather conditions on autonomous vehicles: How [120] B. Chen, X. Chen, Q. Wu, and L. Li, “Adversarial evaluation of au-
rain, snow, fog, and hail affect the performance of a self-driving car,” tonomous vehicles in lane-change scenarios,” IEEE Trans. Intell. Transp.
IEEE Veh. Technol. Mag., vol. 14, no. 2, pp. 103–111, Jun. 2019. Syst., vol. 23, no. 8, pp. 10333–10342, Aug. 2022.
[95] W. Li, Q. Zhang, J. Zhang, Z. Huang, X. Tian, and D. Tao, “Toward [121] N. A. Spielberg, M. Templer, J. Subosits, and J. C. Gerdes, “Learning
real-world single image deraining: A new benchmark and beyond,” 2022, policies for automated racing using vehicle model gradients,” IEEE Open
arXiv:2206.05514. J. Intell. Transp. Syst., vol. 4, pp. 130–142, 2023.
[96] F. Yu et al., “Bdd100 k: A diverse driving dataset for heterogeneous [122] R. Smith et al., “Open dynamics engine,” 2004. Accessed: Mar. 12, 2024.
multitask learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern [Online]. Available: http://www.ode.org/
Recognit., 2020, pp. 2636–2645. [123] I. Parberry, Introduction to Game Physics With Box2D. Boca Raton, FL,
[97] D. Dunkerley, “Rain event properties in nature and in rainfall simulation USA: CRC Press, 2017.
experiments: A comparative review with recommendations for increas- [124] E. Coumans and Y. Bai, “Pybullet quickstart guide,” 2021. Accessed:
ingly systematic study and reporting,” Hydrological Processes: An Int. Mar. 12, 2024. [Online]. Available: https://raw.githubusercontent.com/
J., vol. 22, no. 22, pp. 4415–4435, 2008. bulletphysics/bullet3/master/docs/pybullet_quickstartguide.pdf
[98] S. Starik and M. Werman, “Simulation of rain in videos,” in Proc. Texture [125] A. Sanders, An Introduction to Unreal Engine 4. Boca Raton, FL, USA:
Workshop, 2003, vol. 2, pp. 406–409. CRC, 2016.
[99] C. Wang, Z. Wang, T. Xia, and Q. Peng, “Real-time snowing simulation,” [126] M. Goslin and M. R. Mine, “The panda 3D graphics engine,” Computer,
Vis. Comput., vol. 22, pp. 315–323, 2006. vol. 37, no. 10, pp. 112–114, 2004.
[100] Y. Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: Automated testing of [127] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and
deep-neural-network-driven autonomous cars,” in Proc. 40th Int. Conf. dynamic vehicle models for autonomous driving control design,” in Proc.
Softw. Eng., 2018, pp. 303–314. IEEE Intell. Veh. Symp., 2015, pp. 1094–1099.
[101] C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene under- [128] M. Behrisch, L. Bieker, J. Erdmann, and D. Krajzewicz, “SUMO–
standing with synthetic data,” Int. J. Comput. Vis., vol. 126, pp. 973–992, simulation of urban mobility: An overview,” in Proc. SIMUL 3rd Int.
2018. Conf. Adv. Syst. Simul., 2011, pp. 23–28.
[102] A. Kar et al., “Meta-Sim: Learning to generate synthetic datasets,” in [129] F. Poggenhans et al., “Lanelet2: A high-definition map framework for
Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 4551–4560. the future of automated driving,” in Proc. 21st Int. Conf. Intell. Transp.
[103] D.-H. Paek, S.-H. Kong, and K. T. Wijaya, “K-Radar: 4D radar object Syst., 2018, pp. 1672–1679.
detection for autonomous driving in various weather conditions,” in Proc. [130] Association for Standardization of Automation and Measuring Systems,
Adv. Neural Inf. Process. Syst., 2022, pp. 3819–3829. “Asam openscenario: User guide,” 2020. Accessed: Mar. 12, 2024.
[104] M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid, “DeepRoad: [Online]. Available: https://releases.asam.net/OpenSCENARIO/1.0.0/
GAN-based metamorphic testing and input validation framework for ASAM_OpenSCENARIO_BS-1-2_User-Guide_V1-0-0.html
autonomous driving systems,” in Proc. ACM/IEEE 33rd Int. Conf. Auto- [131] D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-
mated Softw. Eng., 2018, pp. 132–142. Vincentelli, and S. A. Seshia, “Scenic: A language for scenario specifica-
[105] V. Muşat, I. Fursa, P. Newman, F. Cuzzolin, and A. Bradley, “Multi- tion and scene generation,” in Proc. 40th ACM SIGPLAN Conf. Program.
weather city: Adverse weather stacking for autonomous driving,” in Proc. Lang. Des. Implementation, 2019, pp. 63–78.
IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 2906–2915. [132] D. Krajzewicz, G. Hertkorn, J. Ringel, and P. Wagner, “Preparation of
[106] Z. Yang et al., “SurfelGAN: Synthesizing realistic sensor data for digital maps for traffic simulation; Part 1: Approach and algorithms,” in
autonomous driving,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Proc. 3rd Ind. Simul. Conf., 2005, pp. 285–290.
Recognit., 2020, pp. 11118–11127. [133] S. Maierhofer, M. Klischat, and M. Althoff, “Commonroad scenario
[107] Y. Li, Z.-H. Lin, D. Forsyth, J.-B. Huang, and S. Wang, “Climatenerf: designer: An open-source toolbox for map conversion and scenario
Extreme weather synthesis in neural radiance field,” in Proc. IEEE/CVF creation for autonomous vehicles,” in Proc. IEEE Int. Intell. Transp. Syst.
Int. Conf. Comput. Vis., 2023, pp. 3227–3238. Conf., 2021, pp. 3176–3182.
[108] Z. Ghodsi et al., “Generating and characterizing scenarios for safety [134] MathWorks, “Get started with roadrunner,” 2024. Accessed: Mar.
testing of autonomous vehicles,” in Proc. IEEE Intell. Veh. Symp., 2021, 12, 2024. [Online]. Available: https://www.mathworks.com/help/
pp. 157–164. roadrunner/get-started-with-roadrunner.html
[109] J. A. Michon, “Explanatory pitfalls and rule-based driver models,” Acci- [135] J. Huang, J. Stoter, R. Peters, and L. Nan, “City3d: Large-scale building
dent Anal. Prevention, vol. 21, no. 4, pp. 341–353, 1989. reconstruction from airborne LiDAR point clouds,” Remote Sens., vol. 14,
[110] A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model no. 9, 2022, Art. no. 2254.
mobil for car-following models,” Transp. Res. Rec., vol. 1999, no. 1, [136] L. Nan and P. Wonka, “PolyFit: Polygonal surface reconstruction from
pp. 86–94, 2007. point clouds,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2353–
[111] S. Krauß, “Microscopic modeling of traffic flow: Investigation of collision 2361.
free vehicle dynamics,” Res. Rep. 98-08 German Center Air Space [137] J. Pan, X. Han, W. Chen, J. Tang, and K. Jia, “Deep mesh reconstruction
Navigation, 1998. from single RGB images via topology modification networks,” in Proc.
[112] J. Erdmann, “SUMO’s lane-changing model,” in Proc. Model. Mobility IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 9964–9973.
With Open Data: 2nd SUMO Conf., 2015, pp. 105–123. [138] G.-H. Lee, P.-H. Choi, J.-H. Nam, H.-S. Han, S.-H. Lee, and S.-C.
[113] S. Panwai and H. Dia, “Comparative evaluation of microscopic car- Kwon, “A study on the performance comparison of 3D file formats
following behavior,” IEEE Trans. Intell. Transp. Syst., vol. 6, no. 3, on the web,” Int. J. Adv. Smart Convergence, vol. 8, no. 1, pp. 65–74,
pp. 314–325, Sep. 2005. 2019.
[114] R. Bhattacharyya, S. Jung, L. A. Kruse, R. Senanayake, and M. J. [139] C. Früh and A. Zakhor, “An automated method for large-scale, ground-
Kochenderfer, “A hybrid rule-based and data-driven approach to driver based city model acquisition,” Int. J. Comput. Vis., vol. 60, pp. 5–24,
modeling through particle filtering,” IEEE Trans. Intell. Transp. Syst., 2004.
vol. 23, no. 8, pp. 13055–13068, Aug. 2022. [140] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: Exploring photo
[115] D. Lee, Y. Gu, J. Hoang, and M. Marchetti-Bowick, “Joint interaction collections in 3D,” in Proc. ACM siggraph Papers, 2006, pp. 835–846.
and trajectory prediction for autonomous driving using graph neural [141] S. Agarwal et al., “Building rome in a day,” Commun. ACM, vol. 54,
networks,” 2019, arXiv:1912.07882. no. 10, pp. 105–112, 2011.
Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.
 4876 IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, VOL. 9, NO. 5, MAY 2024

[142] F. Williams, M. Trager, J. Bruna, and D. Zorin, “Neural splines: Fitting Songan Zhang received B.S. and M.S. degrees in
3D surfaces with infinitely-wide neural networks,” in Proc. IEEE/CVF automotive engineering from Tsinghua University,
Conf. Comput. Vis. Pattern Recognit., 2021, pp. 9949–9958. Beijing, China, in 2013 and 2016, respectively, and
[143] A. Boulch and R. Marlet, “Poco: Point convolution for surface reconstruc- the Ph.D. degree in mechanical engineering from the
tion,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, University of Michigan, Ann Arbor, MI, USA, in
pp. 6302–6314. 2021. After graduation, she was a Research Scientist
[144] M. Tancik et al., “Block-NeRF: Scalable large scene neural view syn- with Ford Motor Company in the Robotics Research
thesis,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, Team. She is currently an Assistant Professor with the
pp. 8248–8258. Global Institute of Future Technology, Shanghai Jiao
[145] J. Huang, Z. Gojcic, M. Atzmon, O. Litany, S. Fidler, and F. Williams, Tong University, Shanghai, China. Her research inter-
“Neural kernel surface reconstruction,” in Proc. IEEE/CVF Conf. Com- ests include the accelerated evaluation of autonomous
put. Vis. Pattern Recognit., 2023, pp. 4369–4379. vehicles, model-based reinforcement learning, and meta-reinforcement learning
[146] Z. Li et al., “Neuralangelo: High-fidelity neural surface reconstruc- for autonomous vehicle decision-making.
tion,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023,
pp. 8456–8465.
[147] A. Mondal, P. Tigas, and Y. Gal, “Real2sim: Automatic generation of
open street map towns for autonomous driving benchmarks,” in Proc.
Mach. Learn. Auton. Driving Workshop 34th Conf. Neural Inf. Process.
Weihao Yan received the bachelor’s degree in au-
Syst., 2020.
tomation in 2020 from Shanghai Jiao Tong Univer-
[148] S. Tan et al., “SceneGen: Learning to generate realistic traffic scenes,”
sity, Shanghai, China, where he is currently working
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021,
toward the Ph.D. degree in control science and en-
pp. 892–901.
gineering. His research interests include autonomous
[149] Y. Tian, A. Carballo, R. Li, and K. Takeda, “Real-to-synthetic: Generating
driving systems, computer vision, domain adaptation,
simulator friendly traffic scenes from graph representation,” in Proc.
IEEE Intell. Veh. Symp., 2022, pp. 1615–1622. virtual-to-real transfer learning, scene segmentation,
and foundation models.
[150] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving
models from large-scale video datasets,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit., 2017, pp. 2174–2182.
[151] B. Varga, D. Doba, and T. Tettamanti, “Optimizing vehicle dynamics co-
simulation performance by introducing mesoscopic traffic simulation,”
Simul. Modelling Pract. Theory, vol. 125, 2023, Art. no. 102739.
[152] A. Rousset, B. Herrmann, C. Lang, and L. Philippe, “A survey on parallel
and distributed multi-agent systems for high performance computing
simulations,” Comput. Sci. Rev., vol. 22, pp. 27–46, 2016. Qiyuan Shen received the bachelor’s degree in au-
[153] Y. Chen, S. Chen, T. Zhang, S. Zhang, and N. Zheng, “Autonomous tomation from Northeastern University, Shenyang,
vehicle testing and validation platform: Integrated simulation system China, in 2022. He is currently working toward the
with hardware in the loop,” in Proc. IEEE Intell. Veh. Symp., 2018, Ph.D. degree in control science and engineering with
pp. 949–956. Shanghai Jiao Tong University, Shanghai, China. His
[154] C. Brogle, C. Zhang, K. L. Lim, and T. Bräunl, “Hardware-in-the-loop research interests include the localization of the au-
autonomous driving simulation without real-time constraints,” IEEE tonomous driving system and SLAM, multimodal
Trans. Intell. Veh., vol. 4, no. 3, pp. 375–384, Sep. 2019. mapping, cross-modal localization, geometrical cali-
[155] Ş. Y. Gelbal, S. Tamilarasan, M. R. Cantaş, L. Güvenç, and B. Aksun- bration for vision, and LiDAR.
Güvenç, “A connected and autonomous vehicle hardware-in-the-loop
simulator for developing automated driving algorithms,” in Proc. IEEE
Int. Conf. Syst., Man, Cybern., 2017, pp. 3397–3402.
[156] J. E. Heikkinen, S. Gafurov, S. Kopylov, T. Minav, S. Grebennikov, and
A. Kurbanov, “Hardware-in-the-loop platform for testing autonomous
vehicle control algorithms,” in Proc. IEEE 12th Int. Conf. Develop.
Esystems Eng., 2019, pp. 906–911.
Chunxiang Wang (Member, IEEE) received the
Ph.D. degree in mechanical engineering from the
Harbin Institute of Technology, Harbin, China, in
1999. She is currently an Associate Professor with
Yueyuan Li received the bachelor’s degree in elec- the Department of Automation, Shanghai Jiao Tong
trical and computer engineering from the University University, Shanghai, China. She is working on intel-
of Michigan-Shanghai Jiao Tong University Joint ligent vehicles for more than ten years and has par-
Insitute, Shanghai, China, in 2020. She is currently ticipated in several related research projects, such as
working toward the Ph.D. degree in control science European CyberC3 Project, and ITER Transfer Cask
and engineering from Shanghai Jiao Tong University, Project. Her research interests include autonomous
Shanghai. Her research interest include the secu- driving, assistant driving, and mobile robots.
rity of the autonomous driving system and driving
decision-making, driving decision-making models,
driving simulation, and virtual-to-real model trans-
ferring. Ming Yang (Member, IEEE) received the master’s
and Ph.D. degrees from Tsinghua University, Beijing,
China, in 1999 and 2003, respectively. He currently
holds the position of a Distinguished Professor with
Shanghai Jiao Tong University, Shanghai, China, and
Wei Yuan received the master’s and Ph.D. degrees the Director of the Innovation Center of Intelligent
in automation from Shanghai Jiao Tong University, Connected Vehicles. He has engaged in the research
Shanghai, China, in 2017 and 2021, respectively. He of intelligent vehicles for more than 25 years.
is currently a Postdoctoral Researcher with Shanghai
Jiao Tong University. His research interests include
autonomous driving systems, computer vision, deep
learning, vehicle control, end-to-end learning-based
vehicle control, and decision-making.

Authorized licensed use limited to: Universita degli Studi di Napoli Federico II. Downloaded on September 24,2024 at 06:58:54 UTC from IEEE Xplore. Restrictions apply.