Questioning Based on Driving Simulator Data

Analysis: Are Autonomous Vehicles Ready to

Solve Urban Mobility Problems?

Ismail Furkan Colakoglu

Transport Engineering, Planning and Management MSc
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Contents
List of Figures ................................................................................................................................................ 2
List of Tables ................................................................................................................................................. 2
List of Acronyms ........................................................................................................................................... 3
1. Introduction .............................................................................................................................................. 4
2. Brief Information About Simulation Process ............................................................................................ 6
3. Data Analysis............................................................................................................................................. 7
3.1. Preparation of the Data ..................................................................................................................... 7
3.2. Detailed Analysis ............................................................................................................................... 8
4. Comments............................................................................................................................................... 11
5. Conclusion .............................................................................................................................................. 11
6. References .............................................................................................................................................. 12
7. Appendix ................................................................................................................................................. 12

1
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

List of Figures
Figure 1. SAE Levels (SAE Levels of Driving Automation™ Refined for Clarity and International Audience,
2021) ............................................................................................................................................................. 6
Figure 2. Q-Q Plot (Statistics online, 2022)................................................................................................... 9
Figure 3. T Distribution (Statistics online, 2022) ........................................................................................ 10
Figure 4. Velocity/Time Graphs .................................................................................................................. 13
Figure 5. Acceleration/Time Graphs ........................................................................................................... 13
Figure 6. Gas Position/Time Graphs ........................................................................................................... 13
Figure 7. Brake Position/Time Graphs ........................................................................................................ 13
Figure 8. Steering Position/Time Graphs .................................................................................................... 14
Figure 9. Person 1 Steering Position and Gas Position Comparisons ......................................................... 14
Figure 10. Person 2 Steering Position and Gas Position Comparisons ....................................................... 14
Figure 11. Person 3 Steering Position and Gas Position Comparisons ....................................................... 15
Figure 12. Person 4 Steering Position and Gas Position Comparisons ....................................................... 15
Figure 13. Person 6 Steering Position and Gas Position Comparisons ....................................................... 15
Figure 14. Person 7 Steering Position and Gas Position Comparisons ....................................................... 16
Figure 15. Person 8 Steering Position and Gas Position Comparisons ....................................................... 16

List of Tables
Table 1. Information About Participants (Goodman, 2022) ......................................................................... 6
Table 2. Test 1 and Test 2 Significant Times and Parameters ...................................................................... 7
Table 3. Test 1 and Test 2 Significant Times and Parameters After Extraction ............................................ 8
Table 4. Test 1 and Test 2 Significant Parameters and Analyses .................................................................. 8
Table 5. Shapiro–Wilk Test Results (Statistics online, 2022) ........................................................................ 9
Table 6. Normality Test for Group 1 and Group 2 ...................................................................................... 10
Table 7. Description of Parameters (Goodman, 2022) ............................................................................... 12

2
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

List of Acronyms
AV: Automated Vehicle

H0: Null Hypothesis

H1: Alternative Hypothesis

N/A: Not Applicable

P#: Person Sequence Number

POP: Population

Q-Q: Quantiles

SA: Sample

SAE: Society of Automotive Engineers

SP: Steering Position

T#: Test Sequence Number

ToC: Transition of Control

α: Significance level

** Significant action is used to refer to any action that will lead to a change from autonomous mode to
standard driving mode.

3
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

1. Introduction
As with any emerging technology, AVs bring forth a range of concerns and obstacles in the realm of
transportation. It's crucial to thoughtfully acknowledge and tackle these issues as AVs continue to be
developed and integrated into our transportation systems. Some of the significant risks and hurdles linked
to AVs include:

Safety Concerns: Although AVs hold the potential to enhance road safety by reducing accidents caused
by human errors, they are not immune to technical malfunctions or unexpected situations. Ensuring the
safety of AVs and their capability to handle diverse driving conditions and scenarios presents a major
challenge.

Cybersecurity: AVs heavily rely on intricate software and communication systems, making them
susceptible to cybersecurity threats like hacking. Such threats have the potential to compromise vehicle
safety and the privacy of data.

Regulatory Complexity: Crafting appropriate regulations and standards for AVs is a complex endeavor.
Regulators must strike a balance between the need for safety and innovation while facilitating a seamless
transition from traditional vehicles to autonomous ones.

Ethical Dilemmas: AVs may encounter ethical decisions, such as choosing between safeguarding vehicle
occupants or minimizing harm to pedestrians. Resolving these ethical dilemmas in alignment with societal
values is a formidable challenge.

Infrastructure Adaptation: AVs may necessitate changes to existing road infrastructure, including
enhancements in signage, road markings, and communication infrastructure. Ensuring the compatibility
of roads with AV technology can be both expensive and time-consuming.

Data Privacy: AVs collect copious amounts of data for their operation. Safeguarding the privacy of this
data, which includes location information and driving habits, is a concern for both individuals and
regulators.

Building Consumer Trust: Establishing public trust in AV technology is paramount for its widespread
adoption. High-profile accidents or cybersecurity breaches could undermine confidence in autonomous
vehicles, potentially slowing their adoption.

Economic Disruption: The widespread adoption of AVs has the potential to disrupt various industries,
such as trucking, taxi services, and insurance, which could lead to job displacement and economic
challenges in these sectors.

Technical Complexity: Developing and maintaining the advanced technology needed for AVs poses
considerable challenges. Ensuring that these vehicles can handle various weather conditions, road types,
and traffic situations is an ongoing effort.

Mixed Traffic Environments: AVs will coexist with human-driven vehicles for an extended period.
Managing interactions between autonomous and traditional vehicles presents unique challenges, given
the unpredictability of human drivers.

Efforts are actively underway to address these risks and challenges through research, testing, regulatory
measures, and industry standards. Collaborative endeavors involving governments, private-sector
stakeholders, and academic institutions are essential in navigating the intricate landscape of AV
integration, all with the aim of ensuring that this technology positively contributes to transportation
systems while mitigating potential risks.

4
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Connected and Autonomous Vehicles (AVs) have the potential to influence traffic demand in various ways,
and the net effect will depend on a range of factors and how they are managed. Here are some ways AVs
could impact traffic demand:

Efficiency Improvements: AVs can be programmed to drive more efficiently, which may reduce traffic
congestion. They can maintain consistent speeds, follow optimal routes, and react quickly to changing
traffic conditions. This efficiency could encourage more people to use AVs and reduce traffic congestion.

Ridesharing and Carpooling: AVs could facilitate ridesharing and carpooling more effectively. With
autonomous vehicles, it may become more convenient and cost-effective for people to share rides,
reducing the number of individual vehicles on the road.

Reduced Need for Parking: If AVs can drop passengers off and pick them up without parking, it may
decrease the demand for parking spaces in congested areas, potentially reducing traffic related to parking
searches.

Access for Those Unable to Drive: AVs could provide mobility solutions for people who are unable to drive
due to age or disabilities. This could lead to an increase in the number of people using transportation
services, potentially increasing traffic.

Induced Demand: There's a concept in transportation known as "induced demand," which suggests that
when road capacity improves or travel becomes more convenient (e.g., due to reduced congestion with
AVs), it can lead to an increase in overall travel as people make more trips or choose longer commutes.
This phenomenon could potentially offset some of the congestion-reducing benefits of AVs.

Urban Planning and Policies: The impact of AVs on traffic demand will also depend on how cities and
governments plan for and regulate their use. For example, policies that encourage shared autonomous
rides or congestion pricing could help manage traffic demand.

Economic Factors: The cost of using AVs compared to traditional vehicles, public transportation, or other
modes of transportation will influence their adoption and, consequently, their impact on traffic demand.

Thanks to the development of technology, it is obvious that AVs are going to have a significant role
regardless of their automation level for future transport needs. It should not be forgotten that an
innovation which will cause a paradigm shift will also bring risks.

5
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

AVs have 6 different categories in terms of their automation:

Figure 1. SAE Levels (SAE Levels of Driving Automation™ Refined for Clarity and International Audience, 2021)

As can be seen above, Level 3 may be seen as a transition level of automation between the levels where
people are needed as drivers and the levels where people are just passengers. In this way, when it comes
to a hybrid transition level, it is essential that machines communicate with people and work in harmony,
as well as people take into account external stimuli to take control at necessary moments. In this report,
after the level 3 AV gives a warning to the drivers about an obstacle on the road in the simulation
environment, the reaction times of the drivers according to the distracted and alerted situations are
examined.

2. Brief Information About Simulation Process

The simulator cockpit included almost everything needed to have a real driving experience such as seat,
seatbelt, gear pedals, handbrake, gear, steering wheel, etc. In this study, 8 participants have been joined
to test the ToC times.
Table 1. Information About Participants (Goodman, 2022)

6
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

The SAE level 3 AV car-based, simulation included 2 different scenarios:

1) Drivers were under alerted condition and weather and road conditions were good.

2) Drivers were distracted using phones, coffee, etc. and weather and road conditions were
adverse.

AV was driven automatically for a while, then around 66th-second participants were warned due to the
obstacle on the road was for taking the control. Then, if the drivers took the following actions, the control
was transferred to the drivers:

• Received input of brake or accelerator (gas) greater than 10%

• Changes of steering position by more than +/-2 degrees

If there was no action, the car would crash the obstacle at 86th seconds. Then, the data was gathered for
the analysis of ToC purposes.

3. Data Analysis
The process from the raw data to wisdom should start with understanding and preparing the raw data.

3.1. Preparation of the Data

The raw data set includes the parameters below:

• Timestamp*, Pathnum, Dfrint, Velocity*, Steer*, Gas*, Acc*, Brake*, LatAcc, LatPos, Indicator,
TTC, Steerspeed
o * → Parameters used

Details of the parameters can be found in the Appendix Table7. For this study just 6 parameters (*) were
used. Since the initial values of some parameters could not be obtained as desired due to a bug, the initial
values of these parameters were changed to the values they should normally be.

Subsequently, warning time (first time that the timestamp reaches 66th seconds) and the significant
actions (minimum requirement for AV to cancel autonomous mode) and their times have been
investigated.
Table 2. Test 1 and Test 2 Significant Times and Parameters

As can be understood from Table 1. Person 5 has no driving license and experiences. During the
experiment, she did unexpected behaviours that can be understood by looking at the reaction times she

7
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

gave in Test1 and Test2 during experiments. Also, she showed significant break action just before the
possible accident time (86th seconds). Therefore participant 5 was excluded from the data set for the
more stable analyses. The other results showed that reaction times are more for the distracted scenarios
as expected.
Table 3. Test 1 and Test 2 Significant Times and Parameters After Extraction

3.2. Detailed Analysis

After data preparation, further detailed analyses were made for understanding the difference between
ToC timings between the 2 scenarios. Graphs (Figure 4,5,6,7,8) that can be found in Appendix show that
all participants changed at least 1 evaluated parameter (except braking) within 5 seconds after the
stimulus. It indicates that everyone reacted to the warning somehow in 5 seconds.

As mentioned above for transferring the control to the drivers, there are 3 different conditions. Since the
effects of the distraction on the ToC timing is evaluated, further analyses were made by focusing on ToC
timings. Steering & gas positions were investigated deeply because as can be understood from Table 4.
AV mode had been stopped only because of these 2 parameters' change.
Table 4. Test 1 and Test 2 Significant Parameters and Analyses

In all cases, except Person 4 test 2, the vehicles were exited the autonomous mode with the change of
the steering angle. It is expected that the reaction times given in distracted situations will be longer than
in alerted situations. For this reason, statistical tests were used to prove this hypothesis.

8
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Initially, data (time differences) was tested via Shapiro–Wilk test to understand whether it is normally
distributed or not. Since n≤50, Shapiro-Wilk tables were used to calculate the p-value (Statistics online,
2022).
Table 5. Shapiro–Wilk Test Results (Statistics online, 2022)

Since p-value > α (0.05), H0 was accepted. It was assumed that the data is normally distributed. In other
words, the difference between the data sample and the normal distribution was not big enough to be
statistically significant. A non-significance result could not prove that H0 is correct, only that the null
assumption could not be rejected. Briefly, The Shapiro-Wilk test did not show a significant departure from
the normality, W(14) = 0.902, p = 0.122 and data were normally distributed (Statistics online, 2022).

Figure 2. Q-Q Plot (Statistics online, 2022)

After realizing that the distribution is normal, the next step is to investigate whether there is a statistically
significant difference in the exit times from the autonomous mode between alerted and distracted
scenarios. Before starting the test, the null hypothesis was accepted that alerted and distracted situations
do not have a statistically significant effect on the ToC timings. Since H0 is non-directional, distribution is
normal, sample size<30 and variances are not equal (0.1207≠0.9176) Welch’s t-test (two-tailed) was
applied to test H0 (Thorpe, 2021).

When entering raw data, the tool ran the Shapiro-Wilk normality test separately and computed outliers,
as part of the test calculation, and generated the R code for data (Statistics online, 2022).

9
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Table 6. Normality Test for Group 1 and Group 2

The results of the Welch’s t-test (two-tailed) for the data were explained below (Statistics online, 2022):

• Normality assumption was checked based on the Shapiro-Wilk Test.(α=0.05)

• P-value=0.00277224, (p(x≤T) = 0.00138612). This means that the chance of type1 error (rejecting
a correct H0) was small: 0.002772 (0.28%). The smaller the p-value the more it supports H1.
• Since p-value < α (0.05), H0 was rejected with 95% confidence level. The difference between the
average of the Test 1 and Test 2 was big enough to be statistically significant.
• The test statistic T=-4.358637, was not in the 95% critical value accepted range: [-2.3300: 2.3300].
• x1-x2=-1.68, was not in the 95% accepted range: [-0.9000 : 0.1761]. The statistic S'=0.385.
• The observed standardized effect size was large (2.33). That indicated that the magnitude of the
difference between the average and average was large.
• Although the prior test power was low (0.06250), the H0 was rejected.

Figure 3. T Distribution (Statistics online, 2022)

To sum up, the test showed that there is a statistically significant effect on the ToC timings between
alerted and distracted situations with 95% confidence.

10
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

4. Comments
Comments about the current study and future studies can be found below:

• Since there is no death risk in the simulation environment, people may be more relaxed than in
the real world. It should not be overlooked during studies that it may affect the results.
• The fact that both the weather conditions and the alerted or distracted situations were different
for the scenarios caused 2 different conditions among the scenarios. Instead, a better comparison
can be made if only the alerted or distracted conditions are changed, and other variables are kept
constant. Because people generally are more careful under adverse weather.
• The backgrounds of the participants should have been known better so the possibility of missing
data (i.e., Person 5) would be less.
• People should have been more familiar with the test environment for more accurate results.
• 6 second warning time could have been longer for adverse weather considering the safety.
However, it seems enough since everyone reacted somehow in 5 seconds.
• As can be understood from Figure 4, people drive slowly under adverse weather after they got
control.
• The bigger the sample size, the more accurate studies. Therefore, tests should be done with more
participants for better outcomes.
• Simulation environment may be developed by motion sensors that lead to improve the feeling of
real driving. So, drivers may feel the road surface condition etc. better.
• Lightning and acoustics of the simulation environment may be improved depending on the
budget.
• People with simulation sickness and travel sickness are not always the same people so it should
be considered (Golightly, 2021).
• There is a shortage of research demonstrating the validity of simulations except for aviation, so
the process should be carried out by paying attention to this issue (Golightly, 2021).

5. Conclusion
The study of evaluating the ToC timings of drivers in alerted and distracted situations for level 3 AVs have
resulted in that with 95% confidence, alerted and distracted situations have a statistically significant effect
on ToC timings. Based on the study for level 3 AVs drivers must be alert for safety reasons and
manufacturers must be transparent about AVs capability despite sale apprehensions.
While AVs offer many potential benefits, they also come with significant risks and challenges. Therefore,
it's important to recognize that AVs alone may not be a complete solution to all traffic problems.

In summary, while AVs have the potential to offer significant advantages in terms of safety, efficiency, and
convenience, they are not a golden bullet for solving all traffic problems. Addressing traffic congestion
and improving transportation systems will require a holistic approach that combines AVs with other
solutions, including investments in public transportation, urban planning, and the development of smart
and sustainable cities. Moreover, addressing the risks and challenges associated with AVs will be crucial
to realizing their full potential while minimizing potential negative impacts.

11
 Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

6. References

• Golightly, D. (2021) Human Factors for Intelligent Transport Systems Lecture Notes.
• Goodman, D. (2022) CEG8431 Technologies for Future Mobility Lecture Notes.
• SAE Levels of Driving Automation™ Refined for Clarity and International Audience (2021) [Online].
2021. Available at: https://www.sae.org/blog/sae-j3016-update. (Accessed: 23 February 2022).
• Statistics online (2022) [Online]. 2022. Available at: https://www.statskingdom.com/. (Accessed:
25 February 2022).
• Thorpe, N. (2021) Transport Research Methods Lecture Notes.

7. Appendix
Table 7. Description of Parameters (Goodman, 2022)

12
Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Figure 4. Velocity/Time Graphs

Figure 5. Acceleration/Time Graphs

Figure 6. Gas Position/Time Graphs

Figure 7. Brake Position/Time Graphs

13
Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Figure 8. Steering Position/Time Graphs

Figure 9. Person 1 Steering Position and Gas Position Comparisons

Figure 10. Person 2 Steering Position and Gas Position Comparisons

14
Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Figure 11. Person 3 Steering Position and Gas Position Comparisons

Figure 12. Person 4 Steering Position and Gas Position Comparisons

Figure 13. Person 6 Steering Position and Gas Position Comparisons

15
Questioning Based on Driving Simulator Data Analysis: Are Autonomous Vehicles Ready to Solve Urban
Mobility Problems?

Figure 14. Person 7 Steering Position and Gas Position Comparisons

Figure 15. Person 8 Steering Position and Gas Position Comparisons

16