Exploration of an Unknown

Environment using Deep Reinforcement

Learning
SUPERVISOR ASAD ALI – 46375
DR. ANAYAT ULLAH SARAH GUL – 47850
CO-SUPERVISOR
DR. IMRAN QURESHI
Department of Electrical Engineering
EXPLORATION OF AN UNKNOWN ENVIRONMENT 1
USING DEEP REINFORCEMENT LEARNING
 Outline
• Introduction
• Literature Review
• Design Methodology and Results
• Gantt Chart for Project Progress
• Conclusion
• Future Work
• References

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 2
USING DEEP REINFORCEMENT LEARNING
 Introduction
What is a ROBOT?
• A machine capable of carrying out a
series of actions
Types of robots
• Stationary Stationary robot
Image courtesy: www.seekpng.com
• Non-stationary

Wheeled robot Legged robot Flying robot

Image courtesy: stanleyinnovation.com Image courtesy: www.directindustry.com Image courtesy: www.tbyyf.com
Department of Electrical Engineering
EXPLORATION OF AN UNKNOWN ENVIRONMENT 3
USING DEEP REINFORCEMENT LEARNING
 Introduction
Exploration and navigation of unknown
environments
• The need for robots
• Rescue operations
• Covert operations Sending robots into lunar caves for exploration
• And more Image courtesy: www.universetoday.com

Rescue robot
Image courtesy: www.designnews.com
Department of Electrical Engineering
EXPLORATION OF AN UNKNOWN ENVIRONMENT 4
USING DEEP REINFORCEMENT LEARNING
 Problem, Solution, Objective
Problem
Conventional methods are computationally expensive, require more
time and memory due to which navigation becomes difficult in large
environments.
Solution
The robot should learn to navigate/explore the environment while
avoiding obstacles without a map.
Objective
To design and analyze the navigation/ exploration system of a mobile
robot in a map-less environment utilizing learning-based methods.

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 5
USING DEEP REINFORCEMENT LEARNING
 Learning Based Methods

Supervised Learning
Image courtesy: https://www.javatpoint.com

Reinforcement Learning
Unsupervised Learning Image courtesy: https://www.javatpoint.com
Image courtesy: https://www.javatpoint.com
EXPLORATION OF AN UNKNOWN ENVIRONMENT
Department of Electrical Engineering 6
USING DEEP REINFORCEMENT LEARNING
 Reinforcement Learning
The Agent-Environment interaction in RL [1].

Agent-environment interaction.
Image courtesy: [1]
Main goal is to maximize the reward.
∞

𝐺𝑡 = 𝑚𝑎𝑥 ෍ 𝛾 𝑘 𝑅𝑡+𝑘+1
𝑘=0

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 7
USING DEEP REINFORCEMENT LEARNING
 Q-Learning
The Q – value, Q(St,At)

• In this equation,
• 𝛾 is discount factor ↑ ↓ ← →
• 𝛼 is learning rate S1 0 -1 -2 1
• Q table works as policy.
• Is Q learning applicable for large S2 2 0 -1 0
state space? S3 0 0 3 -2
S4 -3 1 -1 0

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 8
USING DEEP REINFORCEMENT LEARNING
 Double DQN
Two neural networks
• Policy network is used
to take action Q(St, At;θ) argmaxa Q(St+1,At+1;θ’)

• Target Network future argmaxA Q(St, At;θ)

state’s Q-Value
St
(St, At) (St+1)

(St,At,Rt,St+1)

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 9
USING DEEP REINFORCEMENT LEARNING
 Literature Review
• Learning exploration policies for navigation
[2]

• COBRA: Data-Efficient Model-Based RL [2]

through Unsupervised Object Discovery and
Curiosity-Driven Exploration [3]

[3]

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 10
USING DEEP REINFORCEMENT LEARNING
 Literature Review
• Curiosity-driven Exploration by Self-supervised
Prediction [4]

• Curiosity-driven Exploration for Map less Navigation

with Deep Reinforcement Learning [5]

[5]

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 11
USING DEEP REINFORCEMENT LEARNING
 Methodology

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 12
USING DEEP REINFORCEMENT LEARNING
 Obstacle Free Navigation
• Double DQN algorithm
• Discrete state space (-90 to +90 degrees)
• Actions (Left, Right) discrete actions
• Reward Function

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 13
USING DEEP REINFORCEMENT LEARNING
 Obstacle Free Navigation(Cont.)

Fig: The result of Double DQN algorithm without obstacle

Department of Electrical Engineering
EXPLORATION OF AN UNKNOWN ENVIRONMENT 14
USING DEEP REINFORCEMENT LEARNING
 Obstacle Free Navigation(Cont.)

Reward vs Episodes Steps vs Episodes

Fig: The result of Double DQN in obstacle Fig: The result of Double DQN in obstacle free
free environment (Rewards) environment( Steps)

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 15
USING DEEP REINFORCEMENT LEARNING
 Navigation with Obstacles
• States are LIDAR scans (Continuous)
• Double DQN algorithm used to navigate
• Actions (Left, Right, Forward)
• Reward Function

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 16
USING DEEP REINFORCEMENT LEARNING
 Navigation with Obstacles(Cont.)

Fig: The result of Double DQN algorithm with LIDAR

Department of Electrical Engineering
EXPLORATION OF AN UNKNOWN ENVIRONMENT 17
USING DEEP REINFORCEMENT LEARNING
 Navigation with Obstacles

Reward vs Episodes Steps vs Episodes

Fig: The result of Double DQN Fig: The result of Double DQN
algorithm with LIDAR ( Rewards) algorithm with LIDAR ( Steps)

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 18
USING DEEP REINFORCEMENT LEARNING
 Curiosity Driven Exploration
• Double DQN algorithm used to explore
• States are LIDAR scans (Continuous)
• Actions (Left, Right, Forward)
• Reward Function

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 19
USING DEEP REINFORCEMENT LEARNING
 Curiosity Driven Exploration(Cont.)

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 20
USING DEEP REINFORCEMENT LEARNING
 Curiosity Driven Exploration(Cont.)
Training Map Testing Map

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 21
USING DEEP REINFORCEMENT LEARNING
 Curiosity Driven Exploration(Cont.)
Training Testing

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 22
USING DEEP REINFORCEMENT LEARNING
 Gantt chart For Project Progress
22-Aug-2021 11-Oct-2021 30-Nov-2021 19-Jan-2022 10-Mar-2022 29-Apr-2022 18-Jun-2022 7-Aug-2022

Learning ML, DL, RL 11-Oct-21 96

Literature review 15-Oct-21 92

Hello world problem solution 15-Jan-22 31

Exploration problem modeling 15-Feb-22 28

Exploration problem formulation 15-Mar-22 30

Algorithm selection 14-Apr-22 30

Simulation 14-May-22 30

Improvement 13-Jun-22 34

Finalizing project 17-Jul-22 14

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 23
USING DEEP REINFORCEMENT LEARNING
 Conclusion
• Implemented DDQN algorithm to navigate in obstacle free environment
• Implemented DDQN algorithm to navigate with an environment with . .
.obstacles
• Implemented DDQN algorithm to explore with an environment with
. bstacles using Intrinsic Curiosity Reward (ICM).

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 24
USING DEEP REINFORCEMENT LEARNING
 Future Work
• Improvement in ICM reward function
• Implementation on hardware

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 25
USING DEEP REINFORCEMENT LEARNING
 References
[1] R. Sutton and A. Barto, “The Reinforcement Learning Problem” in
Reinforcement Learning: An Introduction. MIT Press, 1998.
[2] T. Chen, S. Gupta and A. Gupta, "Learning Exploration Policies for
Navigation," 2019.

[3] Watters, Nicholas, et al. “Cobra: Data-efficient model-based RL through

unsupervised object discovery and curiosity-driven exploration.”arXiv
preprint arXiv:1905.09275 2019.
[4] D. Pathak, P. Agrawal, A. A. Efros and T. Darrel, "Curiosity-driven
exploration by self- supervised prediction," International conference on
machine learning, pp. 2778-2787, 2017.
[5] O. Zhelo, J. Zhang, L. Tai, M. Liu and W. Burgard, "Curiosity-driven
Exploration for Mapless Navigation," arXiv preprint arXiv:1804.00456,
2018.

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 26
USING DEEP REINFORCEMENT LEARNING
 Thank You!
Questions?

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 27
USING DEEP REINFORCEMENT LEARNING
 Results
Exploration Index

• Graph can be used to stop the

exploration.
• An indication of when to stop
training

Department of Electrical Engineering

EXPLORATION OF AN UNKNOWN ENVIRONMENT 28
USING DEEP REINFORCEMENT LEARNING