Mobile Robots & Machine Vision

An Introductory Idea

Dr. Kaushik Das Sharma

Applied Physics
University of Calcutta
Automation and Robotics in Intelligent Environments
 Control of the physical  Personal service robots
environment  House cleaning
 Automated blinds  Lawn mowing
 Thermostats and heating ducts  Assistance to the elderly and
 Automatic doors handicapped
 Automatic room partitioning  Office assistants
 Security services

11/1/2022
What is robot?
• There’s no precise definition,
but by general notion,
• Robots — machines with sensing,
intelligence and mobility.

• Robota (Czech) — A worker of forced labor

*From Czech playwright Karel Capek's 1921 play “R.U.R”
(“Rossum's Universal Robots”)
11/1/2022
What is robot?

Robot Institute of America: “A robot is a reprogrammable,

multifunctional manipulator designed to move material, parts,
tools, or specialized devices through variable programmed
motions for the performance of a variety of tasks.”

Alternatively: “A robot is a one-armed, blind

idiot with limited memory and which cannot
speak, see, or hear.”
MIT’s KISMET: A robot which exhibits expressions, e.g.,
happy, sad, surprise, disgust.
11/1/2022
What is robot?

 To be qualified as a robot, a machine has to be able to:

Sensing and perception: get information from its
surroundings
Carry out different tasks: Locomotion or manipulation, do
something physical–such as move or manipulate objects
Re-programmable: can do different things
Function autonomously and interact with human beings
11/1/2022
Robotics – in course of time
• Mechanical Automata
• Ancient Greece & Egypt • Motor driven Robots
• Water powered for ceremonies • 1928: First motor driven
• 14th – 19th century Europe automata
• Clockwork driven for • 1961: Unimate
entertainment • First industrial robot
• 1967: Shakey
• Autonomous mobile
research robot
• 1969: Stanford Arm
• Dextrous, electric motor
driven robot arm
11/1/2022
Autonomous Robots

 The control of autonomous robots involves a number of subtasks

 Understanding and modeling of the mechanism
 Kinematics, Dynamics, and Odometry
 Reliable control of the actuators
 Closed-loop control
 Generation of task-specific motions
 Path planning
 Integration of sensors
 Selection and interfacing of various types of sensors
 Coping with noise and uncertainty
 Filtering of sensor noise and actuator uncertainty
 Creation of flexible control policies
 Control has to deal with new situations

11/1/2022
 Different Types

• Wheeled mobile robots

• Legged robots
• Aerial robots
• Underwater robots
• Humanoid robots

11/1/2022
Wheeled Mobile Robots (WMR)

Khepera
Yamabico Magellan Pro Koala

11/1/2022 Sojourner ATRV – 2 Hilare – 2 Bis

 Robot Steering Equations and Movement
Strategy
x  V cos   u x  V  u 2
x  u y2  VR  Right wheel velocity
 uy  VL  Left wheel velocity
y  V sin   u y    tan 1

  ux  V  Linear velocity

   g   old   new

 If the absolute value of change in the orientation is less than 10°, then VL = V and VR = V,
the robot will move straight.
 If the change in the orientation is positive and greater than 10°, but less than 90°, then VL
= 0 and VR = V and the robot will rotate left.
 If the change in the orientation is negative and magnitudically greater than 10°, but less
than 90°, then VL = V and VR = 0 and the robot will rotate right.
 The magnitude of ux and uy are so generated from the controller, that the linear velocity V
and corresponding VL and VR will drive the robot to track the generated path by the vision
11/1/2022sensor.
Wheeled Mobile Robots

 Combination of various physical (hardware) and

computational (software) components

 A collection of subsystems:
 Locomotion: how the robot moves through its environment
 Sensing: how the robot measures properties of itself and its
environment
 Control: how the robot generate physical actions
 Reasoning: how the robot maps measurements into actions
 Communication: how the robots communicate with each other or with
an outside operator
11/1/2022
Non-holonomic constraint

So what does that mean?

Robot can move in some directions (forwards and backwards), but not others
(side to side).
The robot can instantly
move forward and back,
but can not move to the
right or left without the Parallel parking,
wheels slipping. Series of maneuvers
Idealized Rolling Wheel

• Assumptions:
• No slip occurs in the
orthogonal direction of
rolling (non-slipping).
• No translation slip
occurs between the
wheel and the floor
(pure rolling).

Non-slipping and pure rolling

 Wheel Types

Centered orientable
wheel
wheel

ntered
able wheel Swedish wheel: omni-
or wheel) directional

11/1/2022
 Examples of WMR

 Smooth motion
Bi-wheel type robot  Risk of slipping
 Some times use roller-ball to make balance

Exact straight motion

Caterpillar type robot Robust to slipping
Inexact modeling of turning

Free motion
Omnidirectional robot Complex structure
Weakness of the frame
11/1/2022
Mobile Robot navigation

Navigation in Static Navigation in Dynamic

Environments Environments
 Classical theoretical approaches
• Dijkstra’s Algorithm Dynamic Potential Field Method
• A* Algorithm Sensor-based Path Planning
 Potential Field Method Method
 Probabilistic Roadmap Method
Mobile Robot navigation
 Map based environment
 Map less environment

 Most popular navigation technique – SLAM

SLAM: Simultaneous Localization and Mapping

Exit
SLAM is a technique used to build up a map
within an unknown environment or a
known environment while at the same time
keeping track of the current location of the
robot
Given:
 The robot’s controls

 Observations of nearby features

Estimate:
 Map of features

| Adapted from "American Maze"  Path of the robot

by Dale Wilkins |
SLAM: Simultaneous Localization and Mapping
o The problem has 2 stages
• Mapping
• Localization
o The paradox:
• In order to build a map, we must know our position
• To determine our position, we need a map!
o SLAM is like the chicken-egg problem
o Solution is to alternate between the two steps.
SLAM: Simultaneous Localization and Mapping
• SLAM solution techniques:
• Kalman Filter (KF),

• Extended Kalman Filter (EKF),

• Particle Filters (PF),

• Scale Invariant Feature Transform (SIFT).

SLAM: Simultaneous Localization and Mapping

 Initial State and Uncertainty

t=0  Using Range Measurements
SLAM: Simultaneous Localization and Mapping

 Predict Robot Pose and

Uncertainty at time t=1
t=1
SLAM: Simultaneous Localization and Mapping

Correct pose and pose uncertainty

t=1 Estimate new feature uncertainties
SLAM: Simultaneous Localization and Mapping

Predict pose and uncertainty of pose at time

t=2 t=2
Predict feature measurements and their
uncertainties
SLAM: Simultaneous Localization and Mapping

 Correct pose and mapped features

t=2  Update uncertainties for mapped features
 Estimate uncertainty of new features
Mobile Robot Locomotion
• Differential Drive
• two driving wheels (plus roller-ball for balance)
• simplest drive mechanism
• sensitive to the relative velocity of the two wheels (small error result in different
trajectories, not just speed)
• Tricycle
• Steering wheel with two rear wheels
• cannot turn 90º
• limited radius of curvature
• Synchronous Drive
• Omni-directional
• Car Drive (Ackerman Steering)
11/1/2022
Mobile Robot Navigation
 Locomotion — the process of causing an robot to move.
 In order to produce motion, forces must be applied to the robot
 Motor output, payload
 Dynamics – study of motion in which these forces are
modeled
 Deals with the relationship between force and motions.
 Kinematics – study of the mathematics of motion without
considering the forces that affect the motion.
 Deals with the geometric relationships that govern the system
 Deals with the relationship between control parameters and the
behavior of a system.

11/1/2022
Sensor-Driven Robot Control
• To accurately achieve a task in real environment, a robot has to be
able to react dynamically to changes on its surrounding
• Robots need sensors to perceive the environment
• Most robots use a set of different sensors
• Different sensors serve different purposes
• Information from sensors has to be integrated into the control of the robot

11/1/2022
Types of Sensors
Odometry
Laser Ranging and Detection (LIDAR)
Acoustic (sonar, ultrasonic)
Radar
GPS
Gyroscopes, Accelerometers (Inertial Navigation),

Vision (monocular,
stereo)
Problems in sensor based SLAM

 Synchronization of sensory data – Data fusion algorithm

 Sensory data processing at the same time stamp – Out-Of-Sequence
Measurements (OOSM)
 OOSM problem leads to imprecise navigation
A Controller Based Approach of Mobile Robot
Navigation
• Tracking Controller Based Navigation Employing Vision Sensor
Practical Design Procedure
 First, the robot is identified and the robot parameters are estimated, if required.

 Secondly, the control scheme for the robot is developed in simulation.

 During simulation, controllers according to different control strategies are trained.

 Finally, these trained controllers are then utilized to perform the real-life implementation.
Machine Vision - Introduction
Machine Vision
• Elements of a Machine Vision System
• Lens-camera model
• 2D versus 3D machine vision
• Image segmentation – pixel classification
• Thresholding
• Connected component labeling
• Chain and crack coding for boundary representations
• Contour tracking / border following
• Object recognition
• Blob analysis, generalized moments, compactness
• Evaluation of form parameters from chain and crack codes
MACHINE VISION SYSTEM
LENS-CAMERA MODEL
 HOW CAN WE RECOVER THE
“DEPTH” INFORMATION?

•Stereoscopic approach: Identify the same

point in two different views of the object and
apply triangulation.
•Employ structured lighting.
•If the form (i.e., size) of the object is known,
its position and orientation can be determined
from a single perspective view.
•Employ an additional range sensor
(ultrasonic, optical).
3D MACHINE VISION SYSTEM
XY Table

Laser Projector
Digital Camera

Field of View

Plane of Laser Light Granite Surface Plate

P(x,y,z)
Stereo Vision
Stereo Vision
2D vs. 3D machine vision
• No depth information in 2D vision

• Triangulation problem in 3D vision

• Disparity Registration in 3D vision

Neural Bases of Disparity Registration

X
Neural Bases of Disparity Registration

X
Neural Bases of Disparity Registration

X zero disparity
Neural Bases of Disparity Registration

X uncrossed disparity
Neural Bases of Disparity Registration

X uncrossed disparity
Neural Bases of Disparity Registration

X
Neural Bases of Disparity Registration

X “crossed disparity
Neural Bases of Disparity Registration

X “crossed disparity
2D or Monocular Vision

h  h   xl 
  tan  
1
  tan 
1
   tan 1  
 yb   yb  yl  y
 b  y l 

 u    2v 
y  h tan (   )    (   ) x  y tan    
 2  S y    Sx 
Image Processing and Path Planning
Techniques
Image Processing
Planning Techniques

120 2

100
1.5

Y axis (in M eters)

Y axis (in Pixels)

80

60 1

40
0.5
20

0 0
-40 -30 -20 -10 0 10 20 30 40 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
X axis (in Pixels) X axis (in Meters)

Image Plane Path Real Plane Path

Robot Steering Equations and
x  v cos   u x
y  v sin   u y
  

V  u 2
x  u y2 
 uy 
  tan 
1

 ux 
Calculation of left and right wheel velocities
A Simulation Result
End notes
• The best features of SLAM and Tracking Controller
Based Navigation can be combined to achieve more
robust and reliable robot navigation in both static and
dynamic environment.
• Accurate and efficient Sensor Data Fusion Algorithm
may be developed for sophisticated applications.
• The concept of Swarm Robotics can be exploited with
Vision Empowered Tracking Controller Based Approach.