3/20/2022

Self-Driving
Vehicles
MECH-8000-8/9
Week #10

Date Subject, activity, assignment, etc.

20-Jan-22 Introduction to Self Driving Vehicles

27-Jan-22 Self Driving Vehicles Sensing Requirements.

3-Feb-22 Vehicle Dynamics – Longitudinal/Lateral Vehicle models

10-Feb-22 Vehicle Actuator Models/Basics of Control Development

Course 17-Feb-22 Class Cancelled (Winter Storm)

Schedule
24-Feb-22 Reading week, No Lecture

3-Mar-22 Vehicle Longitudinal/Lateral Controls

10-Mar-22 Introduction to Python for Vehicle modeling and Control

17-Mar-22 Midterm Exam

24-Mar-22 Introduction to Computer Vision – Python/OpenCV Basics

31-Mar-22 Introduction to Machine Learning – Python/SciKit Basics

7-Apr-22 Artificial Neural Networks

14-Apr-22 Object Detection and Classification

21-Apr-22 Final Project Submission/Team Presentation

2

1
 3/20/2022

Introduction to
Computer
Vision

 Introduction to Computer Vision

 OpenCV basics
 Filters and Transformations

What is Computer Vision

Science that allows computers to understand images and video and
determine what the computer “sees” or “recognizes”.

2
 3/20/2022

What is Computer Vision (Cont.)

Aims to emulate biological vision functionality.
How can we achieve same functionality with a computer system?

What is Computer Vision (Cont.)

Human Vision:
• light passes through the cornea (the clear front layer of the eye). The
cornea is shaped like a dome and bends light to help the eye focus.
• Some of this light enters the eye through an opening called the pupil, The
iris (the colored part of the eye) controls how much light the pupil lets in.
• light passes through the lens (a clear inner part of the eye). The lens works
together with the cornea to focus light correctly on the retina.

3
 3/20/2022

What is Computer Vision (Cont.)

Human Vision:
• When light hits the retina (a light-sensitive layer of tissue at the back of the
eye), special cells called photoreceptors turn the light into electrical
signals.
• These electrical signals travel from the retina through the optic nerve to
the brain. Then the brain turns the signals into the images you see.

What is Computer Vision (Cont.)

Computer Vision:
• Using similar light mechanics, Camera collects light from objects.
• Collected light is projected on the digital sensor, where it is converted to
an electrical signal.

4
 3/20/2022

What is Computer Vision (Cont.)

Computer Vision:
• The Role of computer vision is to implement algorithms on the captured
image to extract useful information for the required task.

Challenges
• humans, can easily classify images because of our outstanding
generalization capability.
• Generalization: the act or process whereby a learned response is made
to a stimulus similar to but not identical with the conditioned stimulus.

10

10

5
 3/20/2022

Challenges (Cont.)
• Achieving the same task with computers is very challenging.
• Numerical representation of different images will be very different, a
simple numerical compare wouldn’t work.

11

11

Challenges (Cont.)
• Viewpoints,
• Camera Limitation,
• Lighting,
• Scaling,
• Object Variation,

12

12

6
 3/20/2022

Challenges (Cont.)
• Viewpoints,
• Camera Limitation,
• Lighting,
• Scaling,
• Object Variation,

13

13

Challenges (Cont.)
• Viewpoints,
• Camera Limitation,
• Lighting,
• Scaling,
• Object Variation,

14

14

7
 3/20/2022

Challenges (Cont.)
• Viewpoints,
• Camera Limitation,
• Lighting,
• Scaling,
• Object Variation,

15

15

Challenges (Cont.)
• Viewpoints,
• Camera Limitation,
• Lighting,
• Scaling,
• Object Variation,

16

16

8
 3/20/2022

Digital Image Storage/Representation

Grayscale Images
• Image is stored using finite number of points, each point represents a
pixel.
• A pixel has a value from 0 to 255, where 0 represent the color black (0
light), and 255 represents the color white (maximum light).

17

17

Digital Image Storage/Representation (Cont.)

Color Images
• Each pixel is represented using three values,
Red, Green, Blue (RGB Images)(BGR in Python)
• Any color can be created from these three colors.
• Storage size is three times as much compared to
grayscale.

18

18

9
 3/20/2022

Digital Image Storage/Representation (Cont.)

Color Vs. Greyscale Images
• Color images are richer in data.
• Require more computation resources to process.
• Conversion from Color to Grayscale could be
done to simplify computer vision task.
• HSV is another alternate representation for color.

19

19

Computer Vision Tasks

The classical problem in computer vision, image processing, and machine
vision is that of determining whether or not the image data contains some
specific object, feature, or activity.
• Object recognition (also called object classification) – one or several pre-
specified or learned objects or object classes can be recognized, usually
together with their 2D positions in the image or 3D poses in the scene
• Identification – an individual instance of an object is recognized.
Examples include identification of a specific person's face or fingerprint,
identification of handwritten digits, or identification of a specific vehicle.
• Detection – the image data are scanned for a specific condition. Such as
lane detection for a self driving vehicle/lane centering drive assist.

20

20

10
 3/20/2022

Computer Vision Tasks (Cont.)

The classical problem in computer vision, image processing, and machine
vision is that of determining whether or not the image data contains some
specific object, feature, or activity.
• Algorithms for such tasks can vary from very simple to a more
complicated artificial intelligence based approach.

• Lane Markings are • Edge detection

White algorithmics,
• Grayscale vs Color • Filtering and
images transformation.
• Find pixels with • Feature Detection
white color.

21

21

Python OpenCV
OpenCV (Open Source Computer Vision Library) is a library of
programming functions mainly aimed at real-time computer vision.
The library is cross-platform and free for use under the open-source
Apache 2 License. Starting with 2011, OpenCV features GPU acceleration
for real-time operations.
To install:
pip install opencv-python

22

22

11
 3/20/2022

Color Selection
Find lane markings by only selecting pixels that are very close to white
color.

23

23

Color Selection (Cont.)

Find lane markings by only selecting pixels that are very close to white
color.

24

24

12
 3/20/2022

Color Selection (Cont.)

Challenges:
None white lane markings?
Different weather or lighting conditions?

25

25

Color Selection (Cont.)

Challenges:
None white lane markings?
Different weather or lighting conditions?

More features must be extracted from the

camera image using more sophisticated
computer vision techniques.

26

26

13
 3/20/2022

Convolutions
• In image processing, a kernel, convolution matrix, or mask is a small matrix
used for blurring, sharpening, embossing, edge detection, and more. This
is accomplished by doing a convolution between the kernel and an
image.
• Kernels are used in machine learning for feature extraction to select most
important pixels of an image.

27

27

Convolutions (Cont.)

Demo:
https://setosa.io/ev/image-kernels/ 28

28

14
 3/20/2022

Edge Detection
• Edge detection is a tool that identifies points in a digital image at which
the image brightness changes sharply or has discontinuities.
• Edge detection is important tool in computer vision especially for feature
extraction/detection.
• Uses Derivative to find edges

29

29

Edge Detection (Cont.)

SOBEL
• Sobel edge detector is a gradient-based method based on the first order
derivatives.
• It calculates the first derivatives of the image separately for X and Y axes.
• Sobel uses two 3X3 kernels which are convolved with original image to
calculate the derivatives.
+1 0 −1 +1 +2 +1
𝐺 = +2 0 −2 × 𝐴 and 𝐺 = 0 0 0 ×𝐴
+1 0 −1 −1 −2 −1

30

30

15
 3/20/2022

Edge Detection (Cont.)

SOBEL
+1 0 −1 +1 +2 +1
𝐺 = +2 0 −2 × 𝐴 and 𝐺 = 0 0 0 ×𝐴
+1 0 −1 −1 −2 −1
Calculate:

• Magnitude or “strength” of the edge: 𝐺 + 𝐺

• Approximate strength: 𝐺 + 𝐺

• The orientation of the edge: arctan

31

31

Edge Detection (Cont.)

LAPLACIAN:
• Unlike the Sobel edge detector, the Laplacian edge detector uses only
one kernel.
• It calculates second order derivatives in a single pass and detects zero
crossing.
• Second order derivatives are generally extremely sensitive to noise.
Laplacian Kernel:
0 −1 0
𝐺 = −1 4 −1
0 −1 0

32

32

16
 3/20/2022

Edge Detection (Cont.)

CANNY:
• The Canny edge detector is an edge detection operator that uses a
multi-stage algorithm to detect a wide range of edges in images.
• the Canny technique produces smoother edges due to the
implementation of Non-maxima suppression and thresholding.
• The downside of the Canny algorithm is that it is more complex and less
time-efficient than Sobel.

33

33

Edge Detection (Cont.)

CANNY:
• Smoothing: Smooth the image with a Gaussian filter with spread σ
• Gradient: Compute gradient magnitude and direction at each pixel of
the smoothed image.

34

34

17
 3/20/2022

Edge Detection (Cont.)

CANNY:
• Non-maximum suppression (thinning): Zero out all pixels that are not the
maximum along the direction of the gradient (look at 1 pixel on each
side).
• Thresholding: Threshold the gradient magnitude image such that strong
edges are kept, and noise is suppressed.

35

35

Edge Detection (Cont.)

CANNY:
• Non-maximum suppression (thinning): Zero out all pixels that are not the
maximum along the direction of the gradient (look at 1 pixel on each
side).
• Thresholding: Threshold the gradient magnitude image such that strong
edges are kept, and noise is suppressed.

36

36

18
 3/20/2022

Image Transformation
Image transformations are used to correct distortions and/or change
perspectives.
• Rotation
• Translation
• Resizing
• Perspective Transform
• Cropping
• Dilation and Erosion
• Region of Interest Masking
• Hough Transform 37

37

Image Transformation (Cont.)

Rotation
• OpenCV can perform rotations by using a rotation matrix M.
• A rotation matrix is a matrix used to perform rotation in Euclidean space.
• It rotates points in the xy-plane counterclockwise through an angle 𝜃
about the origin.

M_rotation = getRotationMatrix2D(center, angle, scale)

rotated_image = cv2.warpAffine(image, M_rotation, (width, height))

38

38

19
 3/20/2022

Image Transformation (Cont.)

Translation
• Translation is the shifting of object’s location in X and/or Y direction.
• OpenCV uses Translational matrix T as follows:
1 0 𝑇
𝑇= 0 1 𝑇

Translational_Matrix = np.float32([[1, 0, Tx], [0, 1, Ty]])

translated_image = cv2.warpAffine(image, Translational_Matrix, (width,
height))

39

39

Image Transformation (Cont.)

Resizing
• Resizing using OpenCV can be performed using cv2.resize()
• Preferable interpolation methods are cv.INTER_AREA for shrinking and
cv.INTER_CUBIC for zooming.
• By default, interpolation method used is cv.INTER_LINEAR for all resizing
purposes.

resized_image = cv2.resize(image, None, fx=3, fy=3, interpolation =

cv.INTER_CUBIC)

40

40

20
 3/20/2022

Image Transformation (Cont.)

Perspective Transform
• Image transformations are used to correct distortions and/or change
perspectives.
• Projective transformation (Non-affine) does not preserve parallelism,
length, and angle.

M = cv2.getPerspectiveTransform(Source_points, Destination_points)
warped = cv2.warpPerspective(image, M, (width, height)) 41

41

Image Transformation (Cont.)

Cropping
• Eliminate unnecessary part of an image.
• Image cropping can be performed as follows:
Image_cropped = image[h0:h1 , w0:w1]

42

42

21
 3/20/2022

Image Transformation (Cont.)

Dilation and Erosion
• Erosion means removing pixels at the boundaries of objects in an image
(removing pixels from the white background means increasing thickness
of the letters!)
• Dilation means adding extra pixels to the boundaries of objects in an
image (adding pixels to the white background so letters appear thinner!)

43

43

Image Transformation (Cont.)

Dilation and Erosion
• Image dilation and erosion can be performed as follows:
kernel = np.ones((8,8), np.uint8)
image_erosion = cv2.erode(image, kernel, iterations=1)
image_dilation = cv2.dilate(image, kernel, iterations=1)

44

44

22
 3/20/2022

Image Transformation (Cont.)

Region of Interest Masking
Mask a region of interest so we can focus our search efforts for a specific
feature(s).

45

45

Image Transformation (Cont.)

Hough Transform
• A line can be represented with one of the two forms below.
• ρ is the perpendicular distance from origin to the line.
• θ is the angle formed by this perpendicular line and horizontal axis.

46

46

23
 3/20/2022

Image Transformation (Cont.)

Hough Transform
• In image space, a line is plotted as x vs. y and modeled as:
𝑦 = 𝑚𝑥 + 𝑐 or x𝑐𝑜𝑠 𝜃 + 𝑦𝑠𝑖𝑛 𝜃 = 𝜌

47

47

Image Transformation (Cont.)

Hough Transform
• In parameter space (hough), a line is represented by a point in "m vs. c“.

48

48

24
 3/20/2022

Image Transformation (Cont.)

Hough Transform
• In parameter space (hough), a line is represented by a point in "m vs. c“.

49

49

Image Transformation (Cont.)

Hough Transform
• In parameter space (hough), a line is represented by a point in "m vs. c“.

50

50

25
 3/20/2022

Image Transformation (Cont.)

Hough Transform
• Create an accumulator array for the parameter
space.
• For each point in the image space, increment the
accumulator for the corresponding parameters
(lines that could pass through this point)
• Once all points are done, the line parameters are
located with the max no of votes.

51

51

Image Transformation (Cont.)

Hough Transform
To apply the Transform, first apply Canny edge detection pre-processing.
lines = cv2.HoughLines(image, 𝜌 accuracy, 𝜃 accuracy, threshold)
lines = cv2.HoughLines(image, 1, np.pi/180, 240)
𝜌 : The resolution of parameter 𝜌 in pixels.
𝜃 : The resolution of the parameter 𝜃 in radians.
threshold: minimum votes to be considered a line.
https://opencv24-python-
tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_houghlines/
py_houghlines.html 52

52

26
 3/20/2022

Reminders:
• Please divide in groups of 5-6, send one email
per group to instructor including group
members Full Name, UWindsor Email, and
Student ID.
• Assignment 3 posted online, please complete
and submit by deadline.
Thank You • Submit Student ID card to Blackboard.

53

53

THANK YOU

54

54

27