CHAPTER 2

BACKGROUND& RELATED WORK

2.1. LANE DETECTION OVERVIEW

A lot of research is completed in a computer vision lane recognition area for the
autonomous vehicle application, just as for the driver assistance system. Yet at the
same time, this exploration is going on in light of the fact that the white lane marks on
the dim street look straightforward. However, recognizing these lane marks on
different kinds of streets is troublesome. These different kinds of situations are
examined in the upcoming sections. Numerous autonomous organizations are giving
departure warning systems, lane-keeping system, speed guideline system, etc. Finding
the boundaries of the given path in the picture is additionally essential. By utilizing the
geometry of the floor, one can recognize the boundaries of the street. Lane discovery
algorithms can recognize the edges of the street and lane marking, and furthermore,
estimate the vehicle location in lane. Many different kinds of marks are available on
the road like dashed, strong, bott points are twofold and triple road markings.
Commonly street edges are unmarked, and they are secured with grass or with the rock
banks.

In lane detection, various types of sensors are used for perception. The sensors used
for perception are LIDAR, vision based sensor, and other sensors. LIDAR sensor are
using the remote sensing technology. These sensors calculate the distance by
enlightening a goalby a laser, and analyzes the replicatedillumination. It helps resist
natural light such as intense light or shadow. It also report the 3D structure and the
replicated intensity of an object, and leads to compute the unevenness, bends, and
shoulders on the road. LIDAR derives the slope of the road and rovers pitch angle [51].

LIDAR need is a costly sensor as compared to a solitary camera. Thus, LIDAR used
in complex applications, such as, autonomous vehicles. GPS signal are also precise for
detecting the lane. Hence, there is a need of numerous error correction systemswith
massive computations. In the map based approach, the map frequently alter due to

13
weather, construction and a few other factors. Periodical map updation is needed to
make it useful for the real world.

Fisheye camera is compatible for multi marking detection caused by its extensive
capture range. On the other hand, results and discussion of the marking shapes and
make the length fitting pass complex [52]. The stereo vision system, permit access to
dense 3D data by employing two cameras to detect the road structures such as curbs.
As compared to LIDAR, the stereo vision has detecting deep range, but it is having
weak accuracy and reliability of stereo vision[53].

Geographic information system (GPS) is also used for providing the location and
environment information to help Lane marking detection. Predictive Lane detection
(PLD) is also used for vehicle localization on offline digital maps. PLD estimates the
road geometry that offers strong indications of limiting lane marking model
parameters [54].

When the input images are acquired from the sensors, they contain some noise, so
there is a need for image preprocessing; prepare the input image for image processing.
Preprocessing eliminate the noise, prunes the image parts, and recover image
information. The commonly used noise reduction methods is image binarization by
using the predefined threshold. But fix threshold is not enough for variable
illuminations in road condition so in such conditions, the adaptive threshold can be
used [55] [56]. To remove the pepper and salt noise, one can blur the image [57]. The
image resolution can be increased by performing color channel conversion. Color
space transformation to HSL and YCbCr can be performed [58]. High pass filters are
also used for casting sharp shadows [59].

preprocessing steps also involves pruning the image. Obstacles are detected, and the
area of the obstacle is omitted in the image [60]. Object detection is not reliable due to
the high false-positive rate. So,the ROI (region of interest) is the substitute for object
detection. ROI reduces image size [61]. ROI can be split in several ways, such as, use

14
already defined percent of the image from base to peak [62], use vanishing points and
vanishing line [63], divide the image vertically and horizontally[64][65].

IPM is also used for preprocessing. The pair oflane marking is parallel in a BEV
interplanetary. It makes computations convenient, and backgrounds like Sky and
roadside are not considered in inverse perspective mapping [66]. Many times some
road features are added or block, but they appear in a series of continuous frames.
Image recovery techniques are used to recover the missing features[67]. Temporal
blurring is used torecoverthe absent/brokenstreet markings [57].

There are different types of features available in the road environment image. Lane
marking features are extracted in two steps. The initialstep is feature selection, and
the nextstep is feature extraction. Feature selection determines a suitable feature that
is lying on the front to detect, and feature extraction develops methods for discovering
the pixels of the features. Most commonly used lane marking are shape , color, texture,
and color gradient.

A color extraction can be done by employing 3 multivariate Gaussian distribution, for

white, red and yellow lane marking that represents three major classes of the lane
colors [68]. LIDAR predicts the features of the grayscale image or a colored
image.Gradient feature extraction presents the appearance of the road and the lane. It
used steerable filters [69] [70], canny filters [43] Sobel filters [71], thresholding [72].

The shape feature can be extracted by discovering low-high-lowintensity patterns next

toimage rows [73]. Ridge properties of spectral lines can be extracted [74]. There are
many texture-based extraction methods available. Classification based on the texture,
localize the road shoulders, curbs, and guardrails [74].
Many times multiple features can be expected to make a robust system. For example,
road shape color and color gradients can be combined to model confidence in particle
filter [75].

15
The elevated demonstration of the road can be prepared by using marking
models. Three types of models are available for lane detection: parametric model,
nonparametric models, and semiparametric model. The parametric model is used for
straight lines [68], parabolic curve [70], circumference arcs [73], and hyperbola.

Nonparametric models, the models, are not given a priori , for example, continuous
pixels. The semi-parametric model has both parametric and nonparametric
components. The examples of nonparametric models are, Linear parabolic model,
Piecewise linear model for a polyline model, Hyperbole polynomial, Spline models,
B-spline model, Cubic spline model, cubic Hermite spline model [73] [4].

The semi-parametric model’s advantage is that it can use for any shape that is the
model has no particular geometry shape.Conversely, semi-parametric models causes
the overfitting problem as well as impracticalcurvature marking.

Poly models needs more parameters for fitting so that they can boost the processing
time.
The straight-line model is prevalent in perspective space, but the marking shape
becomes complicated in this region, and it raise complexity of fitting. The advantage
of BEV space is, it can presents the objectsreal shape from a top view.

Model regression is used for marking detection. It aims to get back a exact marking
shape amid the various features. There are three types of methods first is parameter
estimation method, second method is a robust iterative method, and the third is a robust
voting method. In the parameter estimation method, model fitting can be handled by
using the LSM(least square method) [70]. It computes the perimeter arcs and parabolic
curves. Weighted LSM is also used for marking detection. LSM is not robust because
the outlier may become dominant concerning the actual model.
Robust iterative method overcome the drawback of LSM. Here, modification is done
in objective function to limit the pressure of residue.A robust voting approach is used
to solve the problem by focusing on important pixels rather than model parameters.
Example of frequently used robust voting based method for lane mark detection is

16
Hough transform. Hough transform uses four parameters for computation then it will
boost the computation time. If Hough transform uses more than four parameters then
computation time will be more. RANSAC (Random sample consensus) is a
probabilistic voting method that reduces the computation time, and cost of voting
techniques like transform [76]. RANSAC’s drawback is, if the parameters are not
tuned properly then it will result in wrong fitting. Semiparametric and density model
uses density for fitting. [57]. Example of lane detection is demonstrated in figure 2.1.

.
Figure 2.1: Lane detection example

17
2.2. LANE SCENARIOS

Lane discovery assumes a significant job in lane departure warning frameworks.

However, there are few disturbances while capturing the lane images. In the lane
detection system, few conditions have to be taken care of,that needed for developing
the algorithm and the subsystems.

2.2.1. DIVERSITY IN THE APPEARANCE OF LANE AND ROAD

The line marking on the road having a size of 0.1m but observed that lane width is
variable. Lane marking has white or yellow colors, but some unique colors are also
seen on the road. Apart from this, one can find circular reflectors and cat’s eyes on
road. Number of lanes on road also varies. Roads are frequently straight, and curvature
is typically limited.

2.2.2. CLARITY OF THE IMAGE

As the roads are open and visible, show the nearby vehicles create some occlusions.
The roads are also covered with the shadow of the nearby buildings and trees. The
shadows on the road can create misleading edges and textures on the road. When the
vehicle comes out from the tunnel exchange in illumination level and change in
magnitude is noticed, and the overexposed image will appear.

2.2.3. ISSUE RELATED TO POOR VISIBILITY

Many weather conditions affect the visibility of the lane markings on road. It is tough
to identify the lane in rainy season, fog conditions, haze, during the night time, and
when the roads are covered with the snow.

Various effects on the road are depicted in figure 2.2. The variable lanes are showed
in figure 2.2 (a).demonstrationsof the lane marks with different width are depicted in
figure 2.2 (b). The variation in lane marks is depicted infigure 2.2(c).The scene

18
 Figure2.2 : Lane scenarios a)variable lanes b)variable lane mark width c) variable
lane width d)exit from tunnel e) shadow onroad f) road enclosed with snow g) fog h)
heavy rain i) nighttime and reflection on the wet road [7]

19
when the vehicle exits from the tunnel is depicted in figure 2.2(d). The roads
containing shadows are depicted in figure 2.2. (e). the weather effect that is the roads
covered with snow, fog, and heavy rains are depicted in figure 2.2 (f), (g), (h),
respectively. The reflection of the light on the wet road during the night time is
illustrated in figure 2.2 (i).

2.3. LANE MARKINGS

Interpretation of the lane marking is must for developing the lane departure warning
system (LDWs). There are following types of lane markings available on the road
surface:

Continuous centerlines

Continuous centerline can be crossed to enter the road and to leave the road, but
overtaking is not allowed.

Broken centerlines

The broken centerline line can allow surpassing.

Continuous edges lines

The continuous edge lines are used for selecting the street edge. The shoulder of the
street is an area that is at the one side of the lane. The shoulder of the street is used as
an additional path for vehicles as well as this area can be used by cyclists.This areas is
also used by the vehicle when they enter or leave the highway, at the time turning at
road crossing and when a halt is taken along the edge of a road [8]. The different types
of lane markings are described in figure 2.3.

20
 Figure 2.3 : Example of road lane markings [4]

2.4.RELATED WORK

Betke et al. [8] investigated color videos, that are taken from a vehicle while driving
on a freeway. For identifying and tracking the road boundaries, the blend of edge,
information of motion, and color information are used. Recursive least squares filter
is used to identify and track the street borders as well as lane markings. The algorithm
relies on the color information, so the algorithm could not be adapted to the author’s
situations. The video is processed in grayscale. The algorithm offered could not be
modified to a new situation as it relies on color

Borkar et al. [9] describe a lane detection system. Preprocessing is performed on the
camera captured images that include temporal blurring and grayscale conversion.
Next, IPM is used to eliminate perspective, and the picture is changed into BEV.
Adaptive threshold transform grayscale picture into a binary picture. Candida
calculating the low resolution Hough transforms.Coordinated sifting phase reviews the
candidate markers and removes the path marker focues. RANSAC evaluates the
parameters for fitting a scientific model using the improved path markers. Kalman
filter is used at last to anticipate parameters of all path marker line starting with one

21
edge then onto the next. In the matched filtering stage, the candidate markers are
inspected to take out the lane marker centres.

Yueyue Na et al. [10] introduced multiple and traffic monitoring Frameworks. It uses
an acrostic lane detection technique. In the multi Lane traffic monitoring system, the
first phase is Lane detection. This system identifies the location and width of vehicles
radiating sound. Based on Chinese Highway configuration and an array designed in
the microphone, so this method depends on cross array formation. The frequency band
is selected, and then the cross-correlation matrix is calculated from the subarrays for
traffic monitoring operation. After that, cross-segment is built by forming the beam
across the road. Single source exhibition is connected, and using the rank 1 MUSIC
(multiple signal classification) algorithm the passing vehicle azimuth is identified.
Finally, the azimuth vehicle’s probability density function is evaluated for individual
azimuth perceptions by using the Parzen window-based procedure. The peak and
valley patterns of the PDF are assessed to find the Lane midpoint and boundary.

Bok-Suk Shin et al.[11] identify the left and right lane borders independently by using
the particle filter. As it is detecting left and right Lane so it requires two particle filters,
one filter for a left side border and a second filter for right side border identification.
The various particles are combined to identify appealing particle in one image row, it
makes it a super particle, and for modifying detected border points, local linear
regression is used.

Liang Xiao et al. [12] established a lane detection system using a conditional random
field (CRF). The data is combined using a camera and LIDAR. In this system, the
initial pixels are lined up with LIDAR. The LIDAR points and labels are measured as
random variables. The hybrid energy function is minimized for gathering the labels.
The association between the adjacent pixel on the image plane is shown by using the
edges. This association is between attuned LIDAR points and their contrasting pixels
and LIDAR points in the 3D space. The data calm from the sensors is coordinated
probabilistically in this model. The use of the two sensors is done very efficiently.

22
Forgetting the Road region, the half and half CRF model is upgraded with diagram
slices.

Benligiray et al. [13] anticipated a video-based lane detection employing a rapid Vp

assessment procedure. At first, the line segments are removed from the image. At that
point, edge-based disposal of line portions is performed as per the viewpoint qualities
of path markings. Residual line portions are extrapolated and superimposed to
distinguish the picture position where a considerable lot of the straight edge highlights
meet. The discovered area is assumed as a vanishing point (Vp). Later, the line sections
whose expansions do not cross the Vp are disposed of; it is known as an orientation-
based removal. Lastly, the clustering of the residual line segment is done in such a way
that every cluster signifies a lane marking or a road border. The lane segment
possessions that include groups are converged to connote each bunch with a separate
line.

The discovered area is expected as an evaporating point (Vp). Afterwards, the line
sections whose expansions do not cross the Vp are disposed of; it is known as a
direction based evacuation. Ultimately, the grouping of the lingering line fragment is
done so that each bunch implies a path checking or a limit of the street.

Maire and Rakotonirainy [14] developed a system that analyzes driving session videos
collected using the onboard web cameras. Lane designs are identified and tracked by
the scheme, and relative vehicle location is computed relating to its nine lanes. The
video recording analysis is carried out in reverse temporal order. This system has many
benefits when it is compared with the forward analysis, but it is not appropriate for an
online system.

Li et al. [15] introduced Deep Neural Networks for visual lane detection. Here, the
focus is on the detection of the lanes from the traffic scenes that are necessary for
driving automation. There are two categories of deep neural networks involved in
recognizing the lane boundaries from the image, and they are CNN and DEEP CNN.
Deep convolutional neural networks perform the task of regressor as well as classifier.

23
The classifier detects the presence of visual cues like lane marks. If the lane marks get
detected, then the regressorestimations the location and orientation of the lane
mark within the ROI (region of interest). This proposed system can afford to work with
a vast region of interest because the regressor refines the prediction about the target.
Large regions have more affluent appropriate information that helps to progress the
detection accuracy. There are recurrent neurons present on the top of the CNN in
convolution neural network. The resulting network has a memory for identifying
global targets from the local cues. It is not necessary to specify global targets of
fundamental facts. A big set of augmented samples is used for training the network.
The network from the line images predicts the boundaries of the lanes. This system is
used for real-life traffic scene analysis.

Heba et al. [16] describe the Lane quest method. This method identifies the road by
using cell phones, and this system is highly proficient. The use of the wrong cameras
along with the cell phones, had delayed the possibility of lane detection in extreme
types of weather circumstances.

Manh et al. [17] describes a method for pedestrian. This method is visionbased
method to assist them on path detection. This proposed system has a less complicated
arrangement and increased vigor as well as precision. Itneglects shadow affected
paths.

Jihun Kim et al. [18] established a CNN system for lane detection. This system was
feasible; it uses RANSAC (Random Sample Consensus) algorithm for clear road
situation and AI-based machine learning algorithms for complex road situations.
Extreme Learning Machine (ELM).The ELM is used for calculating the network
weights. ELM calculates weight among the output layer and hidden layer in a single
iteration. Due to this, the learning time gets reduced when it produces the result using
minimal training data. The networks that have a single hidden layer to them
conventional ELM can be applied. Authors of the paper modified the back propagation
algorithm for finding the hidden layers targets and successfully learn network weights
though preservingenactment.

24
Chanho Lee [19] introduced a vision method for tracking and Lane detection using a
region of interest. The introduction of Lane detection and tracking is provided in the.
Region of interest is a curved trapezoidal region that covers the lane markings and
locates vanishing point method is used. The lane markings that fall under the region of
interest are detected and tracked. Color cue and gradient cue is formed together, and
the lanes are grouped by using the scan line test to check the features of the lane
markings.

Kodi et al. [20] detect the lane curvature using a modified lane detection algorithm. An
autonomous vehicle has an essential role in this. The curves of the road in the
autonomous vehicle are identified using a ground truth marking toolbox.

Yingna Su et al.[21] reduced the vanishing point method using a stereo- rig. In this
method, there is no need for the parametric leg for Lane detection. The vanishing point
of the road depends upon the visual odometry and divergence framework. The visual
odometry disparity map is used in searching space to reduce the vanishing point.
Visual odometry detects the straight lines as well as curves. The minimum cost map is
achieved by using Dijkstra's shortest-path Lane model. The outer edge of the lens is
illustrated by finding them to quiet lanes.

Chung-Bin Wu et al.[22] describes the system for LDWS and lane detection. This
system is using an ultra-low complexity block. The region near the vehicle is
determined using a parameterize Region of Interest (ROI). For identifying the lane
marks in various conditions, pixels intensity are used to recognize the lane markings
in different conditions, and intensity of the pixels on the lane markings in ROI is
increased. The computational burden reduced by dividing the ROI into non-
overlapping blocks. Two disentangled masks suggested one for getting the block and
second for gradients.

Rui et al. [23] describe a technique to identify various roads from the provided image.
The System allows multiple road lane detections in dense and high traffic. The authors

25
do the physical set up of the vehicle by using the air suction cushions that causes less
change in the rising edge of the camera.

Apurba Das et al. [24] have developed a binarization method. In this, the researchers
modified the MBMT (Min-Between-Max Thresholding) algorithm and named it as
MMBMT (Modified Min-Between-Max Thresholding). The results of the MMBMT
were prominent. In this approach, the ground truth is the baseline frame. So, if there
is a minimum change, it will induce an extreme change in performance, and due to
this, manual detection needed in this technique.

Soonhong Jung et al. [4] suggested a lane detection algorithm that is dependent upon
the spatiotemporal pictures. Pixels on the scan line are collected, and a balanced
spatiotemporal image formed. The direction of the lane points becomes smooth, and
straight lane formation performed. Next, the binarization of the spatiotemporal image
is performed. The predominant similar straight lines are identified by using the Hough
transform. The intersection is performed by identifying the right and left lane points.
The spatiotemporal approach is robust for the missing and occluded lanes when it is
compared with available frame-based approaches.

Kortli et al. [32] utilized a Hough Transform for lane and disappearing point detection.
Vanishing point detection was utilized to get the ROI that will lessen the
unpredictability of the algorithm for the following casing. The Otsu's threshold was
actualized to conquer the brightening variation issue. The Hough Transform extricates
the lane limits and screens the vehicle's situation as for the lane and gives notice if the
vehicle is going to leave the lane. Pre-processing step connected for smoothing and
gradient detection of the picture. After an RGB to grayscale picture conversion, we
apply a Gaussian filter to smooth picture gradient. Subsequent stage, we pick a Sobel
operator on the smoothing picture for edge detection. Otsu's threshold strategy is
connected to manage a lighting issue. Hough Transform is connected with the end goal
to distinguish a vanishing guide that serves to decide the ROI, which lessens the
computational unpredictability for the next casing. Finally, the framework decides

26
whether the vehicle floats of the correct lane or not, contingent upon the vehicle
position

Ju Han Yoo et al. [33] built up the strong path detection strategy utilizing the vanishing
point estimation model. Utilizing the vanishing point conspire enhanced the detection
rate in the other situation. The competitor line sections extricated for the detection
procedure can contrast from the beginning while at the same time executing the
equivalent in 3D space.

Tan and du [34] describes afeasible3D stereoreplication strategy to detect path. The
model detected the path with the assistance of the remaking approach. The system was
appropriate for the automaticvehicles precision. Additionally, vechicle path level
enhancement considered.3D sterio system considers street highlights like marking on
the road that gives the fake detection.

Mingliang et al. [35] developed a Two-Stage Feature Extraction (LDTFE) lane

detection approach. The LDTFE demonstrate they can acclimate to the troublesome
situations like as little line sections and line ebbs and flows. In specific cases, the street
may have nonexistent lines prompting path detection disappointment.

Kim [ 36] has introduced the RANSAC method for the detection of the lane. Here,
straight lanes are identified by combining neighboring and scattered points from the
image scenes. Due to the complexities and illumination in road scenes, this method
becomes unreliable. RANSAC often happen to be unpredictable because of the
complexity and illumination in road pictures. CNN is used to address these problems.
CNN enhances the image and extracts the ROI before applying RANSAC [37].

yam et al. [126]suggested a TFALDA calculation technique for lane detection.

TFALDA represents the 3 feature based automatic lane detection algorithm. These
algorithms are reasonable for speedy robotized detection of lane limits in various
circumstances without a dreary manual introduction, or earlier data in transit. The

27
quality of the calculation is to consolidate every 1 of the 3 of the fundamental attributes
of a lane boundary and control of the experienced development. TFALDA could form
into a strong wide assortment street circumstances by enhancing the parameters of a
transformative calculation, rather than physically improvingand testing. The lanes can
be the distinguished reverse point of view change in a picture plane view from the
guiding edge next street and the data expected to control the bearing of the channel's
yield can be gotten straightforwardly.

Jung and Kelber [127] tended to the issue of trackingand lane detection. A direct
model was utilized to surmised lane limits in the principal edge of a video arrangement,
utilizing a blend of the edge distribution function, and the Hough transform. Straight
parabolic model is utilized in the consequent casings. Straight piece of the model is
utilized to fit the close to the vision field, while the parabolic model fits the far-field.
The proposed line detection system is applied autonomously to every lane boundary.
In their work, data on conditions between lines is utilized for increasingdetection
results.

Kreucher C.et al. [93] introduced a Likelihood of Image Shape (LOIS) method for
discovering the robust lane markings in the existence of occlusion and a plurality of
illumination circumstances. While developing the algorithm the author has taken into
account the road laws during a series of images. The caution of a crossing is pending.

Litkouhi, et al. [94] described lane estimator design theory as well as lane controller
theory. They have used the Kalman filter for estimating the curvature of the roadway
and the vehicles relative location. The estimation inputs were the kinematic variables
of the vehicle that are given by the directional control model, plus information of the
lane boundary that was given by the video camera model. Here, implemented model
is used for input information of lane, but detection of the lane is not conferring in this
paper.

Gopalan et al. [95] developed a learning methodto detect and track lane markings. In
their proposed system they have proposed following things:

28
 1. The lane markings shared the nearby road regions related information, and this
information is modeled by using a pixel-hierarchy feature descriptor.
2. The appropriate related information selection for lane marking detection will
be carried out by using the robust boosting algorithm.
3. The lane markings are detected by the particle filters.
In this proposed approach the center point is the data quality, as there are many
cases such as rainy or foggy road situations where the image inputis not sufficient
for detecting the lane making.
Cheng and Chiang [96] proposed a navigation system. This system, is for the
intelligent robotic wheelchair that automatically follows lane. This system, needs
barrier free surroundings, and for automatic tracking navigation it uses video paint line
detection. In our lane detection application these conditions are not considered.

Isa [97] have done experimental studies on dynamics enactment of longitudinal and
lateral control for autonomous vehicle using the image processing. They have
described the algorithms for vehicle lane detection and vehicle lane tracking. They
have used color cue segmentation, edges were detected by using the canny edge
detection, and Hough transform is used for lane detection.

Taylor et al [98] described the parameterized model for extract the lanesfrom the lanes
from the image. The model extracts the position of the lane, width of the lane,
orientation of the lane. It also extracts stereo rig’s height and inclination relating to the
road. Here, stereo vision is used for lane detection.

Fletcher et al. [99] presented a monotony detector that evaluates the road scene. The
method uses the information about the lane butt the authors have not discussed the
detection job.

Hsieh et al. [100] develop an automatic traffic surveillance system using a single
camera. An essential traffic parametersare estimated from the video sequences.Here,
they have proposed an automatic method to detectall probable lane separating lines by
examining trajectories of the vehicles.

29
Maire and Rakotonirainy [102] develop a system for analyzing the driving videos. The
videos are gathered by the on-board Web-cameras. The system estimates the vehicles
relative position about its lane by detecting and tracking the lane marking. The reverse
temporal order is used for analyzing the video recording. This system is not suitable
for an online system when we compare it with the forward analysis.

McCall and Trivedi [101] developed a VIOLET system. It is a video based lane
estimation and trackingsystem. Steerable filters are used in VIOLET for the lane-
marking detection. This system uses front camera, several sensors, speed of the
vehicle. Steering of the vehicle and road and vehicle model as an input.

Andrade, D.C et al. [133] implemented a method for lane detection. This method also
tracks the basic requirements for setting up the features for driver assistance system.
The DAS consists of LDW and LKS. The sharpness of the image is improved by
minimizing the input image. The ROI is presented using the minimum protected
distance by the vehicles. Here, they obtained the smooth lanes for lane tracking
purpose. The proposed method achieved improved accuracy levels by using the road
shape.

2.5.RELATED WORK TO IPM

The images are taken by the camera and the captured images have perception effect.
It is essential to eliminate the perspective effect. So, inverse perspective mapping is
used to eliminate the perspective effect triggered by camera. The remote scene gets
condensed due to the perspective effect and it makes the further processing
complicated.

Alberto Broggi’s team performed research about inverse perspective mapping. This
team is from University of Parma, Italy. Initially the team projected the theory of IPM
[77]. After that, these theories were merged with image processing algorithms, parallel

30
processors as well as stereo camera vision system for PA-PRICA system that checks
and analyze the image. The PA-PRICA system was working with SIMD computer
architecture, for forming a full common obstructiondetection and lane detection
system named GOLD system [78]. On the ARGO investigational vehicle the GOLD
system was installed to accomplish the goal of automatic driving

Many researchers used the inverse perspective mapping technique [77] or they have
combined the similar mapping methods with other image processing algorithms for
detecting the lanes or the obstacles.

The inverse perspective mapping is used by Wen. L. Ji [79] for obtaining the 3D
information like vehicle’s front light, lane, and the distance.

S. Sehestedt et al. [80] have estimated multiple lanes by using the blend of IPM and
clustered particle filter technique depend upon the lane feature.

Yang [81] attuned translation the IPM characteristic that was proposed by Broggi [70].
Here, on the world exterior a straight horizontal line in the picture is estimated as an
arc. Though, a straight horizontal line in the image is projected to a straight line and
the straight vertical line in the image is projected to a straight line on the world surface
and the continuation will overtake the camera vertical projection point.

Jiang et al. [82] described a fast IPM algorithm for detecting lanes and obstacles.

Nieto et al. [83] used vanish point estimation technique to alleviate inverse perspective
mapping image. Tan et al. [84] developed a system to detect the vehicle on the side
blind spot; they merged the inverse perspective mapping and optic flow.

Muad et al. [85] established a lane tracking system by using inverse perspective
mapping techniques and they have also discussed the factors that influence IPM.

31
Cerri et al. [86] estimated the free driving space on the highway by using the stabilized
sub-pixel precision inverse perspective mapping time and image correlation.

The image position is important as based on the various image pixels the perspective
effects are different. On the other hand, when the perception effect present in the
image is removed then each pixel has identical piece of the information content. It
allows the homogeneous allotment of the information among all pixels in the image.

We can also say that, less information is present in the pixel which is at the lower part
of the image. The pixels that are present in the middle of the image contain more
information. This effect of non-homogenous distribution of the information content
among all the pixels in the image, an IPM method is used. IPM removes the perspective
effect.

The perspective effect removal process needs some information like the particular
acquisition conditions such as camera orientation, position and optics, and the view
characterized in the image such as road. This comprises the a priori information.
The main aim of the perspective removal process is to resample the input image, after
that remap all pixels on the original camera’s image to a diverse location and produce
a new 2-D array of pixels. The final output of the process is an image with a top view
of the scene. This obtained image can be observed from a significant height.

32
 (a)

(b)
Figure 2.4: Demonstration of IPM (a) Camera captured image (b) Image after IPM

33
2.6.OVERVIEW OF FUNDAMENTAL ALGORITHMS

Lane detection is essential in ADAS. Many algorithms are proposed for lane detection
approaches. These days’ machine learning algorithms are becoming popular among
the researchers. The outcome of these algorithms is also better. Some machine
learning based algorithms are reviewed in this chapter. These algorithms are also
related to our research. Here, we are going to overview crow search algorithm, earth
worm algorithm and deep convolutional neural network. These algorithms are used in
the proposed EW-CSA based DEEP CNN approach.

2.6.1. CROW SEARCH ALGORITHM

Crow Search Algorithm is announced as a nature inspired meta-heuristic analyzer.

Crows are viewed as the most insightful bird. Crows are small in body size but they
have largest brain, but compare to human brain their brain is lower. There is abundant
proof available of crow’s intelligence. They have exhibited mindfulness in reflect tests
and have the tool-making capacity. Crows’ have the power to remember the faces, and
they communicate with each other in an effective way. If any unfriendly thing is
approaching to them then warns each other[87-90].

Crows observe other bird’s and see that where other bird’s are hiding their food, and
then they steal that food once the owner go off. If the crow has stolen the food then he
will take extra care to move to the hiding places [91].

The principles of CSA are listed as follows:

1. All the crows stay in the group.

2. Crows keep in mind their hiding positions.
3. For stealing the food they follow each other.
4. Crows guard their food store from being appropriated by a probability

34
CSA implementation for optimization

The implementation of CSA performs the following steps:

1. Initialize problem and variable parameters

2. Initialize position of the crow and memory of crows
3. Evaluate fitness function
4. Generate fresh position
5. Check the viability of fresh positions
6. Estimate fitness function of fresh positions
7. Update crow memory
8. Check termination criterion

suppose, there is a 𝑑 dimension environment, 𝑁 is the number of crows, and the crow
position at each iteration 𝑡 is given by the vectorai,t (= 1,2, … … N; t = 1,2, … . . t m ).

ai,t = [ai,t i,t i,t

1 , a 2 , … … , a d ]andt m denotes the maximum iteration. Each crow is having

their own memory of its hiding location. hiding position of the at every iteration is
denoted by hj,t .

Crow 𝑖 has the best position and every crow in its neighborhood find its best position
to hide the place for storing food.

Assume that a crow is visiting its hiding place on iterationhj,t . After that, crow 𝑖 is
following the crow 𝑗 in crow𝑗’s hiding place. So, the probable conditions are given
below:

• Crow 𝑗 is not aware that crow𝑖 is following him.Consequently, crow 𝑖

move towards the hiding place of crow 𝑗.Crow‘s updated new position
is given below.

ai,t+1 = ai,t + r × 𝑓𝑙 i,t × (hj,t − ai,t ) (2.1)

35
 Here, 𝑟 is denoted as a random number, and this random number is
between 0 and 1. The 𝑓𝑙 i,t denotescrow's long fly on iteration𝑡.
The figure shows the condition 1 and its effect on the search ability.
The crow gets directed towards the local search (about ai,t ) for the
small number of 𝑓𝑙 , and the crow gets directed towards the global
search (away from ai,t ) .

Here, figure (a) illustrates the condition if𝑓𝑙 < 1. Next crow position

is lying between ai,t and hi,t .

Figure (b) shows the condition𝑓𝑙 > 1. The position of the next

crow is 𝑖 on the path outsidehi,t .

• Crow 𝑖 is not familiar thatcrow 𝑗 is following him. So, to protect the

storage of the food since not to be search, crow 𝑗 will move to the new
position to fool the crow 𝑖. The following calculation expresses this
condition:

ai,t + r × 𝑓𝑙 i,t × (hj,t − ai,t ), rj > 𝐴𝑃j,t

ai,t+1 = f(x) = { (2.2)
a random position, otherwise

(a) 𝑓𝑙 < 1

36
 (b)𝑓𝑙 > 1

Figure 2.5 :The diagram of CSA position and the result of 𝑓𝑙 on the search ability.

2.6.2. Earth Worm Algorithm (EWA)

Earth worm algorithm is a nature stimulatedmetaheuristic algorithm used for

optimization. As we know earthworms are used for aerating the soil by using their
burrowing act and helps to enhance the soil with the help of their waste nutrients. This
earthworm’s contribution inspires the algorithm called earthworm optimization
algorithm (EWA). The Earth worm algorithm has two types of reproduction system
that is reproduction 2 system and reproduction 1 systems.

In the reproduction 1 system it produces only one progeny by itself and in

reproduction 2 systems it produces one or more than one offspring at once and to
produce this it required nine superior crossover operators, such as, Single point
crossover, n-point crossover or multipoint crossover, Uniform crossover, Discrete
crossover, Arithmetical crossover, Simulated binary crossover, Wright's heuristic
crossover, Dynamic fuzzy connective based crossover, Average bound crossover
[105][106].

37
The crossover operators are used for merging the genes of two or more parents for
producing better offspring. The main idea behind it is, if the information is exchanged
between good chromosomes then it will produce better offspring.

Single point crossover: Single point crossover is the most used technique. In this
crossover technique, a single crossover point is selected by selecting a random number
between (1-N-1) on the body parent chromosome [107] [124]. Here, N-1 is the length
of the chromosome. The parent’s chromosome is split at a single point and points in
either chromosome are exchanged between the parent’s chromosomes. Single point
crossover is shown in figure 2.6 (a).

Multipoint crossover: Multipoint crossover is a simplification of the single point

crossover. Here, ncrossover points are randomly selected and then they split along
those points and organize, and next alternated between parents. For example, we have
taken 2 point crossover, it means N=2. The value of N varies from 1-N-1 and then the
genes are exchanged [112] [113]. Multipoint crossover is shown in figure 2.6 (b).

Uniform crossover: Uniform crossover does not split the parent’s chromosome and
recombine it like single point crossover and multipoint crossover. Uniform crossover
treats each chromosome independently to select for the offspring. Initially the number
of crossover points is not fixed. It uses the crossover mask to recombine the genes of
the parent’s chromosome. For example, the number of crossover points are x in
chromosome. The value of x is a random value and this value is less than the length of
the chromosome. Based on the random value crossover mask is generated [118]
[119].Uniform crossoveris shown in figure 2.6 (c).

Discrete crossover:Discrete crossover[106] is similar to the traditional one-point and

uniform crossover.

Arithmetic crossover:The arithmetic crossover [111] is used for real value encoding.
A linear combination of the two chromosomes is defined by the Arithmetic crossover
operator and randomly two chromosomes are chosen for crossover. After that, two
offspring’s are produced that are linear combination of their parents.

38
Average bound crossover:An Average bound crossover [105] uses average crossover
and bound crossovers. In average bound crossover, four offspring’s are produced by
a pair of parents and out of that four, two best offspring’s are selected.

Hybrid crossover:The hybrid crossover [120] uses a variety of types of crossovers,

here numerous offspring solutions are obtained from the similar set of parents.

Geometrical crossover:Geometrical crossover[121] are defined based on the solution

space distance not on the representation.

Simulated binary crossover: Simulated binary crossover [114] is used for simulating
the result of one-point crossoveron a sequence of binary alphabets in a constant
domain.

Wright's heuristic crossover:Wright's heuristic crossover [115] used the fitness

function value of the parent’ssolutions. It will be used for producing theoffspring from
the parent’s duo with a bias to the superior one.A few crossover operators like
Unimodal normal distribution crossover (UNDX) [110], UNDX-m [117], and Parent
centric crossover (PCX) [116] has several parents.

In our research we have used, multipoint crossover,uniform crossover, and single point
crossoverso, we have discussed these three in details. There are many other crossover
that are developed by the users are available.

39
 (a)

(b)

(c)

Figure 2.6 : (a) Single point crossover (b) multi point crossover (c) Uniform cross over

40
2.6.3. DEEP CONVOLUTIONAL NEURAL NETWORK

Currently, Neural networks are the majority mainstream machine learning

calculations. It has been unequivocally demonstrated after some time that neural
networks beat different calculations in precision and rapidity. With different variations
like CNN, Auto Encoders, RNN, Deep Learning and so forth, neural networks are
gradually turning out to be for information researchers or machine learning specialists
what direct relapse was one for analysts. It is along these lines basic to have a major
comprehension of what a Neural Network is, the means by which it is made up and
what is its span and confinements.

The convolution neural network (CNN) is mostly used in object recognition research.
The features of CNN are it gives high detection accuracy, feature learning is done
automatically, and it gives end to end recognition. A few researchers have applied
CNN in deep learning methods for lane detection and as compare to the traditional
methods of image processing the lane detection accuracy is increased from 80% to
90% [128]

Li et al. [129] developed a lane detection system using deep RNNandCNN. Here, a
small ROI image is provided to the CNN. The CNN provided two outputs. First output
is a discrete classification that indicates the visual cues are lane markers or not.
However, if the lane is detected then the output will be the uninterrupted estimate of
orientation of lane and lane location. The global lane recognition is done using the
video sequences, here, local lane position in the image is not used, RNN uses internal
memory scheme to recognize the structure of the lane in the series of data. Training is
done by merging the scenes from the three various cameras facing front area, left area,
and rear area individually. RNN identifies the lanes and connects the lanes that are
enclosed by obstaclesorvehicles.

In neural network a neuron is a placeholder for a mathematical function. The neuron

affords an output by applying the function on the provided inputs. A neural network
having extra(more than 1) hidden layer is normally referred to as a DCNN. It has

41
activation functions are step, sigmoid, tanh, and ReLU. The most frequently used
activation function is ReLU.

The ReLUis the most frequently used activation function in deep learning models. If
the function get any negative value then it will return 0. But, if the function gets any
positive value then it returns that positive value back.

Convolutional Neural Networks (CNN)

CNN is one of the variations of neural networks utilized intensely in the field of
Computer Vision. It gets its name from the kind of hidden layers it comprises of. The
hidden layers of a CNN commonly comprise of pooling layers, convolutional layers,
normalization layers, and fully connected layers. Here it basically implies that as
opposed to utilizing the ordinary activation functions characterized above, convolution
and pooling functions are utilized as activation functions.

Convolutional layer

In this layer, convolution is performed. Convolution operates on two signals or two

images. One input can be image and the second input is filter or you can say kernel on
an input image. it means convolution is taking two input images and produces a third
one as output [126].

Pooling layer

Pooling is a sample-based discretization process. Here, dimensionality of the image is

reduced and it allows the assumption about the features delimited in the sub regions
binned. Two types of pooling are available and they are max and min pooling. Max
pooling selects the maximum value from the selected region and min pooling selects
the minimum value from the selected region. The figure shows the max pooling.

42
Figure 2. 7: Architecture of CNN

Figure 2.8: Max pooling example

43
In this way, as should be obvious a CNN is essentially a deep neural system that
comprises hidden layers having pooling and convolution functions notwithstanding the
activation function for presenting non-linearity.

Fully Connected layer

Figure 2.9: Fully connected Layer

The data is converted into a 1-dimensional array for inputting it to the next layer. We
flatten the output of the convolutional layers to create a single long feature vector. The
fully associated layer is a classification model where you can classify lanes and roads,
dogs and cats etc.

The other applications of Deep neural network and CNN character recognition and
traffic signal recognition [135], to classify histology slides of brain and breast tissues
[136], classify pneumonia [137], medical image pre-processing [138] and medical
image analysis [140], and to diagnose different serious brain haemorrhages [139].

2.7. CHALLENGES AND LIMITATIONS OF THE LANE DETECTION

SYSTEM

There are various limitations and challenges for lane detection system

➢ There is a need for detecting numerous edges to mark the single lane line. This
is detection process involves modification from low to high intensity and high

44
 to low intensity, but it creates confusion complication for recognizing the
proposed inbound edges for the given lane mark [132].

➢ Many researchers have developed various algorithms for Lane detection. Out
of these algorithms, maximum algorithms focus was curved Lane detection by
using a linear parametric lane model or nonlinear parametric Lane model. The
parametric Lane model more likely failed because the line borders were
blocked by the tessellated Shadow patterns [21].

➢ While detecting the length if Global constraint is included on the shape of Lane
edges then it has been observed that the lane detection algorithm suffers from
the noise for blockage [43].

➢ It is very difficult to make the lane departure warning system fully autonomous
because of the weather problems like fog, heavy rain and roads covered with
snow. Sometimes the faded lines also not get recognize properly. Another
reason is the cost required is very high for accurate mapping of the information.
Moreover, the solution given for onboard road and for the length perception is
in limited form [7].

➢ Another, challenge for vision-based intelligent vehicles is the size of the real-
time data is very big and it is a very essential issue because there is an
involvement of complex computation. Hence there is a requirement of beside
come computer systems for obtaining real-time performance [131].

➢ Some researchers have developed a Lane detection system for highway

implementation but when these systems are used in urban areas, it will not give
accurate Highway oriented Lane detection because of the disturbances and
dense traffic. So we cannot use one single evaluation method for assessing all
the existing systems.

45
➢ Lane detection is also influenced by the hardware which is used for capturing
the road images. The camera or any vision hardware news for recording the
data influence the lane detection system. The cameras have various resolution
and field of view that influences lane detection accuracy.

➢ The single-lane detection system has its limitations to deal with varying road
and traffic conditions. So it is essential to make a backup system that will
enrich the functionality of the Advanced driver assistance system

➢ The researchers can develop algorithm level integration, system-level

integration, and sensor level integration system. The algorithm level
integration easy to implement and have a low cost. System-level integration
integrates Lane detection systems and other perception systems like road and
vehicle detection to increase the robustness and accuracy of the system. But
algorithm level integration and system-level integration still rely on camera
vision system and have their inevitable margins. Sensor level integration is
very reliable for detecting the lens under various conditions but they are very
expensive.

➢ The lane detection system does not have a standard evaluation method that can
be used as a comprehensive assessment of the Lane detection system
performance.

➢ LDWS and Lane Keeping Systems depend on prominent lane markings. They
regularly cannot decode blurred, missing, or off base lane markings. Markings
shrouded in snow or old lane markings left noticeable can thwart the capacity
of the system.

➢ Lane Departure Warning Systems additionally face numerous legitimate

restrictions concerning independent driving. As expressed beforehand, this

46
 framework requires consistent driver input. Vehicles with this innovation are
restricted to helping the driver, not driving the vehicle.

➢ Lane Departure Warning Systems' most significant confinement is that it is not

in complete control of the vehicle. The framework does not consider different
vehicles out and about and cannot supplant great driving habits.

Summary

Lane detection is an important part of the advanced driver assistant system. In this
chapter, we have taken the overview were we have studied the methods used for
preprocessing, then what features we can extract, the feature extraction methods we
have seen the shape features, color features , texture features and multiple feature
extraction. High level presentation can be done by using the parametric, noparametric
and semi parametric model. Different regression model techniques are available for
model fitting. Lane conditions or issues are creating problems for lane detection, like
lane width variation, poor visibility, climatic conditions etc. Different types of lane
markings are available on the road such as broken centerlines, continuous and
continuous edge lines. Many researchers have introduced various methods for lane
detection. An overview of the work done in the ADAs is taken. IPM is used for the
transformation of the image. The image is transformed into BEV image. A look on
the existing work on IPM is given. Metaheuristic algorithms like CSA and EWA are
used for the optimization purpose. These two algorithms are nature inspired
algorithms. At present days machine learning algorithms are doing well and are very
popular because of the precision and the speed. The Deep CNN algorithm is a machine
learning algorithm that performs very well and it gives maximum detection accuracy.

47