IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO.

8, AUGUST 2024 9001

An Integrated Vision-Based Perception and Control

for Lane Keeping of Autonomous Vehicles
Tesfamichael Getahun and Ali Karimoddini , Senior Member, IEEE

Abstract— This paper proposes a vision-based control method perspective, lane keeping is a fundamental control component
for autonomous vehicle lane keeping. This paper aims to that drives the car to the center of the lane to execute a planned
provide a reliable alternative solution for localization and path toward the destination. Either way, lane keeping requires
path planning-related challenges in environments where GPS is
unavailable or unreliable, such as rural areas with dense forest appropriate sensors to localize the car with respect to lane
cover, where tree canopies may obstruct GPS signals, and urban boundaries and assess the vehicle’s lateral position, followed
landscapes characterized by tall buildings that can similarly by smooth control of the car to correct drifting out of the lane
disrupt navigation accuracy. The proposed method consists of center. Such technology can significantly increase the safety of
a robust lane detection algorithm to generate a reference path of driving cars and leverage their autonomy level. Such a higher
the vehicle and a model predictive controller (MPC) for tracking
the reference path. The lane detector extracts lane markings from level of safety and autonomy could be achieved only if the
image frames to determine ego-lane boundaries from which the performance of the sensors and perception systems are robust
lane center is calculated in the vehicle’s coordinate frame. The enough to the variation of environmental conditions.
MPC uses a kinematic vehicle model to generate the lateral and To keep the vehicle in the lane, the control subsystem of the
longitudinal control values necessary for smoothly tracking the vehicle generates appropriate steering and acceleration/braking
reference path. The proposed technique has been implemented
and tested on a Lincoln MKZ hybrid vehicle equipped with commands to drive the vehicle to follow the desired path and
a computing platform having Intel’s quad-core Xeon proces- speed. This could be achieved by different control techniques
sors. The developed framework was experimentally validated that take the planned path as input and produce control
by deployment in a rural test track. Experimental results show commands as output. In [2], a proportional (P) controller
that the proposed vision-based lane detection method performs is used to correct the lateral offset of a car. To achieve a
well under various challenging road conditions such as shadows,
road texture variations, interference from other road signs, and better performance, it is also common to use a Proportional-
missing lane boundaries. The multi-threaded implementation of Integral-Derivative (PID) controller [3] for lateral control of
lane detection and the MPC allowed us to run the integrated a car. A lateral controller is designed in [4] in the form
system at the speed of 25 HZ. The integration of lane detection of a full-state feedback controller. A sliding mode controller
and MPC resulted in smoothly keeping the car in the center of
is used in [5] for lane keeping. The works in [6] and [7]
the lane over a curved test track while respecting the physical
constraints of the car. compare PID, LQG, H∞ , adaptive, and fuzzy controllers for
vehicle lane keeping. Most of these methods separately design
Index Terms— Lane detection, model predictive control (MPC),
visual perception, lane keeping, autonomous vehicles.
lateral control (for steering) and longitudinal control (for
acceleration/braking), ignoring their coupling effects which
may negatively impact the controller performance, particularly
I. I NTRODUCTION at higher speeds and curvatures. Further, these methods rely
on the current and past history of feedback signals, which may
W ITH the rapid development of autonomous and semi-
autonomous technologies, we are witnessing the
adoption of different levels of autonomy equipped with
result in generating slow or jerky control signals. In contrast,
MPC generates an optimal control signal over a finite time
various Advanced Driver Assistance Systems (ADAS) [1]. horizon, incorporating the vehicle dynamics and respecting
From a semi-autonomous driving perspective, Lane Keeping various constraints. Using the motion model enables the MPC
Assist (LKA) is an important ADAS component that keeps the to predict the future states of the vehicle and estimate the
vehicle centered within its lane. From the autonomous driving required control parameters by iteratively solving a constrained
optimization problem [8]. The MPC takes the planned path as
Manuscript received 17 July 2022; revised 10 May 2023, 30 September a reference input and provides the actuator commands that
2023, and 5 February 2024; accepted 24 February 2024. Date of publica-
tion 27 March 2024; date of current version 1 August 2024. This work can bring and keep the vehicle close to the reference path.
was supported in part by the U.S. Department of Transportation (USDOT) The planning module often uses a high-definition map or
University Transportation Program (UTC) under Grant 69A3552348304 and navigation map and GPS sensor together to localize and plan
in part by the North Carolina Department of Transportation (NCDOT) under
Award RP2022-16 and Award TCE2020-03. The Associate Editor for this the vehicle’s path to reach its goal [9]. Having the current
article was Z. Xiao. (Corresponding author: Ali Karimoddini.) state of the system and its desired path both in a global
The authors are with the Department of Electrical and Computer Engineer- coordinate system facilitates the design of the MPC. However,
ing, North Carolina Agricultural and Technical State University, Greensboro,
NC 27411 USA (e-mail: [email protected]). GPS sensor data can be very noisy and unreliable depending
Digital Object Identifier 10.1109/TITS.2024.3376516 on the scenarios on the road. For instance, GPS data can be
1558-0016 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9002 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

completely lost or highly uncertain in urban roads, tunnels, and may result in missing the lane boundaries. Then, the MPC
and rural roads surrounded by tall trees which affects the takes the reference path generated by the lane detector as an
localization and planning system. One potential solution to input and provides optimal steering, and acceleration values to
the problem is to use vision-based localization and planning smoothly follow the path. A kinematic vehicle model is used
for lane following or lane keeping until GPS data is recovered. in the MPC implementation for predicting the vehicle states
Vision-based localization and planning can be achieved by and corresponding actuator commands over a defined horizon.
extracting visual cues from the image frames creating an In summary, the contributions of this paper include:
equivalent representation of the real-world environment to • A new robust lane detection technique is developed to
determine the next actions of the vehicle. For lane following, handle challenging road scenarios such as roads with
first, it is necessary to find the lane boundaries indicated by degraded lane markings, texture changes, shadows, light-
white and yellow paintings on the road. For this purpose, the ing variations, and missing lane boundaries. Novel about
visual perception component extracts the lane markings and the proposed technique is adopting the Gabor filter,
determines the location of the lane boundaries with respect to which is a Gaussian-like (Gaussian-modulated sinusoid)
a common reference frame for the road section ahead of the filter, for lane marking enhancement. The extracted lane
vehicle. Different techniques such as the conventional Canny markings are then further processed to verify that the
edge detector and Hough transform [10], [11], [12] and deep- extracted pixels belong to the lane lines. Finally, a curve
learning-based approaches [13], [14], [15] have been used in is fitted to the extracted lane markings to represent the
the literature for robust vision-based lane detection. The lane lane boundaries and the lane center line, which are then
detector can provide the location of lane boundaries, from transferred to the vehicle coordinate frame as a path to
which the lane center can be estimated as a reference path be tracked by the MPC.
to steer the vehicle in between the lane boundaries for a safe • The developed vision-based lane detection is integrated
and smooth traffic flow. Common vision-based lane-keeping into a nonlinear model predictive controller design in
control techniques consider the error in the form of the devi- order to generate optimal actuator control inputs in the
ation from the lane center in the current image frame, based form of the steering angle and acceleration/braking values
on which a controller can be designed to generate correction to track the center of the lane while respecting the vehicle
control signals. For example, the work in [16] developed physical constraints.
a vision-based path-following method using classical image • The proposed integrated vision-based control technique
processing techniques and convolutional neural networks to is applied to a real electric vehicle (Lincoln MKZ) called
determine road geometry and view angles from which the AggieAuto as shown in Figure 13 and tested over a
steering angle control command is estimated and fed to a PID curved test track, demonstrating a smooth performance
controller. Similarly, [16] used a vision-based PI controller for for keeping the car close to the center of the lane.
the steering angle of a car. In [17], a visual servoing technique To the best of our knowledge, this is the first work that is
is designed including a nonlinear feedback controller for the applying vision-based MPC to control an autonomous car in
angular velocity of the car, while considering a constant linear an experimental setting. The proposed lane marking extrac-
velocity. In [18], a vision-based nonlinear backstepping control tion algorithm, enhanced with Gabor filter, provides a robust
strategy is used for steering and braking coupled dynamics yet computationally effective performance, which enables the
followed by an adaptive fuzzy sliding mode reaching control integration with the MPC controller for real-time control of
law to reduce the chattering problem often associated with vehicles.
sliding mode control laws. Most of these methods use the The rest of the paper is organized as follows: Section II
current and past history of error value with respect to a single illustrates the structure/architecture of the proposed system
goal point on the center lane, which may result in non-smooth with a brief description of the main blocks, section III
control signals. The challenge here is to select the goal point, presents the lane detection algorithm in detail including pre-
which is usually chosen based on the road curvature. For processing, mapping to bird’s eye view, Gabor filtering, lane
curved roads, the goal point is usually selected closer to the marking extraction, curve fitting, and validation of detection
vehicle to minimize corner-cutting otherwise goal point can be results. Section IV presents how a reference path is generated
far away for straight roads. This varying look-ahead distance based on the lane detection output, and Section V covers
for the selection of the goal point can further impact the the design and integration of a model predictive controller
performance of the controller. To address these challenges, with the lane detector. Section VI presents the experimen-
instead of targeting a single goal point, in this paper we tal results and compares the performance of the proposed
propose to integrate vision-based localization with MPC to method with the ground truth data. Section VII concludes
generate smooth control signals to follow a set of waypoints. the paper.
In this paper, a vision-based control method for self-driving
vehicles is proposed. The proposed technique comprises a
path generator based on ego-lane detection and vehicle control II. T HE P ROPOSED V ISION -BASED P ERCEPTION AND
modules. The lane detection algorithm reliably estimates the C ONTROL S TRUCTURE
lane center from lane boundary detection and computes a This section presents the high-level architecture of the
path for the vehicle to follow under various challenging road proposed system for vision-based perception and control of
conditions such as strong shadows that cover the lane markings an autonomous car.

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9003

Fig. 2. The block diagram representation of the proposed lane detection

method.
Fig. 1. The high-level architecture of the proposed vision-based navigation
and control system. This paper focuses on the Lane Detection, Path Generator,
and the Controller.
A. Pre-Processing
The pre-processing stage involves image rectification based
A. Implementation Architecture on camera calibration information, image transformation to
The proposed high-level architecture, shown in Figure 1, change the view of the image via Inverse Perspective Map-
consists of two main modules: Path Generator and Controller. ping (IPM), color conversion, and filtering operations which
The Path Generator block is responsible for detecting road are briefly described below.
lanes and generating a path along the center of the lane. 1) Camera Calibration and Image Correction: Camera
The Controller block estimates actuation commands (steer- calibration is simply the process of determining the camera
ing, acceleration, and braking) to optimally track the input parameters and its distortion coefficients. Calibration infor-
path fulfilling vehicle physical constraints. The Controller mation is used to correct radial and tangential distortions
uses a kinematic vehicle model to predict the vehicle’s state introduced by the camera sensor. It is also used to map
and receives feedback signals from the vehicle’s sensors the image pixel dimension to real-world units, which are
to generate smooth control signals. Current orientation and appropriate for the operations at the Path Generator block.
acceleration are received from the IMU sensor placed at the This is a standard procedure for which the details can be found
center of gravity of the vehicle. Steering angle feedback is in [19] and [20]. A chessboard of size 9×6 is used to calibrate
obtained from the Electronic Power Steering (EPS) system the camera and apply corrections on each frame.
of the vehicle through the CAN bus. Commands from the 2) Inverse Perspective Mapping: The second stage of the
Controller and feedback signals from the actuators (EPS, pre-processing block performs mapping the image from per-
Engine Controller Modules (ECM), and Electronic Braking spective view to top-view, also known as Bird’s eye view, using
Controller Module(EBCM)) are exchanged via Interface Unit. the Inverse Perspective Mapping (IPM) method.
The Interface Unit facilitates and monitors the communication Mathematically, the mapping operation is performed by
between actuators/sensors and the Controller. It is a finite- using a transformation matrix H . For this purpose, one can
state-machine (FSM) based implementation to monitor the manually select 4 points from the source image and their
vehicle’s modes of operation. In addition, it performs sanity corresponding 4 points from the destination image, and then,
checks, corrections, and mapping of the control commands compute H to map the source to destination points. The source
to a proper data format compatible with the CAN protocol. points are picked usually from the corner coordinates defining
Feedback signals from the actuators are also handled and trapezium-shaped Region Of Interest (ROI) in the perspective
converted to a proper format by the interface unit to be view of the image. The destination points are selected by trial
provided to the Controller. and error until the desired bird’s eye view is achieved after
This paper focuses on the main developed software compo- transformation (typically can be calculated after a few trials).
nents, including the Path Generator and the Controller, which Since the camera is fixed on the vehicle, the value of H is
are presented in detail in the next sections. required to be calculated only once.
Then, each of the pixels from the source image can be
mapped to the top view image as:
III. L ANE D ETECTION
[x ′ , 1]T = H × [x, 1]T (1)
This section describes the proposed lane detection method
which takes a stream of images from the camera and outputs where x ′ is the pixel coordinate in the IPM (destination) image
ego-lane boundaries L l and L r . The main components of the and x, is the coordinate of the pixel in the source image with
developed Lane Detection block are shown in Figure 2 and adjusted dimension to perform homography. Since lane bound-
the functionalities of each of the components are presented as aries appear parallel in the bird’s eye view, measurements like
follows. lane width, marking thickness, and length can be estimated

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9004 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

Fig. 3. Result of the pre-processing stage before applying Gabor filtering: Fig. 4. The effect of Gabor filter on enhancing lane marking in the bird’s
Left: raw color image captured by the camera - before pre-processing, Right: eye view; Left: Gabor filter kernel, g, Right: Gabor filtered image, I f , with
image after rectifying, filtering (like Gaussian filtering), transforming to bird’s the zoomed-in section showing the effect of Gabor filter on the surrounding
eye view, and color conversion. area of the lane marking.

and directly associated with the actual lane measurements. that determines the shape of the filter. As an example, γ value
To perform this mapping, the road is assumed to be flat or of 1.0 creates a circular filter, otherwise, the shape of the
with a slight slope in the ROI. Otherwise, the lane lines filter will be elliptical. In our lane detection algorithm, the
may not appear parallel in the mapped image for some road wavelength, λ, and the standard deviation, σ , are chosen to
segments which results in less accurate detection. Practically, be equivalent to the lane marking thickness. Further, since the
this assumption works well if the look-ahead distance in the real and imaginary parts of the filter are orthogonal, they carry
ROI is not too far. In our experimental setup, we found that a redundant information, and hence, we only use the real part
look-ahead distance of 30 meters is practically good enough of the Gabor filter as:
for lane-keeping control because the slope variation in that x ′2 + γ 2 y ′2 x′
range has little effect on the lane detection accuracy and g(x, y) = ex p(− )cos(2π + ψ) (3)
2σ 2 λ
this range is sufficient. To further accommodate the effect
We tuned the Gabor filter parameters based on the geo-
of slight slope variations on detection performance, we used
metrical structure of the lane boundaries in the bird’s eye
range thresholds for lane width and lane marking thickness.
view, dimensions of lane markings, and orientation of the
The details are presented in
lane lines to achieve better lane marking enhancement. In our
3) Color Conversion: Conversion of image color from RGB
experimental setup, in the bird’s eye view image, the lane
to YCrCb is the third step in the pre-processing stage. The
marking thickness is on average 7 pixels wide. Therefore,
YCrCb color space is selected for two reasons: first, the ‘Y’
to enhance the lane markings better we chose to set the
channel consists of the grayscale version of the image which
standard deviation σ and wavelength λ as 7.0. The values
captures the white lane markings, and second, the difference
of γ , which determines the ellipticity (aspect ratio) of the
between the ’Cr ’ and ’Cb’ channels captures the yellow lane
kernel, and ψ, the phase shift of the Gabor function, are
markings better than the grayscale image. The Cr channel is
fine-tuned experimentally as 0.138 and 1.43 to achieve the
particularly important to handle yellow markings on a grayish
best performance. After applying the IPM, lane lines will
background such as bridges or other concrete road sections.
have mostly vertical orientation. Hence, for θ, two values of
Therefore, to better both white and yellow lane marking colors,
0 and π are selected for the Gabor filter. Using (3), a 2D filter
the channels of YCrCb image are split and separately filtered
kernel of the desired size is calculated for θ = 0 and θ = π,
to enhance the marking features. Figure 3 shows an example of
keeping other parameters constant. The final kernel is the sum
an image that has gone through pre-processing steps including
of the kernels corresponding to each θ value. This results in
mapping to the bird’s eye view and color conversion.
a filter kernel stretched vertically with a width equivalent to
4) Gabor Filtering: Gabor filter is a Gaussian-like (Gaus-
the average lane marking thickness.
sian modulated sinusoid) filter with more parameters than the
Figure 4 shows a Gabor filter kernel of size 7 × 7. For an
regular Gaussian filter. It is rarely used for lane detection
input image, I0 , the filtered image, I f , is calculated by taking
algorithms although it is a popular filtering technique for
2D convolution with the grayscale image calculated as:
texture segmentation due to its ability to localize features in
the spatial-frequency domain [21]. In our proposed framework, I f (x, y) = g(x, y) ∗ I0 (x, y) (4)
we adopt the Gabor filter as an important component of
the pre-processing block in order to enhance lane marking
features. A 2D complex Gabor filter can be described as: B. Lane Marking Extraction
One of the contributions of this paper is a lane marking
x ′2 + γ 2 y′2 x′
g(x, y) = ex p(− )ex p(i(2π + ψ)) (2) extraction algorithm presented here. Existing lane detection
2σ 2 λ methods such as [22], [23] and [24] apply threshold operations
where x ′ = xcosθ + ysinθ and y ′ = −xsinθ + ycosθ; on a filtered color or grayscale image to convert it into a
Parameters x and y are the pixel coordinates in an image; binary form. The issue is that converting to a binary form
λ is the wavelength of the sinusoid; θ is the orientation of at this stage would eliminate important visual patterns. In the
the Gaussian in the x-y plane; ψ represents the phase offset proposed method, the lane marking extraction operates directly
and it controls the intensity of the kernel; σ is the standard on the grayscale image. The intensity plot of a single row
deviation of the Gaussian envelope, and γ is the parameter and multiple rows from the Gabor filtered image is shown in

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9005

lane marking. This process is repeated for the remaining

pixels’ positions in A1 (Lines 14-22). The reason we
introduced two thresholds (the minimum and maximum
acceptable thickness of the lane marking) is that when
transforming the image to a bird’s eye view, the lane
markings around the top area of the ROI get stretched and
blurred. This effect is more pronounced when the slope of
the road changes. Consequently, the thickness of the lane
marking appears thicker in that region. This uncertainty
can be handled by checking if lane marking thickness is
within the range of the two introduced thresholds. At the
end of this Step, a list of pixels’ positions representing
the edges of the potential lane markings is obtained and
stored in A2 . The vector A1 can then be emptied to release
memory.
• Step 3: For each two successive pixels in A2 , calculate
their average intensity from the Gabor filtered image, i.e.
Iavg = average[I0 (A2 (i)) : I0 (A2 (i + 1))], as shown
in Figure 6(d). If the average intensity is higher than
Fig. 5. Top: The intensity plot of one of the rows of the image in Fig. 4. the defined intensity threshold value T2 , then potential
Bottom: The 3D plot of 100 × 512 image section to visualize the pattern lane marking edges are considered found. Therefore, the
created by Gabor filter.
pixels’ positions representing the edges are saved in a new
list A3 (Lines 24-31). When this process is completed, the
Figure 5 to visualize the pattern created by the Gabor filtering. list A2 can be removed to release memory. Then, based
As shown in Figure 5, the markings are enhanced by the Gabor on the new list, A3 , lane marking pixels can be placed
filter while the adjacent regions are suppressed to nearly zero on a new image. At this stage, all potential lane marking
intensity level. The proposed technique uses this pattern to pixels’ positions in the image are extracted and placed in
extract lane markings in the following three steps: a new image I f for further analysis and visualization as
• Step 1: Applying Gabor filter to an image and looking shown in Figure 6(e).
at the intensity of pixels at each row, the lane markings The pseudocode of the lane extraction procedure is
are potentially located at intensity spikes (See spikes in described in Algorithm 1. For the filtered image given
Figure 5 for a real image given in Figure 4 and the spikes in Figure 4, the output of the lane marking extraction
in Figure 6.b for a synthetic image given in Figure 6.a). algorithm, I f , is shown in Figure 7.
To limit our search scope, we first need to carefully
exclude the pixels that cannot represent the lane mark-
ings. For this purpose, for each row of the filtered image, C. Post-Processing
we collect the location of each pixel with an intensity In the post-processing block, we aim to refine the extracted
value less than a predefined threshold, T1 (Lines 7-13). lane marking pixels by (1) analyzing a group of pixels in
This information, stored in the vector A1 , indeed includes a defined window whether they belong to the same lane
the pixels that cannot be on a lane marking. To improve boundary (2) verifying that lane marking locations do not
the detection reliability, the threshold value T1 is set significantly deviate with respect to the previous image frame,
to a small non-zero value as the Gabor filtering may and (3) fitting a curve to the resulted lane boundaries, as it
not necessarily put the surrounding pixels’ intensity to will be explained next.
zero depending on the texture of the road as shown in 1) Lane Marking Analysis: The extracted lane marking in
Figure 6(b). the previous block is analyzed for identifying lane lines. The
• Step 2: Calculate the horizontal distance between two lane marking extraction algorithm uses lane marking thickness
consecutive pixels’ positions in A1 , collected in the first and intensity parameters as a measure to find pixels belonging
step. If the calculated thickness is in the range of two to lane boundaries. However, due to shadows, road signs,
threshold values, L Mmin ≤ thick ≤ L Mmax , then and other factors, the extracted pixels may not all belong
these two points are potentially part of the edges of to the lane lines. In the post-processing block, therefore,
lane marking, where L Mmin and L Mmax represent the we eliminate or minimize pixels that do not belong to lane
minimum and maximum acceptable thickness of the lane markings by analyzing the distribution of the extracted pixels’
marking. In this case, the information about these two positions. This is done by dividing the image into smaller
pixels’ positions is stored in a new vector, A2 . As shown sections/windows with a m number of rows, where m can be
in Figure 6(c), only the distance between the two left any number smaller than the width of the image. The smaller
and the two right points, highlighted by red circles, meet the value of m, the more accurate will be the analysis results,
the thickness requirements, and hence, these points can though it increases the computation cost. Dividing the image
be selected as a potential representation of the edge of into small windows and starting from the bottom of the image,

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9006 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

Fig. 6. Lane marking extraction illustration for Algorithm 1. (a) A synthetic Gabor filtered image with two lane boundaries, (b) Intensity plot of a single
row, yi , and the threshold T1 where all pixels’ coordinates with intensities lower than T1 are placed in vector A1 (as explained in Step 1 and indicated in
Lines 7-13 of Algorithm 1). (c) filtering based on thickness of lane marking by iterating through the elements in A1 and calculating the distance between
consecutive pixel coordinates for each row (as explained in Step 2 and indicated Lines 14-22 of Algorithm 1), (d) filtering based on average intensity of pixels
using the intensity threshold T2 (as explained in Step 2 and indicated and Lines 24-31 of Algorithm 1), and (e) the filtered image I f .

the number of pixels and the average location of pixels in deviation margin should be adjusted based on the flatness of
each window are calculated. Using the distribution information the road and the amount of curvature of lanes. For example,
(number of pixels and average position), pixels deviating from on highways, the road is generally flat with low curvatures.
the average position higher than a threshold will be removed. However, in road sections with high curvatures, the lane lines
The number of valid pixel counts in each window determines far from the vehicle can significantly deviate in consecutive
whether the pixels in that particular window belong to a lane frames.
boundary or not. This is done for the left and right boundary 3) Curve Fitting: In the previous stage, the lane markings
lines. At this stage, the detection of the pixels for the left and in a defined ROI are extracted, analyzed, and verified. In the
right lane lines is completed. The certainty of detection is also Curve Fitting and Validation stage, a least-square regression
quantified based on the total number of pixels found in each method is used to fit a second-order polynomial for each lane
window. The final analysis is a comparison between the left boundary as:
and right line detection status. If one of the lines is detected (
and the other one is missing or is detected with very low L l (y) = al y 2 + bl y + cl
(5)
certainty, then the missing line will be estimated by shifting L r (y) = ar y 2 + br y + cr
the pixels of the other line with higher certainty by an average
lane width. where a, b, and c are the coefficients of the polynomial.
2) Verification of Lane Marking Locations: In the Verifi- Although the RANSAC method in [24], is a common
cation of Lane Marking Locations stage, previous detection method to fit curves from data samples with potential outliers,
information is used to verify the lane boundaries that were its computational cost (number of iterations for curve fitting) is
obtained at the Lane Marking Analysis stage. Usually, the not fixed. Therefore, to reduce the computation cost, since the
lane line position does not change much on consecutive outlier pixels are already eliminated in the preceding block,
frames especially when the camera’s frame rate is high [25]. we simply apply a direct least-square curve fitting instead of
Therefore, the location of lane line pixels in the current frame the RANSAC method. For further verification, we compare
is approximately the same as the previous pixel locations the newly generated polynomial with a polynomial from the
with some deviations. The Verification block calculates the previous frame to check the consistency of the lines. In some
error between the detection (location of the detected line) of the image frames, the number of extracted lane marking
in the current and previous frames. If the detection in the pixels is not enough to fit a consistent lane curve. In such
new frame is outside the expected region by a high margin, cases, we add some pixels around the starting points of the
it will be ignored otherwise the detected line will be averaged lane lines to give more weight to the lower half of the image.
with previous detections to get the final result. The acceptable This ensures that the lane lines on the lower half of the image

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9007

Algorithm 1 Lane Marking Extraction

Input : Gabor filtered image, I0
Output: Image with lane markers, I f ,
Lane marker coordinates, A3
1 A1 = [ ], A2 = [ ], A3 = [ ]
2 I f = zer os(si ze(I0 ))
3 T1 = Intensity_threshold1
4 T2 = Intensity_threshold2
5 L Mmin = minimum lane marking thickness
Fig. 8. The final result of the lane detection after postprocessing: Left: The
6 L Mmax = maximum lane marking thickness lane output in the bird’s eye view, Right: Detected lanes mapped to perspective
7 for y = 0 : r ows(I0 ) − 1 do view where the blue line indicates the lane center computed from the left and
8 for x = 0 : columns(I0 ) − 1 do right lane boundaries.

9 if I0 (x, y) < T1 then

A1 ← [A1 : (x, y)]
i.e., pck = (xck , yck ) = ( L l (yk )+L r (yk )
10
, yk ), where pck is the
11 end 2
average center point corresponding to sample lane boundary
12 end
points taken at row k. The lane center for the current frame can
13 end
be represented in discrete form by Pct = { pc1 , pc2 , . . . , pcn }
14 for n = 0 : length(A1 ) − 2 do
where n is the number of samples taken from the lane
15 if y A1 (n+1) = y A1 (n) then
boundary curves. From the sample points, a second-order
16 thick = x A1 (n+1) − x A1 (n)
polynomial can be obtained, P t = a t y 2 +bt y +ct by applying
17 if L Mmin ≤thick≤L Mmax then
curve fitting. To get a consistent curve, the final lane center is
18 A2 ← [A2 : A1 (n)]
calculated as the weighted average of data from the last three
19 A2 ← [A2 : A1 (n + 1)]
frames as p̂ck = (w1 P t (yck )+w2 P t−1 (yck )+w3 P t−2 (yk ), yck )
20 end
when sampling data at row k, k = 1, · · · , n, where w1 , w2 ,
21 end
and w3 , are weights assigned to frames at time t, t − 1 and
22 end
t − 2, respectively, and w1 > w2 > w3 to provide higher
23 delete (A1 )
weight for recent frames. In our implementation, we used
24 for i = 0 : length(A2 ) − 2 do
w1 = 0.60, w2 = 0.30, and w3 = 0.10, which are determined
25 Iavg = average[I0 (A2 (i)) : I0 (A2 (i + 1))]
experimentally. Finally, the lane center line is transformed
26 if Iavg > T2 then
from the image frame to the vehicle coordinate frame using
27 A3 ← [A3 : A2 (i)]
the calibration information of the camera as:
28 A3 ← [A3 : A2 (i + 1)]
29 I f [A2 (i) : A2 (i + 1)]=I0 [A2 (i) : A2 (i + 1)] xv = (H − yi )
D
30 end H
31 end Lm
yv = (W/2 − xi ) (6)
32 delete(A2 ) Lp
33 return I f , A3
where H is height of the image; W is the width of the image;
L p is the lane width in pixels; L m is the actual lane width
in meters; D is the look-ahead distance in meters equivalent
to H in pixels, and (xi , yi ) is the pixel position in the image
frame and (xv , yv ) is its corresponding location in the vehicle
coordinate system. These parameters are symbolically shown
in Figure 9.
The output, Pr = {(xv1 , yv1 ), (xv2 , yv2 ), . . . , (xvn , yvn )},
of the Path Generator block is a collection of n points sampled
from the lane center curve in the vehicle frame, where 3 ≤
n ≤ H.
Fig. 7. Lane marking extraction algorithm output represented by I f .
V. D ESIGNING A V ISION -BASED M ODEL P REDICTIVE
remain stable and consistent with the previously generated C ONTROLLER
polynomial. The final result of the lane detector is shown in This section explains our developed MPC for tracking the
Figure 8. path generated by the vision part of the system. The role of
the MPC is to generate commands for controlling the vehicle’s
IV. PATH G ENERATION lateral and longitudinal dynamics so that the vehicle closely
Once the lane boundaries are calculated, the Path Gener- follows the given reference path at a desired speed. More
ation block calculates the center of the lane by taking the specifically, the MPC generates a sequence of steering angle
average of each sample point of the lane boundary lines, and acceleration values to track the path provided by the

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9008 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

Fig. 9. Image and vehicle coordinate systems. For a given point (xi ,yi ) in
image frame, its corresponding location (xv ,yv ) in vehicle coordinate system
can be found using (6), where (0,0) indicates the origin of the respective
coordinate systems.The parameters H , W , and L p are image height, width, Fig. 11. Kinematic vehicle model [26].
and lane width in pixels, respectively.

Fig. 10. The structure of the developed Model Predictive Controller. The
controller generates steering angle (δ) and acceleration (α) commands to be
sent to the actuators through the CAN bus.
Fig. 12. Reference signals yr e f (i) and ψr e f (i), calculated at x(i).

Path Generator while satisfying the physical constraints of

The symbolic representation of these parameters is shown
the actuators. The general structure of the controller is shown
in Fig. 11.
in Figure 10.
Our aim is to design an MPC control strategy to track
The MPC uses a kinematic bicycle motion model to estimate
the generated path Pr = {(xv1 , yv1 ), (xv2 , yv2 ), . . . , (xvn , yvn )},
the state of the vehicle in a defined prediction horizon. Let’s
which was calculated in Section IV. For this purpose, we first
assume that the state of the car is q = [x, y, ψ, v]T , where
fit a second-order polynomial to these points to find the desired
x and y capture the vehicle’s position, ψ is its orientation,
trajectory in the vehicle frame:
and v is its speed, and its input vector is u = [δ, α]T .
From the standard discrete-time kinematic vehicle model for Pr (x) = a ′ x 2 + b′ x + c′ (8)
a front-wheel drive car, q(i + 1) = g(q(i), u(i)), can be
calculated as [26]: Then, to find the yr e f (i) and ψr e f (i), i = 0, · · · , H p − 1,
we first calculate x(i + 1) = x(i) + v cos(ψ(i) + β)dt. Then,
x(i + 1) = x(i) + v cos(ψ(i) + β)dt as graphically shown in Figure 12, from (8), we have:
y(i + 1) = y(i) + v sin(ψ(i) + β)dt yr e f (i) = a ′ x(i)2 + b′ x(i) + c′
v sin(β) d Pr
ψ(i + 1) = ψ(i) + dt ψr e f (i) = tan −1 ( |x(i) ) = tan −1 (2a ′ x(i) + b′ ) (9)
lr dx
v(i + 1) = v(i) + α dt (7) Due to the fact that the desired trajectory is given in the
vehicle frame, as shown in Figure 12, for each iteration, i =
where i indicates the current time step, β = tan −1 ( lLr tan(δ)) 0, 1, · · · , H p − 1, initially we have y(0) = 0, x(0) = 0, and
is the slip angle at the center of gravity (CG) point, lr and ψ(0) = 0. The initial value for the velocity, v(0), is received
l f are distances from the center of gravity of the vehicle to from the vehicle speedometer through the CAN bus. Designing
the rear and front axles, and L = lr + l f . The acceleration, α, and implementing an MPC using the kinematic model in (7)
and the steering angle, δ, are the control inputs of the model. and the reference trajectory in (9), the MPC generates an

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9009

Fig. 13. AggieAuto test vehicle, Lincoln MKZ hybrid, equipped with
required sensing and control systems.
Fig. 14. Hardware components of the developed system.

optimal solution in the form of the sequence of steering angle 150 deg/s and an accelerometer with ± 5g rating, is used to
and acceleration values over the prediction horizon, H p , that measure the vehicle’s acceleration and orientation.
minimizes the lateral position error and the orientation error Data from the camera and IMU sensors are streamed
while driving the car with a desired velocity and fulfilling through a network switch (Gigabit Ethernet with PoE capa-
constraints on the input control signals. The control signals bilities from Netronix) to a computing platform where the
can be found through the following minimization: data processing and decision making take place. The com-
H p−1 puting platform is a computer with Intel with quad-core Xeon
processors and 32GB RAM. The control signals are then com-
X
min (w1 ∥y(i) − yr e f (i)∥2 + w2 ∥ψ(i) − ψr e f (i)∥2 +
α,δ municated with the actuators in the car such as the Electronic
i=0
Power Steering (EPS), braking controller modules (EBCM),
w3 ∥v(i) − vr e f ∥2 + w4 ∥δ(i)∥2 + w5 ∥α(i)∥2 +
and Engine Control Module (ECM) through the CAN interface
w6 ∥δ(i) − δ(i − 1)∥2 + w7 ∥α(i) − α(i − 1)∥2 ) (from Dataspeed Inc.). Additionally, the vehicle’s driving
s.t q(i + 1) = g(q(i), u(i)), mode can be conveniently selected by a button press on the
yr e f (i) = a ′ x(i)2 + b′ x(i) + c′ , custom developed switch-box interfaced with the computing
system or graphical user interface with engage or disengage
ψr e f (i) = tan −1 (2a ′ x(i) + b′ ), buttons, allowing the safety driver to switch between manual
y(0) = x(0) = ψ(0) = 0 and v(0) = v(t), and autonomous modes.
δmin ≤ δ(i) ≤ δmax ,
αmin ≤ α(i) ≤ αmax B. Software Structure of the Testbed
δ(−1) = δ(t − dt), We used ROS melodic distribution to implement the soft-
α(−1) = α(t − dt), (10) ware modules of the developed perception and control system.
The overall software structure including ROS nodes and topics
where H p is the prediction horizon, δ(−1) is the input is shown in Figure 15. The ROS topics shown in Figure 15
applied at t − dt, i.e., the previous sampling step, dt is the are mostly custom message types except the topics from
sampling time, αmin and αmax are the minimum and maximum the IMU and the Camera nodes which are published by the
acceleration, δmin and δmax are the minimum and maximum vendor and community-contributed ROS packages with some
steering angle and steering angular rate, and w1−7 are the customization for our implementation convenience.
weighting factors.
Solving (10) over the prediction horizon H p , we find δ ∗ (i)
C. Experimental Setup
and α ∗ (i), i = 0, · · · , H p − 1, and we apply δ(t) = δ ∗ (0) and
α(t) = α ∗ (0) as the control signals to the actuators. In addition to the main hardware components shown in
Figure 14, a GPS sensor from SwiftNav is also installed
on the vehicle to record the vehicle’s path while driving
VI. E XPERIMENTAL R ESULTS
autonomously. The GPS is linked to a base station through a
A. Hardware Structure of the Testbed radio signal to achieve real-time kinematic (RTK) positioning
The proposed approach is implemented on a Lincoln MKZ of the vehicle’s location with higher accuracy (< 8cm location
hybrid as shown in Figure 13. The hardware structure for the accuracy). The experimental setup including the test vehicle,
developed autonomous vehicle is shown in Figure 14. A single the secondary GPS systems, and its base station is shown in
front-facing camera is used for lane detection. This is a color Figure 16. The test track is a curvy road of about 525 meters
camera by Leopard Imaging Inc. configured to run at 36 fps (1722 feet). The top view of the test track and its elevation
streaming image frames of resolution 1920 × 1080 pixels. profile is shown in Figure 17. The ground-truth path for the
It is mounted outside the vehicle, in the middle of the test track is carefully created by manually recording the GPS
top roof rack. An inertial measurement unit (IMU), Epson coordinates of the centers of the lane using the SwiftNav GPS
M-G320, containing a gyroscope with a maximum angular rate module in the RTK mode.

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9010 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

Fig. 18. Sample labeled images: Left: Label for a lane with both left and
right lane markings available, and Right: Label for a lane with only left lane
marking where the right boundary is virtually added based on lane width
information or curb references.

Fig. 15. Software structure of the proposed system. The boxes are the ROS
nodes in our implementation and the labels on each arrow connecting the
nodes indicate the topics published or subscribed by the nodes.

Fig. 19. Performance evaluation in the perspective view. Estimated lane

boundaries are first transformed from a bird’s eye view to a perspective view
since ground truth is available in the perspective view. Then, the error (offset)
between the ground-truth points and points on the detected lane boundary is
estimated in the perspective view.

Fig. 16. Experimental setup which includes a test vehicle deployed on

a curved forested rural AV test track, and the SwiftNav GPS unit with
on the detection of the other boundary and shifting it by an
radio receiver mounted on the top of the vehicle for evaluation of the average lane width as shown in Figure 18.
controller performance. A base station is placed on the side of the road with To compare the output of our lane detector with the ground-
a surveyed/known static location to provide accurate position information.
truth data, we convert the estimated polynomial in (5) from
the bird’s eye view to the perspective view to be able to
compare it with the ground-truth in the perspective view. This
evaluation process is shown in Figure 19. Mathematically,
we first discretize the estimated polynomial in (5) by finding
n points Pbev = {(xbev1 , ybev1 ), · · · , (xbevn , ybevn )}, where
xbevi = L(ybevi ) = aybev 2
i
+ bybevi + c, for i = 1, · · · , n.
We then transform each sample point to the perspective view
using the inverse transformation matrix, H −1 , as:
[x pr di , y pr di , 1]T = H −1 × [xbevi , ybevi , 1]T (11)
Then, for Pbev = {(x pr d1 , y pr d1 ), · · · , (x pr dn , y pr dn )}, we fit
a second-order polynomial representing estimated lane bound-
Fig. 17. Top view of the forested rural AV test track and its elevation
(altitude) profile showing several turns and steep slopes. ary in the perspective view, given by
L ′′ (y) = a ′′ y 2 + b′′ y + c′′ (12)
D. Lane Detection Performance
To evaluate the performance of the lane detector, we labeled where a ′′ , b′′ , and c′′ are the coefficients of the polynomial.
1225 image frames from the Caltech lane dataset [24] in the The error between a ground-truth sample point and its
perspective view as a ground truth. The dataset was collected associate predicted point on L ′′ (y) along the horizontal axis
from Washington and Cordova streets in Pasadena, California (row) can now be computed as:
at different times of day by Caltech’s autonomous vehicle team Err ori = |x gr t i − x pr d i |, i = 1, · · · , k (13)
(Alice) for the famous DARPA Urban Challenge in 2007. The
label for each lane boundary contains up to 10 discrete points where (x gr t i , ygr t i ) ∈ Pgr t is a ground-truth point and
i.e. Pgr t = {(x0 , y0 ), . . . , (xk , yk )}, where k ≤ 10. Our labels (x pr d i , y pr d i = ygr t i ) is its associated point on the detected
also include virtual lane boundaries corresponding to the road lane in perspective view on L ′′ (y), i.e. x pr d i = L ′′ (ygr t i ) =
segments with only a single lane boundary on the left side a ′′ ygr t i 2 + b′′ ygr t i + c′′ .
of the lane. These labels are required to fairly evaluate our For a lane boundary to be considered correctly detected
lane detector which completes missing lane boundaries based or true positive (TP), more than 70% of the predicted

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9011

TABLE I TABLE III

E VALUATION ON THE C ALTECH DATASET T HE E VALUATION OF S OME OF THE SOTA L ANE D ETECTION M ETHODS
ON O UR DATASET W ITH N IGHTTIME D RIVING S CENES ,
C ONTAINING 201 S ELECTED I MAGES

our method and similar methods reported by [25] on the

same dataset where our proposed method performs better
and consistently regardless of change of scenes. We also
assessed the performance of the developed method on our
dataset1 containing nighttime driving scenarios with glares
from other car’s headlights. The performance, as indicated
Fig. 20. Sample outputs of the proposed detector on the Caltech dataset in Table III), remains unaffected as long as the ego car’s
showing a robust performance for various challenging scenarios such as headlights are on. A comparison with some of the state-of-the-
shadows from trees, road texture changes, and other road sign interference. art (SOTA) methods (such as CLRerNet [27], SCNN [28] and
Red colored lane boundary indicates that it is obtained by shifting the other
lane boundary by an average lane width. ERFNet [29]), also shown in Table III, demonstrates that the
proposed lane detection method has comparable performance
TABLE II (95% accuracy). Additionally, we evaluated our method on
P ERFORMANCE C OMPARISON ON THE C ALTECH DATASET the TuSimple test set [30], which includes 2782 frames. This
dataset is widely used for lane detection benchmarking and
includes images with various road conditions, such as varying
road textures, unrepaired lane markings, and reflectors as
lane boundaries. The achieved performance(96% accuracy),
as shown in TableIV, is consistent with the performance
on our dataset and the Caltech dataset. This demonstrates
the efficiency of the proposed method in detecting ego-lane
boundaries for lane-keeping applications. In our evaluation,
points need to have an error less than a defined threshold we utilized pre-trained models with the TuSimple and CULane
(in our case 15 pixels, which is approximately the same as datasets [31], and testing was performed on the TuSimple
the thickness of double yellow lines) with respect to their and our own test sets. As a result, models initially trained
associated ground-truth points. Otherwise, the prediction is on the CULane dataset may exhibit poor performance as the
considered incorrect or false negative (FN). Estimates of lane models require some form of adaptation. For instance, VGG16
lines by the lane detector without a matching ground-truth and ResNet34 models, trained on the CULane dataset, failed
label are counted as false positives (FP). Evaluation on to perform on the TuSimple test data as shown in Table III
the Caltech dataset shows average detection accuracy (TP) and Table IV. In contrast, our method performed consistently
of 95% and below 1% false positive rate (FP) as shown across these datasets. The ResNet34, VGG16, and ERFNet
in Table I despite various challenges in the dataset such models are adapted to perform lane classification tasks with
as strong shadows, varying lane widths, and several image classifier branch/head [31]. The results presented in Table III
frames without right side lane markings/boundaries as demon-
strated in Figure 20. Table II shows the comparison between 1 https://github.com/ACCESSLab/vision_control

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9012 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

TABLE IV TABLE V
T HE E VALUATION OF S OME OF SOTA L ANE D ETECTION M ETHODS ON T HE C ONSTRAINTS ON THE C ONTROL I NPUTS
THE T U S IMPLE DATASET, W HICH C OMPRISES 2782 I MAGE F RAMES

TABLE VI
T HE T UNED C ONTROL PARAMETERS

library to handle the multi-threaded/parallel processing. The

lane detector is also integrated with the MPC in our test
vehicle to evaluate the real-time performance. The average
computation time of the MPC including a least-square curve
fitting is 20 ms (≥ 50 Hz). Since the lane detector and the
controller are implemented as separate nodes, they can run
in separate cores/threads with ROS configuration for parallel
operations. Although the Path Generator and the Controller
modules as shown in Figure (1) are inherently serial in terms
of data flow, the Controller module can run independently for
a few milliseconds in the prediction horizon using the same
path data (about 30 meters) while the Path Generator updating
path data at 25Hz. Therefore, with parallel implementation of
the lane detector and the MPC, the experimental result for the
integrated system shows that the real-time performance was
unaffected and remained at 25 Hz.

F. Path Tracking Performance

The MPC described in Section V has been used to track
Fig. 21. Sample outputs of the proposed detector on nighttime driving scenes
with glares and low-intensity challenges. the reference path produced by the Path Generation block.
We used the Interior Point Optimizer (Ipopt), an open-source
package, for solving the nonlinear optimization problem
and Table IV do not include the FP value as all the test defined in (10) using the vehicle model in 7. The model
images have ground truth lane boundaries. Therefore, the FP parameters for the Lincoln MKZ are measured as L = 2.85m
value is 0. and lr = 1.5m. The constraints on the control inputs and
the tuned parameters of the designed MPC are provided in
Tables V and VI, respectively.
E. Run-Time Efficiency The path tracking result of the MPC in our test vehicle is
The proposed lane detector is implemented in C++ using shown in Figure 22, over the 525 meters curved test track
the OpenCV library support as a separate ROS node. We tested indicating a very good tracking performance (with less than
the proposed method on a computer platform having 4 CPU 0.2 m lateral error) of the integrated lane detection and MPC
cores of which 2 cores are allocated for the path generator blocks. Figures 22, 23, and 24 show the position, orientation,
module. The achieved average speed was 25 fps for input and speed of the car with respect to the reference values.
images of resolution of 960 × 540 pixels. We experimentally As can be seen from these figures, the car has followed the
determined the number of CPU cores to get the maximum lane center while smoothly reaching the set speed of 10.6 mph
speedup letting the OpenCV package backed with Intel’s TBB (4.73 m/s) and has remained close to 10.6 mph except when

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9013

Fig. 22. Path tracking output of the MPC where the red color indicates the Fig. 25. Steering angle and acceleration values generated by the MPC to track
actual path and the blue curve indicates the reference path in the x-y plane. the reference path. The steering angle and acceleration values are constrained
Vehicle offset, ed , from the reference path (lane center path) is shown in to be in between −42◦ to 42◦ and −2.5 m/s 2 to 2.5 m/s 2 , respectively.
green. The maximum offset from the lane center is less than 0.25 m for the
525 m test track.

Fig. 26. Speed tracking performance of the MPC. In this test, the reference
Fig. 23. Vehicle’s orientation (ψ -yaw) versus time. The reference orientation speed was changed from 5mph to 10mph and then to 15mph to evaluate
is calculated using (9) at x(0) = 0 whereas the actual orientation of the vehicle the longitudinal performance of the controller. The result is consistent with
is 0o with respect to the vehicle coordinate frame. Figure 24 except that in this experiment the vehicle was driving on a road
with a 7.5% slope increase.

slope drop and speed drop of over 1m/s between t = 45s and
t = 60s due to curvature and slope increment simultaneously.
The control signals, calculated by the designed MPC, are
shown in Figure 25. Considering the constraints given in
Table V, the MPC performed the path tracking while meeting
all constraints where the maximum acceleration is below
0.5m/s 2 to reach the set speed of 10.6 mph (4.73 m/s), and
with a relatively small effort on the steering angle to keep the
car close to the center of the lane. A video of this experiment
is available at https://youtu.be/VlWUibPHMZ4.
In addition, we conducted an experiment to evaluate the lon-
gitudinal performance of the MPC. The experiment involved
varying the reference speed while the vehicle tracked the
reference path. Initially, the speed reference was set to 5mph,
which was then changed to 10mph and subsequently to
Fig. 24. Speed profile of the vehicle indicating the performance of 15mph, as illustrated in Figure 26. The resulting speed profile,
the controller maintaining the reference speed which is set to 10.6 mph also shown in Figure 26, demonstrates the MPC’s ability to
(4.73 m/s). The vehicle speed drop between t = 50s and t = 70s is due to track the desired speed across different road slopes (2% and
curved road section and increased slope by 2% (corresponding to x = 200m
to x = 250m in Fig 22). The speed error ev approaches zero after 3 seconds. 7.5% increase) and curvatures. In this experiment, the path
tracking performance closely matches the experiment results
shown in Figure 22, with an average deviation of less than
the MPC is adjusting the speed to handle road curvature and 0.2 m from the lane center. In summary, Table VII presents
slope changes. For example, the vehicle slightly exceeded the the maximum speed and path tracking errors measured on
set speed between t = 5s and t = 30s due to 7.5% road a 525-meter test track with turns and various slopes. Also,

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
9014 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 25, NO. 8, AUGUST 2024

TABLE VII [8] S. Cheng, L. Li, H.-Q. Guo, Z.-G. Chen, and P. Song, “Longitudinal
S UMMARY OF C ONTROL E RRORS collision avoidance and lateral stability adaptive control system based
on MPC of autonomous vehicles,” IEEE Trans. Intell. Transp. Syst.,
vol. 21, no. 6, pp. 2376–2385, Jun. 2020.
[9] P. Falcone, H. Eric Tseng, F. Borrelli, J. Asgari, and D. Hrovat, “MPC-
based yaw and lateral stabilisation via active front steering and braking,”
Vehicle Syst. Dyn., vol. 46, pp. 611–628, Sep. 2008.
[10] T. Getahun, A. Karimoddini, L. H. Beni, and P. Mudalige, “A robust lane
marking extraction algorithm for self-driving vehicles,” in Proc. 15th Int.
Conf. Control, Autom., Robot. Vis. (ICARCV), 2018, pp. 1779–1784.
video demonstrations at different speeds are available at [11] A. A. Assidiq, O. O. Khalifa, M. R. Islam, and S. Khan, “Real time
https://github.com/ACCESSLab/vision_control. lane detection for autonomous vehicles,” in Proc. Int. Conf. Comput.
Commun. Eng., May 2008, pp. 82–88.
[12] C. Y. Low, H. Zamzuri, and S. A. Mazlan, “Simple robust road lane
VII. C ONCLUSION detection algorithm,” in Proc. 5th Int. Conf. Intell. Adv. Syst. (ICIAS),
In this paper, a vision-based control system was developed Jun. 2014, pp. 1–4.
[13] D. Neven, B. D. Brabandere, S. Georgoulis, M. Proesmans, and
for autonomous vehicles. Visual information from a camera L. V. Gool, “Towards end-to-end lane detection: An instance segmen-
sensor was used to identify ego-lane boundaries, based on tation approach,” in Proc. IEEE Intell. Vehicles Symp. (IV), Jun. 2018,
which the lane center of the road was estimated. The lane pp. 286–291.
[14] T. Getahun, A. Karimoddini, and P. Mudalige, “A deep learning approach
center information served as a reference path for the vehicle to for lane detection,” in Proc. IEEE Int. Intell. Transp. Syst. Conf. (ITSC),
follow. A nonlinear model predictive controller was designed Sep. 2021, pp. 1527–1532.
and deployed to handle path tracking by generating the optimal [15] Z. Chen, Q. Liu, and C. Lian, “PointLaneNet: Efficient end-to-end CNNs
actuator control inputs in the form of the steering angle and for accurate real-time lane detection,” in Proc. IEEE Intell. Vehicles
Symp. (IV), Jun. 2019, pp. 2563–2568.
acceleration/braking values while respecting the constraints on [16] K. Okamoto, L. Itti, and P. Tsiotras, “Vision-based autonomous path
the control signals. In terms of performance, the developed following using a human driver control model with reliable input-
lane detection technique achieved an accuracy of over 94% on feature value estimation,” IEEE Trans. Intell. Vehicles, vol. 4, no. 3,
pp. 497–506, Sep. 2019.
the Caltech and TuSimple datasets and a run-time efficiency [17] D. A. de Lima and A. C. Victorino, “A hybrid controller for vision-
of more than 25 fps on a computing platform with Intel’s based navigation of autonomous vehicles in urban environments,” IEEE
Xeon quad-core CPU in our test vehicle. The lane detection Trans. Intell. Transp. Syst., vol. 17, no. 8, pp. 2310–2323, Aug. 2016.
[18] J. Guo, P. Hu, and R. Wang, “Nonlinear coordinated steering and
mechanism and the MPC were integrated and tested on a Lin- braking control of vision-based autonomous vehicles in emergency
coln MKZ hybrid vehicle being deployed in a rural test track. obstacle avoidance,” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 11,
The experimental results demonstrated that the vision-guided pp. 3230–3240, Nov. 2016.
MPC was able to smoothly keep the car in the center of the [19] D. A. Forsyth and J. Ponce, Computer Vision—A Modern Approach, 2nd
ed. Pearson, 2011.
lane while respecting the physical constraints of the car. The [20] Camera Calibration With Opencv—Opencv 2.4.13.7 Documentation.
parallel implementation of lane detection and the MPC enabled Accessed: Apr. 21, 2022. [Online]. Available: https://docs.opencv.
us to run the integrated system at the rate of 25 Hz. The overall org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html
[21] D. A. Clausi and M. Ed Jernigan, “Designing Gabor filters for optimal
performance of the proposed method indicates the potential texture separability,” Pattern Recognit., vol. 33, no. 11, pp. 1835–1849,
of a vision-based controller as a solution to path planning and Nov. 2000.
control challenges in environments where GPS is less accurate [22] J. Piao and H. Shin, “Robust hypothesis generation method using binary
blob analysis for multi-lane detection,” IET Image Process., vol. 11,
or unavailable. no. 12, pp. 1210–1218, Dec. 2017.
[23] F. Coskun, Ö. Tunçer, M. E. Karsligil, and L. Güvenç, “Real time
R EFERENCES lane detection and tracking system evaluated in a hardware-in-the-loop
simulator,” in Proc. 13th Int. IEEE Conf. Intell. Transp. Syst., Sep. 2010,
[1] A. Shaout, D. Colella, and S. Awad, “Advanced driver assistance pp. 1336–1343.
systems—Past, present and future,” in Proc. 7th Int. Comput. Eng. Conf. [24] M. Aly, “Real time detection of lane markers in urban streets,” in Proc.
(ICENCO’), Dec. 2011, pp. 72–82. IEEE Intell. Vehicles Symp., Jun. 2008, pp. 7–12.
[2] A. Broggi, M. Bertozzi, A. Fascioli, C. G. L. Bianco, and A. Piazzi, [25] J. H. Yoo, S. Lee, S. Park, and D. H. Kim, “A robust lane detection
“The ARGO autonomous vehicle’s vision and control systems,” Int. J. method based on vanishing point estimation using the relevance of
Intell. Control Syst., vol. 3, no. 4, pp. 409–441, 1999. line segments,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 12,
[3] R. Marino, S. Scalzi, and M. Netto, “Nested PID steering control for lane pp. 3254–3266, Dec. 2017.
keeping in autonomous vehicles,” Control Eng. Pract., vol. 19, no. 12, [26] R. Rajamani, Vehicle Dynamics and Control. Berlin, Germany: Springer,
pp. 1459–1467, Dec. 2011. 2011.
[4] S.-J. Wu, H.-H. Chiang, J.-W. Perng, C.-J. Chen, B.-F. Wu, and [27] H. Honda and Y. Uchida, “CLRerNet: Improving confidence of lane
T.-T. Lee, “The heterogeneous systems integration design and imple- detection with LaneIoU,” in Proc. IEEE/CVF Winter Conf. Appl. Com-
mentation for lane keeping on a vehicle,” IEEE Trans. Intell. Transp. put. Vis., Jan. 2024, pp. 1176–1185.
Syst., vol. 9, no. 2, pp. 246–263, Jun. 2008. [28] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep:
[5] W. Hongbo, C. Li, and Z. Weihua, “Lane-keeping control based on an Spatial CNN for traffic scene understanding,” in Proc. AAAI, 2018,
improved artificial potential method and coordination of steering/braking pp. 1–8.
systems,” IET Intell. Transp. Syst., vol. 13, no. 12, pp. 1832–1842, [29] E. Romera, J. M. Álvarez, L. M. Bergasa, and R. Arroyo, “ERFNet:
Dec. 2019. Efficient residual factorized ConvNet for real-time semantic segmenta-
[6] S. Chaib, M. S. Netto, and S. Mammar, “H∞ , PID and fuzzy control: tion,” IEEE Trans. Intell. Transp. Syst., vol. 19, no. 1, pp. 263–272,
A comparison of controllers for vehicle lane keeping,” in Proc. IEEE Jan. 2018.
Intell. Vehicles Symp., 2004, pp. 139–144. [30] (2017). Tusimple Benchmark. Accessed: Sep. 10, 2023. [Online]. Avail-
[7] K. Lee, S. E. Li, and D. Kum, “Synthesis of robust lane keeping able: https://github.com/TuSimple/tusimple-benchmark
systems: Impact of controller and design parameters on system perfor- [31] Z. Feng, S. Guo, X. Tan, K. Xu, M. Wang, and L. Ma, “Rethinking
mance,” IEEE Trans. Intell. Transp. Syst., vol. 20, no. 8, pp. 3129–3141, efficient lane detection via curve modeling,” in Proc. IEEE/CVF Conf.
Aug. 2019. Comput. Vis. Pattern Recognit., Jun. 2022, pp. 17062–17070.

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.
GETAHUN AND KARIMODDINI: INTEGRATED VISION-BASED PERCEPTION AND CONTROL FOR LANE KEEPING 9015

Tesfamichael Getahun received the B.S. degree in Ali Karimoddini (Senior Member, IEEE) received
electrical engineering from Bahir Dar University, the Bachelor of Electrical and Electronics Engi-
Bahir Dar, Ethiopia, in 2005, the M.Tech. degree in neering degree from the Amirkabir University of
electrical engineering from IIT Bombay, Mumbai, Technology, Iran, in 2003, the Master of Science
India, in 2009, the M.S. degree in robotics and degree in instrumentation and automation engineer-
automation from Aalto University, Espoo, Finland, ing from the Petroleum University of Technology
in 2014, and the Ph.D. degree in electrical engi- in 2007, and the Ph.D. degree in electrical engi-
neering from North Carolina A&T State University, neering from the National University of Singapore,
Greensboro, USA, in 2022. Currently, he serving Singapore, in 2013. He is currently a Professor with
as a Post-Doctoral Researcher at North Carolina the ECE Department, North Carolina A&T State
A&T State University. His research interests include University. He is also the Director of the CR2 C2
computer vision, ADAS, and the development of autonomous vehicles. Regional University Transportation Center, the Director of the NC-CAV Cen-
ter of Excellence on Advanced Transportation Technology, and the Director of
the ACCESS Laboratory. His research interests include control and robotics,
autonomy, resilient control systems, smart transportation, connected and
autonomous vehicles, human–machine interactions, cyber-physical systems,
flight control systems, and multi-agent systems.

Authorized licensed use limited to: Universiti Tun Hussein Onn Malaysia. Downloaded on November 01,2024 at 02:15:05 UTC from IEEE Xplore. Restrictions apply.