StayInLane: Lightweight Lane Detection

System
Dr. S V Hemanth Nuthalapati Harsha Vardhan Peddinti Naisha Reddy
Associate Professor, Student of Computer Science and Student of Computer Science and
Department of CSE, HITAM, Engineering, HITAM, Engineering, HITAM
Hyderabad, India Hyderabad, India Hyderabad, India
21E51A0582 21E51A0592

Shaik Sardar Ahmed Venna Srihaas

Student of Computer Science and Student of Computer Science and
Engineering, HITAM Engineering, HITAM
Hyderabad, India Hyderabad, India
21E51A05A5 21E51A05C5

Abstract — This project presents the development of an efficient computational efficiency with high accuracy. The U-Net model is
lane detection system tailored for low-end devices. Utilizing a deep particularly well-suited for pixel-level segmentation, making it
learning approach, the system employs a U-Net architecture ideal for detecting lane lines even in complex conditions such as
integrated with a pre-trained VGG16 encoder to achieve accurate
curves, intersections, and occluded lanes. By using transfer
segmentation of lane markings in real-time. The primary challenge
addressed by this solution is the limited computational resources of learning with a pre-trained VGG16 encoder, we can extract robust
embedded systems, such as Raspberry Pi and older mobile devices, features with minimal training data, significantly reducing the
which often hinder the deployment of advanced vision-based training time and computational cost. Furthermore, to ensure that
algorithms in automotive applications. By optimizing the model the system runs efficiently on embedded devices, we incorporate
through techniques like model pruning and quantization, we techniques such as model pruning and quantization, which
achieve high performance with minimal hardware requirements. reduce the model’s size and lower the precision of weights to 8-bit
This approach reduces latency and improves frame rates, ensuring
integers. This optimization allows our model to achieve real-time
smooth lane detection even in challenging conditions like poor
lighting or adverse weather. The solution is scalable, offering a inference speeds while maintaining high accuracy, even on devices
cost-effective alternative to existing high-power solutions, thus with limited processing power and memory.
democratizing access to advanced driver assistance systems
(ADAS) for a broader range of vehicles. Our results demonstrate The significance of this project lies in its ability to
robust lane detection capabilities while maintaining low memory deliver an effective lane detection system that democratizes access
usage and high efficiency, making it ideal for real-world to ADAS features, especially in markets where vehicles may not be
applications on resource-constrained devices. equipped with cutting-edge hardware. By optimizing deep learning
models for low-end devices, we can enhance vehicle safety and
Keywords: Lane Detection, U-Net Architecture, VGG16, Real- make advanced driver assistance technologies more accessible to a
time Inference, Embedded Systems, Deep Learning, Model
broader range of consumers, thus contributing to safer roads and
Pruning, Quantization, ADAS, Low-End Devices.
reduced accidents.
I. INTRODUCTION
II. LITERATURE REVIEW
Lane detection remains a cornerstone of safe autonomous
driving systems and advanced driver assistance systems (ADAS),
as it allows vehicles to navigate lanes accurately and avoid Traditional computer vision methods, such as Canny
unintended lane departures. However, achieving reliable lane edge detection and the Hough Transform, have been foundational
detection under real-world conditions, especially on resource- in lane detection systems due to their simplicity and low
limited hardware, remains a challenging task. While high-end computational requirements. These techniques work by detecting
systems can leverage deep learning models with powerful GPUs to edges in grayscale images and fitting lines to estimate lane
deliver impressive results, these approaches are often impractical boundaries. For example, the Canny edge detector identifies
for deployment on lower-end devices due to their high intensity gradients to highlight lane markings, while the Hough
computational demands. Transform fits geometric shapes to detected edges, assuming lanes
are straight or slightly curved.
Traditional methods such as Canny edge detection and
the Hough Transform are widely used in lane detection. While However, these methods exhibit significant drawbacks in
these techniques are computationally efficient, they are highly real-world conditions. They are particularly sensitive to lighting
sensitive to noise, road irregularities, and varying lighting variations, shadows, and worn-out lane markings, leading to
conditions, leading to inconsistent results. For instance, these unreliable performance in challenging scenarios. For instance, they
methods may fail when lanes are obscured by shadows, faded may fail on roads with complex geometries, occluded lanes, or
markings, or during adverse weather conditions, limiting their adverse weather conditions like rain or fog, where lane lines are not
effectiveness in real-world scenarios. To overcome these distinctly visible. This inconsistency in performance highlights the
limitations, our proposed solution utilizes a deep learning-based need for more robust lane detection systems that can adapt to
approach tailored for low-end devices. Specifically, we leverage a dynamic environments. Our project addresses these challenges by
U-Net architecture with a VGG16 encoder, which balances adopting a deep learning-based approach that leverages a U-Net
architecture with VGG16 as an encoder. By using a solution, suitable for deployment on existing vehicles without
segmentation model, you can capture lane markings at the pixel requiring additional sensors.
level, offering improved resilience to noise, shadows, and poor
While advanced lane detection systems are increasingly
lighting conditions, which traditional methods struggle to handle.
common in high-end vehicles, their adoption in mid-range and
With the rise of deep learning, there has been a shift towards data-
budget-friendly vehicles remains limited due to cost constraints.
driven approaches for lane detection. Convolutional Neural
Most existing solutions require expensive hardware like GPUs or
Networks (CNNs) and advanced architectures like Spatial CNNs
specialized sensors to achieve real-time performance, limiting their
(SCNN) and U-Net have shown significant improvements over
accessibility. There is a growing need for efficient ADAS solutions
traditional methods. These models are capable of learning complex
that can be integrated into vehicles with standard cameras and low-
features and spatial patterns, which allows them to detect lanes
power processors. The primary focus of your project is to
even in non-ideal conditions, such as during nighttime driving or
democratize access to lane detection technology by optimizing
on roads with poor lane markings.
deep learning models for low-end devices. By reducing hardware
One notable approach is the SCNN, which captures long- requirements, your solution can bring advanced ADAS features to
range spatial dependencies across an image, making it well-suited a broader market, enhancing vehicle safety across different socio-
for lane detection in scenarios where lanes are partially occluded or economic segments. This has the potential to significantly impact
curved. However, these deep learning models generally require road safety in regions where cost-effective solutions are crucial for
powerful hardware and substantial computational resources, adoption.
making them impractical for real-time deployment on low-end
devices, especially those with limited processing capabilities. The
III. PROBLEM STATEMENT
proposed system optimizes the deep learning model for low-end
devices by using a lightweight U-Net architecture combined with Lane detection systems are critical components of
model compression techniques like quantization and pruning. Advanced Driver Assistance Systems (ADAS), contributing
This ensures that your model can run efficiently on embedded significantly to vehicle safety by helping drivers stay within their
systems and older hardware without compromising accuracy. lanes and preventing accidents. However, traditional lane detection
To enable the deployment of deep learning models on approaches, such as those based on classical computer vision
resource-constrained devices, researchers have explored various techniques (e.g., edge detection and Hough Transform), struggle
optimization techniques. Model pruning involves removing with reliability under varying environmental conditions. These
redundant parameters, while quantization reduces the precision of methods are often sensitive to changes in lighting, weather, and
the model weights from 32-bit floating points to 8-bit integers. road surface quality, leading to decreased accuracy on worn or
Studies have shown that these optimizations can significantly poorly marked roads. While deep learning-based lane detection
reduce memory usage and computational latency, making it models, like CNNs and U-Net architectures, have demonstrated
possible to deploy complex models like CNNs on devices with improved accuracy and robustness, their high computational
limited processing power. Recent research demonstrates that demands make them impractical for deployment on low-end
optimizing models for embedded systems can achieve real-time devices with limited processing power. This presents a significant
performance without significantly sacrificing accuracy. For challenge, particularly for budget-friendly vehicles where
example, deploying quantized models on microcontrollers has integrating high-performance hardware is not feasible due to cost
shown to maintain high detection rates with a fraction of the constraints. As a result, there is a need for efficient, scalable, and
computational resources required for their unoptimized cost-effective lane detection solutions that can be deployed on
counterparts. existing low-power automotive systems.

By employing model pruning and quantization The proposed project addresses these challenges by
specifically for a U-Net-based Lane detection model, your project developing a lightweight, deep learning-based lane detection model
achieves the dual goals of high accuracy and real-time optimized for low-end devices. By leveraging a U-Net
performance, even on low-power hardware. This ensures that the architecture with transfer learning from a pre-trained VGG16
system remains viable for budget-friendly ADAS solutions, encoder, the solution aims to achieve real-time lane detection with
particularly in developing regions where high-end hardware is less minimal hardware requirements. This system is further optimized
accessible. One of the most significant challenges in lane detection using techniques such as model pruning and quantization, ensuring
is ensuring robust performance across various real-world that it can run efficiently on embedded systems without sacrificing
conditions. Recent studies have focused on improving lane accuracy. The goal is to democratize access to lane detection
detection accuracy in environments with poor lighting, occlusions, technology, making advanced ADAS features accessible to a
and adverse weather conditions. For instance, some researchers broader range of vehicles, thereby enhancing road safety in regions
have used data augmentation to train models on diverse road with limited access to high-end automotive technologies.
conditions, enhancing their ability to generalize. Other approaches
involve integrating sensor fusion, combining visual data with IV. PROPOSED METHODOLOGY
inputs from LiDAR or radar sensors to improve robustness.
However, these solutions typically require additional hardware, The proposed methodology for the lane detection system
increasing costs and system complexity. focuses on developing a robust and efficient deep learning-based
model that can run on low-end devices while delivering real-time
Our project also focuses on enhancing robustness using
performance. The methodology integrates various techniques to
purely camera-based inputs, optimizing the segmentation model to
optimize both the accuracy and efficiency of the system. Below is a
handle variations in lighting and weather conditions. The use of
detailed breakdown of the approach:
transfer learning from VGG16 enables the model to leverage pre-
trained features, reducing training times and improving accuracy 4.1 Model Architecture:
on smaller datasets. This results in a cost-effective and scalable
 The project employs a U-Net architecture combined with a Once the model is trained and optimized, it will be
pre-trained VGG16 encoder. U-Net is a well-established converted into a deployable format using tools like TensorFlow
architecture in image segmentation, known for its ability to provide Lite or ONNX (Open Neural Network Exchange). These
high-quality segmentation maps, which is critical for detecting lane frameworks allow the model to be deployed on a wide range of
markings accurately. The VGG16 model, pre-trained on ImageNet, hardware, including low-end embedded devices. The deployment
is utilized as the encoder to extract essential low-level features process will integrate the lane detection model with the vehicle’s
from road images, which is vital for lane detection in complex camera system, enabling real-time lane marking detection as the
driving environments. vehicle moves along the road. The system will be capable of
processing video frames from the camera, detecting lanes, and
The decoder in the U-Net architecture up samples the feature
providing output to the vehicle's control system for navigation
maps back to the original image resolution to reconstruct the lane
assistance. The Metrics such as accuracy, Intersection over Union
markings. This combination allows the model to be more efficient
(IoU), ROC-AUC curves and F1 score are calculated to assess how
and accurate, utilizing the power of transfer learning while
accurately and reliably the model detects lane markings under
avoiding the need to train a large model from scratch.
various conditions.

Fig. 1. U-Net architecture with VGG-16 Encoder

4.2 Data Collection and Preprocessing

Collecting a diverse set of lane detection datasets from

sources such as TuSimple, CULane, and BDD100K, which
contain annotated lane markings under various conditions (e.g.,
different weather, lighting, and road types). Applying augmentation
techniques such as rotation, flipping, brightness adjustments, and
cropping to increase dataset variability and improve the model's
robustness to real-world conditions. Normalize and resize all
images to a fixed resolution of 256x256 pixels to ensure
consistency in input size for the model.
Fig. 2. Proposed Flow Graph
4.3 Splitting of data

The data and augmented data are split into 70-20-10

format where 70 percent of data is used for training, 20 percent of V. ALGORITHM IMPLEMENTATION
data is used for testing and remaining 10 percent for validation. To
The algorithm implementation focuses on developing a
train a model well, the data must be split in to these categories. lightweight and efficient lane detection system that operates
4.4 Training and Evaluation seamlessly on low-end devices. The process begins with collecting
and preprocessing road images, ensuring the data is prepared for
The training process involves freezing the weights of the training.
pre-trained VGG16 encoder to leverage existing features, which The following subsections outline the core components
and how the algorithm functions in the platform.
will reduce training time and computational effort. The model is
trained using binary cross-entropy as the loss function, specifically 5.1 Data Collection & Preprocessing:
suited for binary segmentation tasks. The Adam optimizer is
employed to ensure efficient gradient updates and faster The first step in algorithm implementation for lane detection
convergence, enabling the model to learn effectively. To ensure involves the collection of high-quality datasets like TuSimple or
that the lane detection model meets the required performance CULane, which are commonly used for training and testing lane
standards, several evaluation metrics will be used such as accuracy, detection models. Once the data is collected, preprocessing is
required to normalize and prepare the images for training. This
Intersection over Union (IoU), and F1 score during training to includes resizing all images to a consistent size of 256x256 pixels
monitor model performance. for uniform input dimensions. Additionally, pixel values are
normalized to a range between 0 and 1 to facilitate model
4.5 Testing and Integration
convergence during training.
 Augmentation techniques such as random rotations, In performance evaluation, key metrics such as Frames
horizontal flips, and brightness adjustments are applied to increase Per Second (FPS) are measured to assess how well the model
the variability of the dataset, helping the model generalize better to handles real-time processing. Additionally, accuracy, Intersection
real-world scenarios by simulating different road conditions and over Union (IoU), and the F1 Score are used to evaluate the
lighting changes. model’s effectiveness in detecting lanes accurately.

5.2 Load the Model (U-Net with VGG16 Encoder) The model is also tested under various challenging road
conditions, including low light, rainy weather, and curved roads, to
To build the model, we begin by loading the VGG16 ensure it performs robustly in real-world scenarios.
architecture, using the pre-trained weights from ImageNet, but
excluding the fully connected layers. This allows us to tackle the VI. RESULTS
powerful feature extraction capabilities of VGG16 while avoiding
unnecessary training on its dense layers. This StayInLane project successfully demonstrates the
development of a light weight lane detection model using a pre-
We then freeze the encoder layers to ensure that these trained VGG16 encoder integrated into a U-Net architecture,
pre-learned features remain fixed during our training. For the U- optimized for low-resource devices like Raspberry Pi. The model
Net decoder, we take the output from the final convolutional layer demonstrates effective performance even under challenging road
of VGG16 as the feature map. The decoder consists of a series of conditions, highlighting its practical potential for real-world
upsampling layers followed by Conv2D layers with ReLU deployment in autonomous driving and driver-assistance systems.
activation, which progressively increase the resolution to help Through techniques such as model pruning and quantization, it
reconstruct the lane mask. operates efficiently on devices with limited computational power.
This work provides a solid strength for future improvements in lane
To prevent overfitting and stabilize the learning process, detection, with the goal of enhancing performance in dynamic
we use Batch Normalization and Dropout layers. The final layer is environments.
a Conv2D with a sigmoid activation, which enables binary
segmentation distinguishing lanes from the background. Finally, VII. FUTURE WORK
we compile the model with binary cross-entropy loss and the Adam
optimizer, ensuring efficient training for accurate lane detection. Several directions can be explored to further improve the
lane detection model. One main area is to enhance the model’s
5.3 Model Training robustness in more complex and dynamic environments, such as
heavy traffic, severe weather conditions, or poorly marked roads.
For model training, we begin by giving key parameters, Additionally, exploring more advanced architectures or hybrid
including batch size, learning rate to ensure gradual learning rate models could improve the accuracy and efficiency of lane
reduction over time. We train the model for 50 to 100 epochs, detection, especially for challenging road geometries. Another
depending on when the model converges. The training data is split method is refining optimization techniques to further reduce
into 80% for training and 20% for validation, allowing for proper computational requirements without compromising performance,
evaluation during the process. enabling broader deployment on a variety of low-resource devices.
Integrating additional sensor data, such as LiDAR or radar, could
Metrics like accuracy and Intersection over Union (IoU) are also complement the visual lane detection, improving overall
monitored to track performance. Early stopping is applied to reliability and performance.
prevent overfitting by halting training if the validation loss stops
improving. REFERENCES
5.4 Model Optimization for Low-End Devices [1] Claudine Badue, Ranik Guidolini, Raphael Vivacqua ˆ
Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino
For optimization on low-end devices, model pruning is used Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao,
to remove less important neurons, reducing the overall model size
Filipe Mutz, et al. Self-driving cars: A survey. Expert
and complexity. Additionally, quantization is applied to convert
Systems with Applications, 2021.
model weights into 8-bit integers using TensorFlow Lite, making
the model more efficient for deployment. [2] Z. Qin, H. Wang, and X. Li, “Ultra-fast structure-aware
deep lane detection,” in The European Conference on
Finally, the model is converted to TensorFlow Lite Computer Vision (ECCV), 2020.
format, enabling it to run on devices with limited resources while [3] Hou, Y.; Ma, Z.; Liu, C.; and Loy, C. C. 2019. Learning
maintaining real-time lane detection capabilities. lightweight lane detection cnns by self-attention
distillation. In CVPR, 1013–1021.
5.5 Inference and Lane Detection [4] A. Gurghian, T. Koduri, S. V. Bailur, K. J. Carey, and V.
N. Murali. Deeplanes: End-to-end Lane position
For inference and lane detection, the pre-trained model is estimation using deep neural networks. In IEEE
loaded onto the device, such as a Raspberry Pi or an older Conference on Computer Vision and Pattern Recognition
smartphone, to enable real-time processing. The next step is to Workshops, pages 38– 45, 2016.
capture frames from the vehicle’s camera as it moves. Each frame
[5] Ioffe, S., and Szegedy, C. 2015. Batch normalization:
is resized to 256x256 pixels, and pixel values are normalized to
Accelerating deep network training by reducing internal
match the input requirements of the model.
covariate shift. In ICML. Jung, S.; Youn, J.; and Sull, S.
The preprocessed frame is then passed through the model 2016.
to generate a binary lane mask that distinguishes lanes from the [6] Pascanu, R.; Mikolov, T.; and Bengio, Y. 2013. On the
background. Post-processing is applied to identify and refine the difficulty of training recurrent neural networks. In ICML.
lane lines from the mask. Finally, the lane mask is overlaid onto the [7] B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J.
original frame, and the results are displayed in real-time, providing Pazhayampallil, M. Andriluka, P. Rajpurkar, T.
continuous lane detection for the vehicle. Migimatsu, R. Cheng-Yue, F. Mujica, A. Coates, and A.
Y. Ng, “An empirical evaluation of deep learning on
5.6 Performance Evaluation highway driving,” arXiv preprint arXiv:1504.01716,
2015.
[8] B. Yu and A. Jain, “Lane boundary detection using a
multiresolution hough transform,” in Proceedings of
International Conference on Image Processing, vol. 2,
1997, pp. 748–751.
[9] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the
importance of initialization and momentum in deep
learning,” in Proceedings of the 30th International
Conference on Machine Learning, 2013, pp. 1139– 1147.
[10] Simonyan, K.; and Zisserman, A. 2015. Very Deep
Convolutional Networks for Large-Scale Image
Recognition. CoRR abs/1409.1556.
[11] H. Xu, S. Wang, X. Cai, W. Zhang, X. Liang, and Z. Li,
“Curvelane-nas: Unifying lane-sensitive architecture
search and adaptive point blending,” in European
Conference on Computer Vision (ECCV), 2020.
[12] B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J.
Pazhayampallil, M. Andriluka, P. Rajpurkar, T.
Migimatsu, and R. Chengyue. An empirical evaluation of
deep learning on highway driving. Computer Science,
2015.
[13] S. P. Narote, P. N. Bhujbal, A. S. Narote, and D. M.
Dhane. A review of recent advances in lane detection and
departure warning system. Pattern Recognition, 73:216–
234, 2018.
[14] K. Zhao, M. Meuter, C. Nunn, D. Mller, S. Mller-
Schneiders, and J. Pauli. A novel multi-lane detection
and tracking system. In Intelligent Vehicles Symposium,
pages 1084–1089,2012.
[15] A. Gurghian, T. Koduri, S. V. Bailur, K. J. Carey, and V.
N. Murali, “DeepLanes: End-To-End Lane Position
Estimation using Deep Neural Networks,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops, 2016, pp. 38–45.
[16] TuSimple. TuSimple Benchmark. [Online]. Available:
https://github. com/TuSimple/tusimple-benchmark
[17] R. K. Satzoda and M. M. Trivedi, “On Performance
Evaluation Metrics for Lane Estimation,” in International
Conference on Pattern Recognition (ICPR). IEEE, 2014,
pp. 2625–2630.
[18] H. Deusch, J. Wiest, S. Reuter, M. Szczot, M. Konrad,
and K. Dietmayer. A random finite set approach to
multiple lane detection. 53(2293):270–275, 2012.
[19] M. Aly. Real time detection of lane markers in urban
streets. In Intelligent Vehicles Symposium, pages 7–12,
2008.
[20] J. C. McCall and M. M. Trivedi, “Video Based Lane
Estimation and Tracking for. Driver Assistance: Survey,
System, and Evaluation,” IEEE Transactions on
Intelligent Transportation Systems, vol. 7, no. 1, pp. 20–
37, 2006. [6] R. F. Berriel, E. de Aguiar, A. F. De Souza,
and T. Oliveira-Sa