ĐẠI HỌC QUỐC GIA TP.

HỒ CHÍ MINH

TRƯỜNG ĐẠI HỌC CÔNG NGHỆ THÔNG TIN

KHOA KỸ THUẬT MÁY TÍNH

<TÊN SINH VIÊN>

ĐỐ ÁN MÔN HỌC THIẾT KẾ HỆ THỐNG SỐ VỚI HDL

<TÊN ĐỐ ÁN MÔN HỌC>

TP. HỒ CHÍ MINH, <NĂM>

 ĐẠI HỌC QUỐC GIA TP. HỒ CHÍ MINH

TRƯỜNG ĐẠI HỌC CÔNG NGHỆ THÔNG TIN

KHOA KỸ THUẬT MÁY TÍNH

<TÊN SINH VIÊN> – <MÃ SINH VIÊN>

ĐỒ ÁN MÔN HỌC THIẾT KẾ HỆ THỐNG SỐ VỚI HDL

<TÊN ĐỒ ÁN MÔN HỌC>

GIẢNG VIÊN HƯỚNG DẪN

<TÊN GIẢNG VIÊN HƯỚNG DẪN>

TP. HỒ CHÍ MINH, <NĂM>

 MỤC LỤC
Chương 1. Introduction............................................................................................3

1.1. Chủ đề cấp độ 2.............................................................................................3

1.1.1. Chủ đề cấp độ 3......................................................................................3

1.1.2. Chủ đề cấp độ 3......................................................................................3

1.1.2.1. Chủ đề cấp độ 4...............................................................................3

Chương 2. Software implementation....................................................................4

2.1. Chủ đề cấp độ 2.............................................................................................4

2.1.1. Chủ đề cấp độ 3......................................................................................4

2.1.1.1. Chủ đề cấp độ 4...............................................................................4

2.2. Chủ đề cấp độ 2.............................................................................................4

2.2.1. Chủ đề cấp độ 3......................................................................................4

Chương 3. Hardware implementation...................................................................5

3.1. Datapath hardware architecture.....................................................................5

3.1.1. Chủ đề cấp độ 3......................................................................................5

3.1.1.1. Chủ đề cấp độ 4...............................................................................5

3.2. Controller hardware architecture...................................................................5

 DANH MỤC HÌNH VẼ
Hình 1.1: Tên hình 1...................................................................................................3
 DANH MỤC BẢNG
Bảng 1.1: Tên bảng 1..................................................................................................3
Bảng 2.1: Tên bảng 1..................................................................................................4
DANH MỤC TỪ VIẾT TẮT
 CHAPTER 1: INTRODUCTION

Lane detection is a crucial component in the field of intelligent transportation

systems (ITS) and autonomous driving technologies. As the demand for safer
and more efficient transportation systems grows, lane detection has emerged
as a fundamental technology that enables vehicles to perceive and interpret
their environment. This capability is essential for the development of advanced
driver-assistance systems (ADAS) and fully autonomous vehicles.

Lane detection involves identifying the boundaries of lanes on roadways using

various sensors and algorithms. The primary goal is to ensure that vehicles can
stay within their designated lanes, thereby enhancing road safety and reducing
the likelihood of accidents. The process typically involves capturing real-time
data through cameras or other sensors, processing this data to detect lane
markings, and interpreting the results to provide guidance for vehicle control
systems.

Over the years, numerous approaches have been developed to address the
challenges associated with lane detection. These approaches can be broadly
categorized into traditional methods and modern techniques. Traditional
methods often rely on image processing techniques such as edge detection,
Hough transforms, and color thresholding to identify lane markings. While
these methods have shown effectiveness in controlled environments, they
often struggle in real-world scenarios where lane markings may be faded,
occluded, or obscured by adverse weather conditions.

In contrast, modern lane detection techniques leverage advancements in

machine learning and deep learning to achieve more robust and accurate
results. Convolutional neural networks (CNNs) and other deep learning
architectures have demonstrated remarkable performance in lane detection

7
tasks by learning complex features directly from raw sensor data. These
models can adapt to varying road conditions and can be trained to recognize
different types of lane markings and road textures.

The integration of lane detection systems into vehicles offers numerous

benefits. By providing real-time lane information, these systems can support a
range of ADAS features such as lane departure warning, lane keeping
assistance, and adaptive cruise control. Moreover, in the context of fully
autonomous vehicles, accurate lane detection is essential for path planning and
navigation, enabling vehicles to make informed decisions and navigate
complex road networks safely.

While Convolutional Neural Networks (CNNs) have revolutionized lane

detection by enabling accurate and robust feature extraction, they come with
notable disadvantages. One of the primary drawbacks is their high
computational complexity, which leads to significant processing time and
power consumption, especially when dealing with large datasets and real-time
applications. This computational intensity necessitates the use of Graphics
Processing Units (GPUs), which are designed to handle parallel processing
tasks efficiently. GPUs provide the necessary computational power to train and
deploy CNN models quickly, making them indispensable for deep learning
applications. However, GPUs are often power-hungry and may not be ideal for
all environments, particularly in embedded systems where power efficiency
and thermal management are critical concerns.

In this context, Field-Programmable Gate Arrays (FPGAs) present a compelling

alternative for implementing CNNs. FPGAs offer customizable hardware
acceleration, allowing for parallel processing while being more energy-efficient
compared to GPUs. They provide the flexibility to optimize the architecture
specifically for the CNN workload, leading to reduced latency and power
consumption. Additionally, FPGAs can be reprogrammed to adapt to new

8
algorithms and updates, providing a versatile solution for evolving lane
detection requirements. Building CNNs on FPGAs can thus achieve a balance
between performance and efficiency, making them suitable for real-time
applications in autonomous driving and other embedded systems where
power and speed are critical.

Based on the initial orientation, here are the outline of the project 1 report:

Chapter 1: Overall introduction

Provide a general overview of the objectives of lane classification, then list and
analyze some existing solution groups, including the research within these
solutions, and discuss their strengths and weaknesses. Then, we choose one
approach that suits all of the proposed criteria most.

Chapter 2: Current issues and proposed solutions

Report the current issues in each section and then discuss carefully all the
possible solutions, then illustrate the final result.

Chapter 3: Final result

Provide the final result of lane detection application on both hardware and
software. Also, provide a hardware resource report, performance of our work.

9
Chapter 1: Overall introduction

1.1. Introduction to lane detection algorithm

A traditional approach to the lane detection problem involves the use of

algorithms designed to extract features such as color [1] [2] or employing
techniques like the Hough Transform [3]. Although they achieve high
processing speed, they often lack flexibility and are susceptible to errors due to
the variability of real-world images, such as low lighting conditions, poorly
marked lane boundaries, and other environmental factors.

In recent years, the field of lane detection has increasingly favored deep
learning methods over traditional approaches due to their superior
performance and robustness in addressing complex real-world driving
environments. Unlike traditional methods that rely on handcrafted features,
deep learning algorithms automatically learn hierarchical features from raw
data, enabling them to recognize lane markings under challenging conditions
such as varying lighting, shadows, occlusions, and poor weather. Additionally,
deep learning models trained on diverse and extensive datasets can generalize
well to various driving scenarios, unlike traditional algorithms that require
extensive parameter tuning and may struggle to adapt to different road types
and conditions. One approach for detecting lanes is the segmentation map.
Below is an example of segmentation map that we created.

10
a segmentation map highlights the exact location and shape of lane lines within
an image, enabling precise delineation of lanes. The use of segmentation maps
in lane detection offers several significant advantages, making it a highly
effective approach for advanced driver-assistance systems (ADAS) and
autonomous vehicles. Firstly, segmentation maps provide pixel-level accuracy,
enabling precise localization and delineation of lane markings. This high-
resolution output ensures that even subtle lane features are detected,
enhancing the reliability of the lane detection system. One of the notable
research on this approach is the U-net[4], Enet[5] and LaneNet[6]. However,
deep learning models used for generating segmentation maps are typically
large and computationally intensive, requiring significant memory and
processing resources. The performance result in [6] produces the output at
50fps, which, while impressive, still poses challenges for real-time deployment,
especially on resource-constrained platforms.

To overcome these issues, recent studies have proposed solutions to this issue
through compact lane representation methods that can incorporate more
geometric description information. For instance, Line-CNN [7] employs Line
Proposal Units (LRUs) to predict lanes using predefined straight lines.
PolyLaneNet [8] transforms lane classification into a regression problem.
Despite achieving exceptionally high accuracy and highly optimized outputs,
these methods typically run on GPUs with high energy consumption levels,
making them less practical for onboard mobile systems in vehicles.

In conclusion, while traditional approaches to lane detection like color-based

algorithms or the Hough Transform offer fast processing speeds, they often
falter in real-world scenarios due to their rigidity and susceptibility to
environmental variables. In contrast, deep learning methods have emerged as
a preferred solution, leveraging their ability to autonomously learn complex

11
features from raw data. Techniques such as segmentation maps exemplify this
advancement, enabling pixel-level precision in lane marking detection crucial
for advanced driver-assistance systems (ADAS) and autonomous vehicles.
However, the computational demands of deep learning models, such as those
generating segmentation maps, remain a challenge for real-time deployment
on resource-constrained platforms. Recent efforts towards compact lane
representation methods aim to reconcile these issues by enhancing geometric
detail without compromising efficiency, though practical integration into
mobile vehicle systems remains an ongoing pursuit.

1.2. Relevant researches

In recent years, there has been a growing emphasis among researchers on the
development of specialized hardware systems tailored to specific applications,
driven primarily by the imperative to enhance system performance. Unlike
general-purpose systems reliant on CPUs for managing diverse tasks such as
lane detection, these purpose-built hardware systems can be meticulously
optimized to efficiently execute specific functions, thus markedly improving
performance metrics and overall system efficacy.

ChipNet [9] represents a CNN module designed for lane detection using LiDAR
sensors, showcasing a departure from traditional camera-based methods. This
approach offers distinct advantages: LiDAR exhibits robustness in varying
lighting conditions, performing effectively under both bright sunlight and
complete darkness, unlike cameras susceptible to glare and shadow issues.
Additionally, LiDAR excels in adverse weather scenarios such as rain, fog, and
snow, where cameras often falter. However, LiDAR entails drawbacks such as
higher costs and greater power consumption compared to cameras, alongside
potentially lower resolution that may omit critical details. Nevertheless, owing
to its high accuracy, reliability, and detailed environmental mapping
capabilities, LiDAR remains a preferred choice for real-time autonomous

12
driving applications. The implementation of this system on FPGA Xilinx
UltraScale XCKU115 achieves a notable frame rate of 79.4 FPS.

RoadNet-RT [10], another CNN offering similar outputs to ChipNet, opts for
cameras over LiDAR to balance performance and power consumption
considerations. To accelerate computations at the convolutional layer,
researchers employ data quantization and enhance multiplication efficiency by
executing two parallel 8-bit multiplications on a single DSP48E2 unit, thereby
achieving an impressive output frame rate of 327.9 FPS. However, in both
RoadNet-RT and ChipNet, the final outputs are presented as segmentation
maps, which impose significant computational demands due to their high
resolution, impacting real-time performance critical for autonomous driving
applications. Furthermore, these outputs are typically data-intensive, posing
challenges in resource-constrained environments for further processing.

QuantLaneNet [11], an enhanced iteration of RoadNet-RT, incorporates lessons

learned from its predecessor, specifically employing data quantization and
dual 8-bit multiplication techniques to optimize hardware performance.
Moreover, the presentation of outputs is refined to a 32x64 matrix format,
amalgamating row-wise classification and vertical range matrices, where each
matrix element signifies the presence (1) or absence (0) of a lane. This
adaptation yields an impressive output rate of 640 FPS (on XCKU115 FPGA)
and facilitates broader applications, such as supporting steering systems.
Nonetheless, the implementation of QuantLaneNet SoC with a DMA/Bridge
Subsystem for PCI Express (PCIe) offers notable advantages. PCIe ensures
high-speed data transfer, crucial for handling substantial data volumes swiftly,
thereby enhancing overall system performance. Additionally, PCIe supports
high bandwidth and low latency communication, essential for real-time lane
detection applications. However, PCIe's drawback lies in its limited flexibility

13
when interfacing the FPGA with other hardware components to enhance
system adaptability and integration.

Below is the graph to summarize the main ideas between three systems.

ChipNet[9] QuantLaneNet[11] RoadNet-RT

[10]

Precision 18-bit quant 8-bit quant 8-bit 16-bit

quant quant

Input size 14 x 64 x 180 3 x 256 x 512 3×176×1216

FPGA XCKU115 XCKU115 ZCU102 VC707 ZCU102

Performance 79.4 631 1029 549 655.80 327.90

(FPS)

Output Segmentatio Row-wise Segmentation

format n map map

1.3. Thesis objectives

n this thesis, we aim to make lane detection systems on FPGA more adaptable
and integrated, ensuring they perform well in real-time applications. To
achieve this, we're expanding on the QuantLaneNet framework. Instead of
using PCIe, which transfers data from PC to FPGA, we switch to Ethernet. This
change improves how the system handles data, making it easier to scale up and
compatible with modern communication setups. It also speeds up data transfer
and reduces delays, making the lane detection system more effective in
practical situations. In this thesis, there are 4 main issues we need to solve:

1. Choose the suitable protocols to send packages via Ethernet. With this issue,
we expect to find some common protocols that are risk-free when transmitting.

14
2. Extract data from Ethernet module and send to a customized IP.

3. Extract data from RAM with lwIP functions.

4. Improve throughput by using DMA between RAM.

TÀI LIỆU THAM KHẢO

[1] K.-Y. Chiu and S.-F. Lin, "Lane detection using color-based segmentation," in
IEEE Proceedings. Intelligent Vehicles Symposium, 2005., 2005, pp. 706-711.

15
[2] Y. He, H. Wang and B. Zhang, "Color-based road detection in urban traffic
scenes," IEEE Transactions on Intelligent Transportation Systems, vol. 5, no. 4, pp.
309-318, 2004.

[3] Z.-H. Chen, A. W. Y. Su and M.-T. Sun, "Resource-Efficient FPGA

Architecture and Implementation of Hough Transform," IEEE Transactions on Very
Large Scale Integration (VLSI) Systems, vol. 20, no. 8, pp. 1419-1428, 2012.

[4]

[5]

[6]

16