Performance Enhancement Method for Multiple

License Plate Recognition in Challenging

Environment
khurram khan  (  [email protected] )
Air University
Abid Imran 
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology
Ha z Zia Ur Rehman 
Air University
adnan Fazil 
Air University
Muhammad Zakwan 
Air University
Zahid Mehmood 
COMSATS Institute of Information Technology - Abbottabad Campus

Research

Keywords: Deep convolutional neural network, automatic number plate recognition, modi ed Census
transform, image morphology

Posted Date: May 5th, 2021

DOI: https://doi.org/10.21203/rs.3.rs-449680/v1

License:   This work is licensed under a Creative Commons Attribution 4.0 International License.  
Read Full License
1

Performance enhancement method for

Multiple License Plate Recognition in
challenging Environment
Khurram Khan 1* , Abid I mran 2, Hafiz Zia Ur Rehman 3, Adnan Fazil1,
Muhammad Zakwan 1, Zahid Mehmood 4
1
Department of Avionics Engineering, Air University, Islamabad, Pakistan
2
Faculty of Mechanical Engineering, Ghulam Ishaq khan Institute of Engineering and
Technology, Pakistan
3
Department of Mechatronics Engineering, Air University, Islamabad, Pakistan
4
Department of Electrical and Computer Engineering, Comsats University, Abbottabad, Pakistan
1*
[email protected]

Abstract

Multiple-license plate recognition is gaining popularity in the Intelligent Transport System (ITS)
applications for security monitoring and surveillance. Advancements in acquisition devices have
increased the availability of High definition (HD) images, which also can capture images of
multiple vehicles. Since License Plate (LP) occupies a relatively small portion of an image,
therefore, detection of LP in an image is considered a challenging task. Moreover, the overall
performance deteriorates when the low-resolution factor combines with an unconstrained
environment, such as night, dusk, and rainy, etc. As it is difficult to locate a small object with
varying conditions in an entire image, this paper proposes a two-step approach for plate
localization in challenging conditions. In the first step, the Faster-Region-based Convolutional
Neural Network algorithm (Faster R-CNN) is used to detect all the vehicles in an image, which
results in scaled information to locate plates. In the second step, morphological operations are
employed to reduce non-plate regions. Meanwhile, geometric properties are used to localize
plates in the HSI color space. This approach increases accuracy and reduces processing time. For
character recognition, the Look-Up Table (LUT) classifier using adaptive boosting with Modified
Census transform (MCT) as a feature extractor is used. Both proposed plate detection and
character recognition methods have significantly outperformed conventional approaches in terms
of precision and recall for multiple plate recognition.

Keywords:
Deep convolutional neural network, automatic number plate recognition, modified Census
transform, image morphology
2

1. Introduction
With the ever-increasing traffic situations in modern cities, the demand for intelligent
transportation systems (ITS) is also increasing rapidly. License plate recognition (LPR) is a
crucial component of ITS, which is used to identify vehicles based on their number plates
(Anagnostopoulos, 2014). LPR includes license plate localization to find the location of the plate
in an image, followed by segmentation and recognition of alphanumeric characters of the
localized plate. Most LPR methods are capable to recognize a single vehicle in an image.
In recent years, there is a considerable increase in problems of traffic congestion, security
monitoring and over speeding in modern cities (Asif et al., 2016). That has increased the demand
for identifying multiple vehicles in an image. A high-resolution camera can monitor multiple
lanes containing several vehicles. However, recognizing multiple vehicles becomes challenging
as some plates will have a smaller size or low resolution (based on distance from the camera),
different background colors, distortions, and different contrast, etc. as shown in Fig.1. Moreover,
HD images also increase computational cost(Shen-Zheng Wang & Hsi-Jian Lee, 2003).
A number of methods have been proposed for license plate recognition over the years, using
template matching (Massoud et al., 2013), artificial neural networks (Ibrahim & Kirami, 2017),
adaptive boosting (Md. Mostafa Kamal Sarke et al., n.d.), Support Vector Machines (SVM) (Ho
et al., 2009), and so on. However, most of these methods performed well in constrained
environments, i.e., uniform illumination and fixed plates size, etc. In the recent past, deep
learning (DL) methods have been widely used as powerful tools for machine learning
applications, i.e., object detection, face detection, and recognition, etc. Multiple Convolutional
Neural Network (CNN) approaches have been used for plate recognition. Cascaded CNN (Radzi
& Khalil-Hani, 2011) has a high computational cost and multi CNN(Gerber & Chung, 2016) does
not perform well for varying parameters of size and angle, etc. Faster RCNN has high speed but
performance deteriorates for small objects (low resolution) plates and non-uniform illumination.
Therefore, it can be concluded that existing approaches do not perform well when exposed to
unconstrained environments like varying illuminations conditions, colored background plates
and variations in plate and font sizes. Therefore, this paper proposes MLPR system capable to
handle all the aforementioned problems. Major contributions of this paper are as follows.

• The proposed technique improves the accuracy of plate detection in challenging

environments having non-uniform illumination conditions and low resolution (based on
distance from the camera). Our technique divides plate detection problem into vehicle
detection and plate localization, which results in scaled information for plate
localization and helps to remove background noise and clutters.

• The proposed plate recognition algorithm does not put a restriction for uniform
light conditions, low resolution or angular plates, etc. Moreover, the character
recognition part is robust to varying illumination, low resolution, different orientations,
and multiple fonts, etc. And experimental results have shown that the character size of
6×9 is recognized effectively.
3

Fig. 1. Korean License plates

In this paper, Faster RCNN is used for vehicle detection followed by plate localization using
morphological operations in HSI color space. Geometric properties of area and aspect ratio of
connected pixels are used for character segmentation. Moreover, this paper uses texture-based
feature extraction method MCT, which is robust to illumination changes and low resolution (Park
& Sim, 2011), with lookup table classifier in boosting framework for character recognition.
2. Related Work

This section briefly introduces the recent advances in Plate recognition systems.

2.1 License plate detection

Most of the existing work on plate detection target a single vehicle in an image. Therefore, the
demand for multiple plate detection has increased considerably owing to an increase in multilane
structure in modern cities. Edge detection methods consider an area with a higher density of
characters as an LP. Combining this property with geometric properties of plates has been widely
used to extract LPs. Vertical edge detection is more robust compared with horizontal edge
detection, which provides inaccurate results owing to errors due to the car bumper area(Shen-
Zheng Wang & Hsi-Jian Lee, 2003). A fast and robust vertical edge detection method was
proposed that increases the speed of detection by eliminating unwanted lines(Al-Ghaili et al.,
2013). Yepez and Ko(Yepez & Ko, 2018) proposed a plate detection method based on only
morphological operations. They developed an algorithm to select appropriate structuring element
(SE) from a set of SEs by training these SEs on the whole dataset. This approach could not perform
well for multiple license plate recognition, due to variation in the size of plates in an image.
In (Hsi-Jian Lee et al., 2004), the block processing method was proposed which detects the
area of LP by finding the maximum edge magnitudes among the blocks. Connected component
labeling (CCL)(Anagnostopoulos et al., 2008) was used in binary images to label groups of
connected pixels and use attributes such as the height/width ratio, and area to localize the plate.
In(Cho et al., 2011) a character-based approach was used to localize an LP by calculating the
distance between the characters on LP. Rizwan, et al (Asif et al., 2016) proposed a method for
detecting Chinese plates by taking advantage of the chromatic component of the YDbDr space
model. And eliminates non-plate regions by using an average of energy map and edge
information of the plate.
Feature extraction and classification techniques have also been used. In (Md. Mostafa Kamal
Sarke et al., n.d.), authors combined AdaBoost with Haar-like features in a cascaded manner for
4

license plate detection (LPD) and genetic algorithms (Sang Kyoon Kim et al., 1996) have been
used to classify and identify plates based on color information by using geometric attributes CCL
for localization. In(Khan et al., 2018) proposed an entropy-based feature selection method,
followed by SVM for classification for plate detection. The proposed method performed
segmentation by identification of the luminance channel and then used Otsu thresholding for
binary segmentation of that channel. This method was only able to produce reasonable results on
a small number of images.
Deep learning architectures have also been used for LPD. CNN has been used in a cascaded form,
where the first CNN classifier searches for any text on the image and 2nd classifier is used to reject
false positives, i.e., any other text from the text on LP (Li & Shen, 2016). H. Xiang (Xiang et al.,
2017) proposed CNN based network that extracts low and high features at several stages to
distinguish details of plate and background followed by three-loss layer architecture for accurate
plate detection. To enhance efficiency, Rafique et al. [26] used the advanced structure of Faster R-
CNN [27], which directly detects the plates in an end-to-end manner. A modified YOLO (Xie et
al., 2018) was used for license plate localization, which had the capability of detecting license
plates that had different variations like rotation, skewness and different orientations, this method
had high computational complexity.
Faster –RCNN with VGG 16 as a feature extraction method, without utilizing the fully connected
layers, was used for LPL (Li et al., 2019). This Deep learning method performed combined
training of LPL and optical character recognition within a forward pass.

2.2 Character Segmentation

Character segmentation is a key step used to isolate characters for recognition. Most popular
methods used geometric properties of area and aspect ratio (Kim et al., 2000), horizontal and
vertical projection methods of characters were used to segment the plates (Otsu, 1979), and also
multiple features were combined to segment the characters of LP (Nomura et al., 2005). A
Convolutional Neural Network (CNN) based two-stage process is proposed(Zhang et al., 2018) to
segment and recognize characters (0–9, A–Z). Tarigan et al.(Tarigan et al., 2017) proposed an LP
segmentation technique consisting of horizontal character segmentation, connected component
labeling, verification, and scaling.

2.3 Character recognition

Similarly, many methods and classification techniques have been proposed for recognition. The
template matching method (Massoud et al., 2013) calculates the correlation between character and
templates and the maximum correlation valued template is considered as a character. However, it
has shown poor performance in variable character size, noise, and rotation, etc. Multilayer neural
network is trained to recognize characters(Ibrahim & Kirami, 2017). The multistage classifier was
used to recognize characters with lower case, upper case, digits, and two-line plates. This
technique’s performance deteriorates with varying illumination and small size of characters (Wen
et al., 2011).
In (Shivakumara et al., 2018), CNN and Bi-Directional Long Short Memory (BLSTM) are
combined for plate recognition. CNN was used as feature extraction due to its high discrimination
ability and BLSTM is capable of extracting context information from past information followed
5

by Dense Cluster Voting (DCV) for classification. Bulan et al (Bulan et al., 2017) proposed a
segmentation and annotation free method for plate recognition. They proposed a two-stage
classifier, which first used a winnows classifier for candidate region extraction followed by CNN
for plate classification. For optical character recognition, a segmentation free approach using
hidden Markov models (HMMs) was proposed. In (Du et al., 2013), a research group developed
a robust ALPR technique in unconstraint environments. In (Yang et al., 2018), researchers
presented a novel architecture for Chinese LPR by cascading CNN and Extremal Learning
Machines (ELMs). CNN is applied for feature extraction and the ELM is used as a classifier,
which yields encouraging results with short training time. In (Radzi & Khalil-Hani, 2011), a
cascaded Recurrent Neural Network (RNN) method integrated with a Short-Term Memory
(STM) is proposed to recognize the sequential features. These features are extracted from the
whole license plate via CNN.
3. Methods
This section describes the architecture of the proposed system. Fig. 2 shows the overall
architecture of the method.

Fig. 2 Flowchart of proposed LPD’s method

6

3.1 Vehicle Detection

Object detection is becoming a complex problem with an increase in applications such as
multiple object tracking, and self-driving cars, etc. Many handcrafted (i.e. HoG and Haar)
features and deep learning (R-CNN (Girshick et al., 2014), Faster-RCNN (Ren et al., 2017),
YOLO (Redmon et al., 2016), etc.) methods were proposed. However, some have slow
processing speed and others have a low accuracy rate. Faster-RCNN has shown the best detection
rates among deep learning object detectors with real-time processing capabilities. However,
Faster-RCNN performance deteriorates for small object detection (LPs in our case). Therefore,
this paper uses Faster R-CNN for vehicle detection, which helps in providing relevant and scaled
information in an image. Moreover, faster RCNN shows excellent results for vehicle detection as
vehicle size is large as compared to the plate’s size in multiple license plate detection cases.
Faster-RCNN is divided into two parts: Region Proposal Network (RPN) that generates a
proposal for vehicles region, followed by fast R-CNN for vehicle/non-vehicle classification and
to efficiently refine the proposal and detect the vehicles. To generate feature maps of the input
image, we employ a pre-trained VGG-16 (K. Simonyan & d A. Zisserman, n.d.) model consisting
of 13 convolutional, 5 max-pooling and FC layers. The feature maps are fed to RPN which scans
each map by sliding window and generates proposals with bound boxes for vehicle region. For
multiple vehicle detection scenarios, the network has to detect vehicles of multiple scales and
aspect ratio as the distance between vehicle and camera vary. In order to deal with variable scales,
anchors in RPN are introduced, which uses three scales (128 × 128, 256 × 256 and 512 × 512)
and 3 aspect ratios (1: 1, 1: 2, 2: 1), which results in 9 anchors at each location. As the size of each
region proposal is different from each other, it is difficult to make efficient architecture for
different sizes.
Region of interest (RoI) pooling simplifies the problem and extracts fixed-size feature
representation. The features from RoI pooling are flattened into vectors. Eventually, these vectors
are fed into two fully connected layers, the one for vehicle/non-vehicle classification based on
each RoI SoftMax probability, and the other for predicting the rectangular coordinates for each
region. In this method, faster RCNN architecture is trained using stochastic gradient descent with
momentum (SGDM) that minimizes the error and quickly updates the weights. SGD uses only one
sample data set from the training data to update weights while the gradient descent method (GD)
must consider all training datasets to update the weights/parameters. The initial learning rate of
0.001 was used for training VGG 16 parameters and 0.0001 for the remaining parameters for 50k
iterations. One image was randomly sampled per batch for training. Each image was resized to
600 and 1400 for shorter and longer sides respectively. Fig. 3 shows the results of vehicles detected
in an image using faster-RCNN.
7

Fig. 3 Test Image with vehicle detection

3.2 License Plate Localization

After successful vehicle detection, the next step is to locate the LP by using morphological
operation in the HSI color space. This color space is known to be closely related to the color
visualization of human beings (Gonzalez & Woods, 2006). The Vehicle area is converted to HSI
color space, which separates the color information from the intensity (Gonzalez & Woods, 2006).
The current approach uses hue information to determine the colored background plates by
defining specific criteria, as ours is to find yellow-green and orange plates. Based on our
experiment, the following criteria proved sufficient for our requirement.

If Hue is between (H > 40° & H < 60°)

Background Color is orange
Else if (H>80° & H<120°)
Background Color is green
and if (H > 60° & H < 80°)
Background Color is yellow

White background plates and monochrome images are located using intensity information of
HSI color space. Fig. 4(a) shows the binary image results intensity channel and Fig. 4(b) shows
the segmentation result of the Hue channel.
8

Fig. 4 (a) Plate with white background (b) Plates with a colored background

After segmentation and binarization, the candidate’s area contains regions of connected pixels.
These connected components are labeled using the ‘4-connectivity labeling’ method so that each
pixel in the connected region has the same label. Edge detection methods have matrix
multiplications that increase computational cost. Therefore, morphological operations have been
used instead of edge detection to reduce computational complexity. After binarization, most of
the large connected groups of background are eliminated. However, there is still undesirable
information in an image, that can affect the accuracy of plate detection as can be seen from Fig.4.
To minimize these areas and to enhance the pixel area of the plate, a morphological operations-
based filtering approach consisting of open and close operation is designed. Morphological Open
and close operations are shown in equation (1) and (2), respectively.

A ∘ SE = (A ⊖ SE) ⊕ SE (1)

A ∙ SE = (A ⊕ SE) ⊖ SE (2)

In multiple plate detection, the size of LPs depends upon the distance of a car from the camera.

Fig. 5 (a) License plate with white background (b) colored background after morphological
Operations
9

Thus, having more than one SE for one task can increase the computational load. After testing
and verifying on a number of test images, an optimum SE was selected. Fig. 5(a) and (b) show
the effect of morphological operations on both the binary images. Since most of the non-LP
regions are removed. Finally, we apply two geometric conditions of area and aspect ratio to locate
the license plate. In multiple license plate detection, plate size will vary depending on their
distance from the camera. Therefore, having a multiple area and aspect ratio values is not an
optimum solution. The experiment on a large number of test images was performed to find
optimum values. Therefore, area values between 1500-4000 pixels and an aspect ratio of 0.2 -0.6
are used in the proposed method for plate localization. A similar process is carried on the
remaining detected vehicles. Fig. 6 shows the overall result of detecting multiple license plates
by the proposed methods.

Fig. 6 Detection results of the Proposed method

3.3 Character Segmentation

The recognition of localized LP now proceeds to the segmentation step. This is a crucial step
as recognition totally depends on how well the characters are separated from one another. In this
study, pixel connectivity in binary images is used for segmenting characters (Wu et al., 2007).

First, the LP regions are converted to binary values by using Otsu’s threshold method (Otsu,
1979). Next, a morphological thinning is performed to reduce the joining between the LP
boundaries and text and in between characters that can negatively impact the process. The
connected components are labeled based on pixel connectivity. The labeled pixels are considered
and pixels having the same area and aspect ratio are detected as characters. Fig. 7 shows
segmented characters of some number plates with varying light conditions, different
backgrounds, and multiple sizes, etc.
10

Fig. 7 Character Segmentation

3.4 Character Recognition

In Multiple plate detection scenarios, the size of plates will vary depending on the distance
from the camera. Therefore, the characters isolated during the segmentation step will also have
variable sizes. Therefore, techniques such as template matching do not perform well due to their
requirement of fixed size. The resolution of characters plays a crucial role in the identification of
characters. Moreover, conventional approaches don't perform well on challenging environments,
and various illumination conditions, i.e. rainy time, dusk time, cloudy and underground parking
images, etc.
For character recognition, AdaBoost with modified census transform (MCT) (Park & Sim,
2011) as a feature extractor is used with a lookup table classifier (LUT). LUT is efficient in multi
Gaussian samples classification, whose sensitivity to a fixed number of bins is suitable for the
character recognition process. Table 1 shows the algorithm for character recognition. Texture-
based analysis plays a vital role in vision-based applications, focusing mainly on how to derive
texture features by taking advantage of neighborhood properties. Local binary pattern (LBP)
computes a local representation of texture by comparison of center pixel to its neighboring pixels
in a defined mask. However, LBP features have shown poor results when the center pixel value
is changed due to varying illumination conditions. Therefore, in our proposed method, MCT
features are used and provided excellent results for texture description in the character
recognition process in changing light conditions. Fig. 8 shows a calculation of the MCT feature
with a 3×3 window from segmented character. MCT features first compute the mean intensity
value of the 3×3 window around that specific pixel. For each pixel in the window, MCT assigns
“1” if the current pixel value is higher than mean value and it assigns “0” otherwise. This binary
value is converted to decimal to obtain the feature value. This integer value represents an
independent local pattern. Therefore, a 3×3 kernel can have a total of 511 feature values.
11

Fig. 8 MCT Feature Extraction

Next, the LUT classifier is used for the classification of MCT features at every pixel location
of the character to produce 511 bin feature indices. 511 bin histogram is created Γ(x) for all
samples in the training set. LUT assigns +1 if positive samples are greater and -1 otherwise, as
shown in (3).

+1 𝑖𝑖𝑖𝑖 𝐿𝐿𝑝𝑝𝑝𝑝𝑝𝑝 (𝑥𝑥, 𝑘𝑘) > 𝐿𝐿𝑛𝑛𝑛𝑛𝑛𝑛 (𝑥𝑥, 𝑘𝑘)

ℎ(𝑥𝑥, 𝑘𝑘) = � � (3)
−1 𝑜𝑜𝑜𝑜ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑖𝑖𝑒𝑒𝑒𝑒

Fig. 9 LUT classifier

Fig. 9 shows an example of the LUT classifier, where rows represent the pattern value and
columns represent the weak classifier candidates. AdaBoost is an iterative method that
sequentially selects a weak classifier pixel location with minimum weighted error in every
iteration of learning. Finally, a strong classifier is constructed from the sum of all weak pixel
classifiers as shown in (4).

𝑇𝑇

𝐻𝐻 (Γ) = � 𝛼𝛼𝑡𝑡 ℎ𝑡𝑡 �𝑥𝑥, Γ(𝑥𝑥 )� (4)

𝑡𝑡=1
12

As character recognition is a multiclass problem, we use one against all classification

techniques to construct k=50 classifiers for 50 classes. Each classifier is trained by taking positive
examples from one class and negative examples from the remaining classes. The output of the
multi-class classifier is activated for class having maximum output among all binary classifiers.
For outputs of multiple binary classifiers, a multi-classifier generates a vector output S as shown
in (5).

𝑜𝑜 = 𝑎𝑎𝑒𝑒𝑎𝑎 max ( 𝐻𝐻 (Γ)𝑖𝑖 )

𝑖𝑖=1…𝑘𝑘
(5)
+1 𝑖𝑖𝑖𝑖 𝑖𝑖 = 𝑜𝑜, 𝑖𝑖 = 1 … 𝑘𝑘
𝑆𝑆𝑖𝑖 = �
−1 𝑂𝑂𝑜𝑜ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑖𝑖𝑒𝑒𝑒𝑒

Table 1 Character Recognition with AdaBoost Algorithm

13

4. RESULTS/ DISCUSSION

In this section, the experimental results describe the effectiveness of the proposed method. A
total of 4179 (resolution 1920 x 1080) images were taken, using 2000 for testing and 2179 for
training purposes, in varying illumination conditions and environments, i.e. (night, day, dusk,
cloudy weather, rainy weather, and parking).
Table 2 compares the results in terms of recall and precision ratio of detection method with
existing methods when applied to images with multiple license plates. Fig. 3 shows the results
for challenging illuminating and weather conditions throughout the day.

Table 2 Comparison with Existing Methods

Method Precision ratio Recall ratio
Edge [31] 80.17% 84.89%
AdaBoost + Haar [13] 85.45% 89.95%
CNN [17] 87.05% 91.45%
Multi -CNN 89.19% 93.31%
Faster RCNN 92.87% 94.97%
Proposed 94.57% 96.72%

The proposed method outperforms conventional methods in terms of both precision and recall.
There were 5543 vehicles in 2000 images used for the testing process. The proposed method
detected 5361 LPs correctly with accuracy (recall rate) of 96.72%. Recall and precision are
defined as follows

𝑁𝑁𝑝𝑝 𝑜𝑜𝑖𝑖 𝑐𝑐𝑝𝑝𝑐𝑐𝑐𝑐𝑛𝑛𝑐𝑐𝑡𝑡𝑐𝑐𝑐𝑐 𝑑𝑑𝑛𝑛𝑡𝑡𝑛𝑛𝑐𝑐𝑡𝑡𝑛𝑛𝑑𝑑 𝐿𝐿𝐿𝐿𝑝𝑝

Recall =
𝑇𝑇𝑝𝑝𝑡𝑡𝑇𝑇𝑐𝑐 𝑛𝑛𝑝𝑝 𝑝𝑝𝑜𝑜 𝑣𝑣𝑛𝑛ℎ𝑖𝑖𝑐𝑐𝑐𝑐𝑛𝑛𝑝𝑝

𝑁𝑁𝑝𝑝 𝑝𝑝𝑜𝑜 𝑐𝑐𝑝𝑝𝑐𝑐𝑐𝑐𝑛𝑛𝑐𝑐𝑡𝑡𝑐𝑐𝑐𝑐 𝑑𝑑𝑛𝑛𝑡𝑡𝑛𝑛𝑐𝑐𝑡𝑡𝑛𝑛𝑑𝑑 𝐿𝐿𝐿𝐿𝑝𝑝

Precision =
𝑇𝑇𝑝𝑝𝑡𝑡𝑇𝑇𝑐𝑐 𝑑𝑑𝑛𝑛𝑡𝑡𝑛𝑛𝑐𝑐𝑡𝑡𝑛𝑛𝑑𝑑 (𝑖𝑖𝑛𝑛𝑐𝑐𝑐𝑐𝑖𝑖𝑑𝑑𝑖𝑖𝑛𝑛𝑛𝑛 𝐹𝐹𝑇𝑇𝑐𝑐𝑝𝑝𝑛𝑛 𝑝𝑝𝑝𝑝𝑝𝑝𝑖𝑖𝑡𝑡𝑖𝑖𝑣𝑣𝑛𝑛𝑝𝑝)

Fig. 10 shows the results for challenging illuminating and weather conditions throughout the
day.
14

(a)

(b)

(c)

(d)
15

(e)

(f)

Fig. 10 Results of images takes (a) sunny day (b) night time (c) dusk (d) cloudy (e) Underground
parking (f) Rainy

The recall ratio of the proposed method is 13% higher than of edge detection method since
the edge method was unable to detect color background number plates. Precision is higher when
compared with the AdaBoost method, as the AdaBoost method also detects headlights and text
as a license plate. Li, et al [17] trained a 37 class CNN system for character detection in images
followed by a CNN classifier as a false positive eliminator. This method also produced more false
positives in real-world scenarios where images had text other than LP. The proposed method
results in an average of 3-10% better than conventional methods in multiple license plate
scenarios.
Character recognition performance is evaluated in Table 3. The proposed recognition method
was tested on all the plates successfully detected. We compared the performance of the presented
method with popular methods. First, a scale-invariant feature transform is used for feature
extraction and a support vector machine is used for classification. The second method is a 3-layer
multilayer neural network for character recognition. The third method is a traditional
convolutional neural network having two convolution layers, 2 fully connected layers followed
by SVM for classification.
SIFT and SVM based method was unable to classify characters due to partial occlusion in
16

rainy images, the effect of vehicle headlights in basement images, and exposure to strong sunlight.
ANN has worst results as broken characters and two font characters on plates were
unrecognizable by this method. As CNN can automatically learn features, it has performed better
than both existing methods. However, its performance degraded for low resolution (based on the
distance of camera) characters. The proposed method outperformed these methods in terms of
accuracy in the challenging condition i.e. varying illumination images as per the results shown in
table 3.
Table 3 Character recognition

Method Precision ratio Recall ratio

SIFT-SVM 84.13% 87.97%

ANN 87.36% 91.32%

CNN 90.05% 94.67%

Proposed 94.57% 98.02%

Fig. 11 shows the comparison of character recognition performance of conventional

approaches with the proposed method for low-resolution character and demonstrates the
superiority of the proposed scheme when compared with the benchmarks.

Fig. 11 Accuracy vs Resolution

Table 4 lists the overall (detection + recognition) performance results obtained under different
lighting conditions and weather conditions. Our dataset consists of images taken during-night
time, daytime, dusk, and cloudy weather. Images taken during cloudy and sunny days produce a
17

better result due to consistent light conditions, except for cases when LPs are affected by the
reflection of sunlight from surroundings. Images of a car parked in the basement also produced
good results with exception to a reflection of other cars’ headlights. Worst results were produced
during dusk time owing to the quickly varying illumination during this time of day.

Table 4 Performance comparison for varying conditions

Weather # of Total Detection rate Character
images Vehicles recognition
Sunny 450 1193 97.98% 98.97%

Night 295 839 97.73% 98.31%

Dusk 397 1120 94.12% 96.43%

Cloudy 438 1175 97.87% 98.45%

Parking 205 589 96.43% 98.01%

Rainy 215 627 95.53% 97.91%

Total 2000 5543 96.72% 98.02%

Moreover, dimming sunlight, street lights, and cars’ headlights have a negative impact on the
overall performance of the method, especially when vehicles are at a far distance from the camera.
Results of images taken in rainy conditions are also encouraging. However, some characters were
not recognized owing to images getting blurred due to water pouring down the windscreens of
the cars containing the camera.

The proposed method performed better than the state of art methods due to the following reasons:

• All CNN-based methods used several down-sampling stages, in which most of the
information on the plate was removed before the detection stage. Hence, CNN methods were
not able to achieve better accuracy. However, in the proposed method, a two-step approach
was used instead of several down sampling approaches. In this two-step approach, the vehicle
is first detected and then its license plate region is detected. This resulted in an improvement
in the accuracy of plate detection in comparison with CNN.
• The existing Faster R-CNN methods used nine anchor boxes of three scales and three
aspect ratios to detect various objects. However, in the current method for vehicle detection
with optimized Faster R-CNN, four different anchor boxes of two scales (256 × 256 and 512
× 512) and two aspect ratios (1:1 and 1:2) were used. This resulted in improving the learning
optimization speed and also significantly reduced the false positives. These two factors helped
to improve the overall precision of the system.
18

• The same color of LP with different values of color components in the day and night hours
makes a difficult condition to handle particularly in extreme cases. However, the proposed
algorithm resulted in the efficient handling of this difficulty. In this application, the HSI color
model has been applied, in which any component of the color can be altered separately without
disturbing others. This feature of the proposed model is very effective in dealing with adverse
conditions such as extreme light conditions, where can be dealt with only altering intensity
component. This characteristic of the current model resulted in achieving high precision even
in challenging conditions.

The proposed license plate recognition algorithm has become more efficient by integrating
boosting based methodology with LDA.

• The developed algorithm for license plate recognition used the LDA, which considers
the within-class and between-classes variance of license plates to extract the features. This
particular approach led to the usage of fewer parameters as compared to using the
character segmentation approach. Also, this integrated approach of boosting based
methodology with LDA to extract features resulted in detailed plate information. The
experiment results have also shown that recognition performance has been boosted
significantly by using this approach.
• This study revealed that accurate identification of the license plates in an uncontrolled
environment is still a challenging task. Although the proposed approach resulted in
effective recognition of the task during different times of the day. However, it is still very
difficult to recognize license plates in highly complicated backgrounds, such as occlusion,
broken plates, and plates with mud.
• The proposed algorithm encompasses no heuristic processes, for instance, the plate
colors or character space, rather it recognizes the labels all at once from the whole license
plate image. Moreover, the proposed algorithm avoids intermediate steps, such as
character separation. This algorithm can be trained and tested with only the image and
labels required for training. Due to the aforementioned characteristics, the developed
algorithm yields high license plate recognition accuracy.

5. Conclusion

In this paper, a multiple license plate recognition method, for high--resolution images, was
presented, which works in challenging illumination conditions in real-time scenarios. The
proposed technique divided plate detection into two steps. In the first step, faster-RCNN was used
to detect all the vehicles in an image resulting in scaled information to locate plates. And
morphological operations were used to reduce non-plate regions and geometric properties were
used to localize plate HSI Color space. Then, character recognition is executed by a LUT classifier
using adaptive boosting with MCT as a feature extractor. Experimental results showed that the
19

detection rate of the proposed method is much higher than existing methods, with an overall
detection rate of 96.72% and a recognition rate of 98.02% in multiple LPs and varying illumination
scenarios. The proposed algorithm might be suitable for real-time ITS applications.

Abbreviations
ITS: Intelligent Transport Systems, RCNN: Region-based Convolutional neural Network,
Adaboost: Adaptive Boosting, MCT: Modified Census Transform, LUT: lookup Table, LDA:
Linear Discriminant Analysis,
Availability of data and material
Some data can be shared on request
Competing interests
The authors declare that they have no conflict of interest.
Funding
No Funding
Authors' contributions
All authors have equivalent contributions.
Acknowledgments
N/A
Author’s Information
Khurram Khan received his Ph.D. degree in Electronic & Communication Engineering from Hanyang
University South Korea. His research interests are image recognition, Computer vision, and Machine /deep
learning.
Abid Imran received his Ph.D. degree in Electronics & systems engineering from Hanyang Univerity South
Korea. His research Interests are Bio-Inspired Manipulators, Robot Manipulation, and machine vision.
Hafiz Zia ur Rehaman received his Ph.D. degree in Mechatronics engineering from Hanyang Univerity
South Korea. His research interests are Medical Image processing and Machine /deep learning.
Adnan Fazil received his Ph.D. degree in Electrical engineering from Air Univesity. His research interests
are image Communication, Wireless Communication.
Muhammad Zakwan received his Ph.D. degree in Electronics engineering from KoC Univerity Turkey.
His research interests are optics and Photonics.
Zahid Mehmood received his Ph.D. degree in Electronics Engineering from North Dakota University,
USA.His research interests are Object Detection, Pattern Recognition, Image Enhancement, and Digital
Image/Video Processing.

References

Al-Ghaili, A. M., Mashohor, S., Ramli, A. R., & Ismail, A. (2013). Vertical-Edge-Based Car-
License-Plate Detection Method. IEEE Transactions on Vehicular Technology, 62(1),
26–38. https://doi.org/10.1109/TVT.2012.2222454
Anagnostopoulos, C.-N. E. (2014). License Plate Recognition: A Brief Tutorial. IEEE Intelligent
Transportation Systems Magazine, 6(1), 59–67.
https://doi.org/10.1109/MITS.2013.2292652
20

Anagnostopoulos, C.-N. E., Anagnostopoulos, I. E., Psoroulas, I. D., Loumos, V., & Kayafas, E.
(2008). License Plate Recognition From Still Images and Video Sequences: A Survey.
IEEE Transactions on Intelligent Transportation Systems, 9(3), 377–391.
https://doi.org/10.1109/TITS.2008.922938
Asif, M. R., Chun, Q., Hussain, S., & Fareed, M. S. (2016). Multiple licence plate detection for
Chinese vehicles in dense traffic scenarios. IET Intelligent Transport Systems, 10(8),
535–544. https://doi.org/10.1049/iet-its.2016.0008
Bulan, O., Kozitsky, V., Ramesh, P., & Shreve, M. (2017). Segmentation- and Annotation-Free
License Plate Recognition With Deep Localization and Failure Identification. IEEE
Transactions on Intelligent Transportation Systems, 18(9), 2351–2363.
https://doi.org/10.1109/TITS.2016.2639020
Cho, B. K., Ryu, S. H., Shin, D. R., & Jung, J. I. (2011). License plate extraction method for
identification of vehicle violations at a railway level crossing. International Journal of
Automotive Technology, 12(2), 281–289. https://doi.org/10.1007/s12239-011-0033-9
Du, S., Ibrahim, M., Shehata, M., & Badawy, W. (2013). Automatic License Plate Recognition
(ALPR): A State-of-the-Art Review. IEEE Transactions on Circuits and Systems for
Video Technology, 23(2), 311–325. https://doi.org/10.1109/TCSVT.2012.2203741
Gerber, C., & Chung, M. (2016). Number Plate Detection with a Multi-Convolutional Neural
Network Approach with Optical Character Recognition for Mobile Devices. Journal of
Information Processing Systems, 12, 100–108. https://doi.org/10.3745/JIPS.04.0022
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich Feature Hierarchies for Accurate
Object Detection and Semantic Segmentation. 2014 IEEE Conference on Computer
Vision and Pattern Recognition, 580–587. https://doi.org/10.1109/CVPR.2014.81
Gonzalez, R. C., & Woods, R. E. (2006). Digital Image Processing (3rd Edition). Prentice-Hall,
Inc.
Ho, W. T., Lim, H. W., & Tay, Y. H. (2009). Two-Stage License Plate Detection Using Gentle
Adaboost and SIFT-SVM. 2009 First Asian Conference on Intelligent Information and
Database Systems, 109–114. https://doi.org/10.1109/ACIIDS.2009.25
Hsi-Jian Lee, Si-Yuan Chen, & Shen-Zheng Wang. (2004). Extraction and recognition of license
plates of motorcycles and vehicles on highways. Proceedings of the 17th International
Conference on Pattern Recognition, 2004. ICPR 2004., 4, 356-359 Vol.4.
https://doi.org/10.1109/ICPR.2004.1333776
Ibrahim, T., & Kirami, K. (2017). License Plate Recognition System Using Artificial Neural
Networks. ETRI Journal, 39(2), 163–172. https://doi.org/10.4218/etrij.17.0115.0766
K. Simonyan, & d A. Zisserman. (n.d.). Very deep convolutional networks for large-scale image
recognition. ICLR, 2015.
Khan, M. A., Sharif, M., Javed, M. Y., Akram, T., Yasmin, M., & Saba, T. (2018). License
number plate recognition system using entropy-based features selection approach with
SVM. IET Image Processing, 12(2), 200–209. https://doi.org/10.1049/iet-ipr.2017.0368
Kim, K. K., Kim, K. I., Kim, J. B., & Kim, H. J. (2000). Learning-based approach for license
plate recognition. Neural Networks for Signal Processing X. Proceedings of the 2000
IEEE Signal Processing Society Workshop (Cat. No.00TH8501), 2, 614–623 vol.2.
https://doi.org/10.1109/NNSP.2000.890140
Li, H., & Shen, C. (2016). Reading Car License Plates Using Deep Convolutional Neural
21

Networks and LSTMs. ArXiv:1601.05610 [Cs]. http://arxiv.org/abs/1601.05610

Li, H., Wang, P., & Shen, C. (2019). Toward End-to-End Car License Plate Detection and
Recognition With Deep Neural Networks. IEEE Transactions on Intelligent
Transportation Systems, 20(3), 1126–1136. https://doi.org/10.1109/TITS.2018.2847291
Massoud, M. A., Sabee, M., Gergais, M., & Bakhit, R. (2013). Automated new license plate
recognition in Egypt. Alexandria Engineering Journal, 52(3), 319–326.
https://doi.org/10.1016/j.aej.2013.02.005
Md. Mostafa Kamal Sarke, Sook Yoon, & Dong Sun Park. (n.d.). A Fast and Robust License
Plate Detection Algorithm Based on Two-stage Cascade AdaBoost. 8(10), 3490–3507.
https://doi.org/10.3837/tiis.2014.10.012
Nomura, S., Yamanaka, K., Katai, O., Kawakami, H., & Shiose, T. (2005). A novel adaptive
morphological approach for degraded character image segmentation. Pattern
Recognition, 38(11), 1961–1975. https://doi.org/10.1016/j.patcog.2005.01.026
Otsu, N. (1979). A Threshold Selection Method from Gray-Level Histograms. IEEE
Transactions on Systems, Man, and Cybernetics, 9(1), 62–66.
https://doi.org/10.1109/TSMC.1979.4310076
Park, S.-K., & Sim, D.-G. (2011). New MCT-based face recognition under varying lighting
conditions. International Journal of Control, Automation and Systems, 9(3), 542–549.
https://doi.org/10.1007/s12555-011-0314-0
Radzi, S. A., & Khalil-Hani, M. (2011). Character Recognition of License Plate Number Using
Convolutional Neural Network. In H. Badioze Zaman, P. Robinson, M. Petrou, P.
Olivier, T. K. Shih, S. Velastin, & I. Nyström (Eds.), Visual Informatics: Sustaining
Research and Innovations (pp. 45–55). Springer. https://doi.org/10.1007/978-3-642-
25191-7_6
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You Only Look Once: Unified,
Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 779–788. https://doi.org/10.1109/CVPR.2016.91
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
Sang Kyoon Kim, Dae Wook Kim, & Hang Joon Kim. (1996). A recognition of vehicle license
plate using a genetic algorithm based segmentation. Proceedings of 3rd IEEE
International Conference on Image Processing, 2, 661–664 vol.2.
https://doi.org/10.1109/ICIP.1996.560964
Shen-Zheng Wang, & Hsi-Jian Lee. (2003). Detection and recognition of license plate characters
with different appearances. Proceedings of the 2003 IEEE International Conference on
Intelligent Transportation Systems, 2, 979–984 vol.2.
https://doi.org/10.1109/ITSC.2003.1252632
Shivakumara, P., Tang, D., Asadzadehkaljahi, M., Lu, T., Pal, U., & Hossein Anisi, M. (2018).
CNN-RNN based method for license plate recognition. CAAI Transactions on
Intelligence Technology, 3(3), 169–175. https://doi.org/10.1049/trit.2018.1015
Tarigan, J., Nadia, Diedan, R., & Suryana, Y. (2017). Plate Recognition Using Backpropagation
Neural Network and Genetic Algorithm. Procedia Computer Science, 116, 365–372.
https://doi.org/10.1016/j.procs.2017.10.068
22

Wen, Y., Lu, Y., Yan, J., Zhou, Z., von Deneen, K. M., & Shi, P. (2011). An Algorithm for
License Plate Recognition Applied to Intelligent Transportation System. IEEE
Transactions on Intelligent Transportation Systems, 12(3), 830–845.
https://doi.org/10.1109/TITS.2011.2114346
Wu, B.-F., Lin, S.-P., & Chiu, C.-C. (2007). Extracting characters from real vehicle licence
plates out-of-doors. IET Computer Vision, 1(1), 2–10. https://doi.org/10.1049/iet-
cvi:20050132
Xiang, H., Yuan, Y., Zhao, Y., & Fu, Z. (2017). License plate detection based on fully
convolutional networks. Journal of Electronic Imaging, 26(5), 053027.
https://doi.org/10.1117/1.JEI.26.5.053027
Xie, L., Ahmad, T., Jin, L., Liu, Y., & Zhang, S. (2018). A New CNN-Based Method for Multi-
Directional Car License Plate Detection. IEEE Transactions on Intelligent
Transportation Systems, 19(2), 507–517. https://doi.org/10.1109/TITS.2017.2784093
Yang, Y., Li, D., & Duan, Z. (2018). Chinese vehicle license plate recognition using kernel-
based extreme learning machine with deep convolutional features. IET Intelligent
Transport Systems, 12(3), 213–219. https://doi.org/10.1049/iet-its.2017.0136
Yepez, J., & Ko, S.-B. (2018). Improved license plate localisation algorithm based on
morphological operations. IET Intelligent Transport Systems, 12(6), 542–549.
https://doi.org/10.1049/iet-its.2017.0224
Zhang, Y., Wang, Y., Zhou, G., Jin, J., Wang, B., Wang, X., & Cichocki, A. (2018). Multi-
kernel extreme learning machine for EEG classification in brain-computer interfaces.
Expert Systems with Applications, 96, 302–310.
https://doi.org/10.1016/j.eswa.2017.12.015
Figures

Figure 1

Korean License plates

Figure 2
Flowchart of proposed LPD’s method

Figure 3

Test Image with vehicle detection

Figure 4

(a) Plate with white background (b) Plates with a colored background
Figure 5

(a) License plate with white background (b) colored background after morphological Operations

Figure 6

Detection results of the Proposed method

Figure 7
Character Segmentation

Figure 8

MCT Feature Extraction

Figure 9

LUT classi er
Figure 10

Results of images takes (a) sunny day (b) night time (c) dusk (d) cloudy (e) Underground parking (f)
Rainy
Figure 11

Accuracy vs Resolution