An Analytical Approach for Enhancing the

Automatic Detection and Recognition of Skewed

Bangla License Plates
2019 International Conference on Bangla Speech and Language Processing (ICBSLP) 978-1-7281-5241-7/20/$31.00 ©2020 IEEE 10.1109/ICBSLP47725.2019.201528

Koushik Roy∗† , Abu Mohammad Shabbir Khan∗† ,

Mohammad Zariff Ahsham Ali† , Sazid Rahman Simanto† , Nabeel Mohammed† ,
Muhammad Asif Atick‡ , Shahidul Islam‡ , Kazi Mejbaul Islam‡
† Department
of Electrical and Computer Engineering, North South University
Email: {koushik.roy, abu.khan1, zariff.ahsham, sazid.rahman, nabeel.mohammed}@northsouth.edu
‡ HeadBlocks

Email: {asif, shahidul, kazimejbaul}@head-blocks.com

Abstract—Although there has been a huge body of work on character detection, and character recognition. Although the
Bangla license plate detection and recognition, the successes ALPR system developed by HeadBlocks (referred to as HB-
of these works have largely been limited to correct detection ALPR henceforth) has over 96% accuracy as determined by
and recognition of undistorted license plates whose images are
taken chiefly from the front or the back of vehicles with slight experiments on their confidential test set, HB-ALPR finds it
angular variations. As a result, most Bangla automatic license difficult to correctly identify license plates that are more than
plate recognition (ALPR) systems in practice struggle when the 30° skewed on the image plane (ip-skewed) or the viewing
license plates are skewed on the viewing or the image planes plane (vp-skewed). Examples of such ip-skewed and vp-skewed
of the license plates. In this paper, we address this issue by license plates are shown in Fig. 1.
proposing an analytical approach that can enhance the ALPR of
both normal and skewed license plates and can be incorporated Since HB-ALPR has been commercially deployed and can-
into existing Bangla ALPR systems without modifying their not be modified without affecting all four stages of its pipeline
internal structures. Specifically, we demonstrate how existing and extensive testing, we approached the problem of correctly
ALPR systems can be treated as black boxes and analyzed to identifying skewed license plates by treating HB-ALPR as a
understand what sort of license plate images they work best on black box. Concretely, we first analyzed what kind of license
and introduce a novel pipeline that combines deep learning and
an algorithmic procedure for transforming images of both normal plate images are ideal for HB-ALPR, and then by combining
and skewed license plates into formats that are best suited for the a deep learning method with an algorithmic procedure, we
ALPR systems. We note that our proposed method can be easily developed a pre-processing step for the license plate images
generalized and applied to non-Bangla license plates as well. that led to a performance improvement of HB-ALPR for both
Index Terms—Bangla, Automatic License Plate Recognition, skewed and challenging non-skewed images.
Deep Learning
The rest of the work is organized as follows. In Section
II, we describe some related work done on Bangla and non-
I. I NTRODUCTION Bangla ALPR. In Section III, we describe our data set, the
Correct recognition of vehicle license plates has numerous analysis we did to understand the strengths and weaknesses
use cases that include penalizing irresponsible driving and of HB-ALPR, and the method we developed to pre-process
parking, keeping track of vehicles coming in and going out the images to the best format for HB-ALPR. In Section IV,
of parking lots, identifying vehicle ownership and so on. we show how our method affected the performance of HB-
While a lot of work on automatic detection and recognition of ALPR, and then in Section V, we present our conclusions and
license plates has been done for decades, there is still a lot of avenues of future work.
progress to be made on correct Bangla automatic license plate
II. R ELATED W ORK
recognition (ALPR). The variance that exists in Bangla license
plates owing to existence of different metro and vehicle types, While there have been a number of works on Bangla
and the scarcity of adequate samples to catch all of that variety ALPR based on algorithmic approaches such as [1] (which
makes Bangla ALPR specially challenging. A Bangladeshi focused on license plate detection only) and [2] (which used
company named HeadBlocks is working with the Dhaka template matching), shallow machine learning methods such
Metropolitan Police (DMP) to address this and has developed as [3] (which used support vector machines), and deep learning
a commercial Bangla ALPR system with a four stage pipeline such as [4] (which used convolutional neural networks), none
that includes license plate detection, character segmentation, of these works present extensive evaluations of their perfor-
mances on skewed license plate images. Moreover, to the best
* denotes equal contribution. of our knowledge, no work has been done so far to augment

978-1-7281-5241-7/19/$31.00 ©2019 IEEE

Authorized licensed use limited to: University of Glasgow. Downloaded on June 03,2020 at 22:23:22 UTC from IEEE Xplore. Restrictions apply.
 (a) IP-Skewed Ex. I (b) IP-Skewed Ex. II (c) VP-Skewed Ex. I (d) VP-Skewed Ex. II
Fig. 1. Examples of Different Types of Skewed License Plate Images

the performances of existing Bangla ALPR systems through license plates on images and then transforming the segmented
pre-processing their input images while treating the systems license plates into uniform rectangular views. The details of
as black boxes. each of the steps are provided in the following sub-subsections.
As in the case of Bangla ALPR systems, most ALPR 1) Instance Segmentation of License Plates: We conducted
systems developed for non-Bangla license plates such as [5], transfer learning on an existing Mask R-CNN [11] model and
[6], [7], and [8] do not demonstrate good performances or fine tuned it for Bangla license plates to determine exactly
show thorough assessments on skewed license plates. which pixels a license plate consisted of and to regress the
Silva and Jung [9], however, focused on building a complete smallest rectangular bounding box that contained the entirety
ALPR system robust to skewness of license plates which of the license plate.
comprised of three main steps: vehicle detection, license plate To train our Mask R-CNN, we divided the 1015 mask
detection and unwarping, and optical character recognition annotated license plate images into a training set of 800 images
(OCR). In this work, they introduced a novel network named and a validation set of 215 images. By selecting a learning rate
Warped Planar Object Detection Network (WPOD-NET) that of 0.001 and loading 2 images to an NVIDIA K80 GPU at
searches for license plates in the region of each detected a time, we trained our model for 100 epochs with the batch
vehicle and computes parameters for an affine transformation size set equal to 100. For the rest of the hyperparameters, we
that enables the rectification of each detected license plate to used the same values as mentioned in [11]. The outputs of our
a rectangular frontal view. Our proposed solution is similar Mask R-CNN model are shown in Fig. 3.
to WPOD-NET in that it too produces a rectangular frontal 2) Perspective Transformation of License Plates: Using the
view of the detected license plate, but it does so in a different predicted masks from our Mask R-CNN model, we generated
way. In addition, our proposed approach is applicable to any the Shi-Tomasi Corners [12] reconstructed from Harris Cor-
ALPR system and is not confined to any specific pipeline as ners [13] of each license plate instance. Through our empirical
WPOD-NET is. studies, we found that padding the mask by 10 pixels led to
III. M ETHODOLOGY
A. The Dataset
Our dataset consisted of around 3000 license plate images of
varying sizes taken from different perspectives using different
cameras. For bringing uniformity, we rescaled all images to
400 × 400 because the character segmentation and recognition
networks of HB-ALPR were trained using images of that
size. While resizing images, we opted to keep their original
aspect ratios the same and added black padding where it was
necessary. An example set of such images after converting (a) 400x400 (b) 600x600
them to the size 400 × 400 is shown in Fig. 2.
We then used three annotators to label the images as vp-
skewed, ip-skewed, or normal (all being mutually exclusive)
and used the majority vote of the annotators to decide the final
label of each image. Next, we randomly selected 1015 images
and created mask annotations of the license plates in these
images using a tool called VIA [10].
B. The Proposed Solution
Our proposed method of pre-processing license plate images (c) 332x400 (d) 3840x2160
consists of two steps: performing instance segmentation of Fig. 2. Images of Different Sizes after Conversion to the Size 400x400

Authorized licensed use limited to: University of Glasgow. Downloaded on June 03,2020 at 22:23:22 UTC from IEEE Xplore. Restrictions apply.
 (a) IP-Skewed Ex. I (b) IP-Skewed Ex. II (c) VP-Skewed Ex. I (d) VP-Skewed Ex. II
Fig. 3. Instance Segmentation of the License Plates in Fig. 1 Using Our Mask R-CNN Model (Best Viewed in Color)

slight improvement in corner detection. C. Analysis of HB-ALPR

Next, for each detected license plate, we put its Shi-Tomasi
Corners in an array of tuples P oints where in each tuple, the To determine what is the ideal size of license plate images
first element (P oints[i]x ) represented the x co-ordinate and for HB-ALPR, we took 15 samples each from the sets of
the second element (P oints[i]y ) represented the y co-ordinate originally 400 × 400, 600 × 600, 1280 × 720, and 3840 × 2160
of a Shi-Tomasi Corner i. sized images and 29 samples from images of different sizes.
The samples were a mixture of apparently good looking and
S = P oints[i]x + P oints[i]y ; ∀i (1) somewhat skewed and distorted images. Note that despite
having different original sizes, all the samples were resized
D = P oints[i]x − P oints[i]y ; ∀i (2) to 400 × 400 pixels.
We then found the boxes of the license plates in these
We then constructed the vectors S and D that respectively
images using our Mask R-CNN model and produced two
store the summations of and differences between the x and y
different types of outputs: one where each pixel outside the
co-ordinates of the Shi-Tomasi Corners.
license plate bounding box was made black (black bbox) and
one where each such pixel was made white (white bbox). At
topLef tx , topLef ty = P oints[argmin(S)] (3)
this point, we had three representations of each image: the
topRightx , topRighty = P oints[argmax(D)] (4) original one (org original), the black bbox image, and the white
bbox image where each representation had the dimensions
bottomLef tx , bottomLef ty = P oints[argmin(D)] (5) 400 × 400. As HB-ALPR uses a fully convolutional network
bottomRightx , bottomRighty = P oints[argmax(S)] (6) in the first stage of its pipeline, we further resized each of
these representations to the size 200 × 200 to see how HB-
After that, we used (3), (4), (5), and (6) to determine the four ALPR performs for small images. Next, we assessed how the
extreme corner points—topLef t, topRight, bottomLef t, and accuracy (determined by for what fraction of license plates all
bottomRight—of each detected license plate and used these the digits on each plate were identified correctly) of HB-ALPR
points to project the detected license plate to a rectangular varied for these six representations using BGR and RGB color
frontal view. Subsections III-C and III-D describe how we schemes as shown in Fig. 4 where each bar presents metrics
decided the dimensions of the rectangle. For now, assume that for each set of 89 samples.
each extreme corner j was mapped to a new rectangle corner
0
j . We computed the 3 × 3 map matrix M for the perspective
transformation using (7).
 0 0   
jx jy 1 = jx jy 1 ×M (7)
Finally, we warped each detected license plate in to the im-
age out on a normal viewing plane using (8) where out(x, y)
denotes the pixel value placed at the (x, y) position of out.
 
out(x, y) = in f1 (x, y), f2 (x, y) (8)

where
M11 x + M12 y + M13
f1 (x, y) = (9)
M31 x + M32 y + M33
M21 x + M22 y + M23
f2 (x, y) = (10) Fig. 4. Accuracy Distribution for Mixture of Good and Challenging Images
M31 x + M32 y + M33

Authorized licensed use limited to: University of Glasgow. Downloaded on June 03,2020 at 22:23:22 UTC from IEEE Xplore. Restrictions apply.
 (a) IP-Skewed Ex. I (b) IP-Skewed Ex. II (c) VP-Skewed Ex. I (d) VP-Skewed Ex. II
Fig. 5. The Deskewed Versions of the License Plates in Fig. 1

D. The Final Output TABLE I

I MPACT OF D ESKEWING ON HB-ALPR
From our analysis, we noticed that the license plates that
occupied a large portion of the images and were causing HB- Accuracy Detection %
ALPR to fail in the 400 × 400 case, were being correctly Type of Image Distribution Distribution
Original Deskewed Original Deskewed
recognized after their images were converted to the size challenging normal 44% 64% 84% 100%
200 × 200. Moreover, the license plates that seemed to have ip skewed 28% 52% 92% 100%
ideal dimensions in the 400 × 400 case were being incorrectly vp skewed 48% 60% 88% 96%
identified by HB-ALPR in the 200 × 200 case because they
became too small. This meant that the performance of HB-
ALPR was being ultimately affected by the size of the license R EFERENCES
plates in the images instead of the size of the images. As [1] S. Azam and M. M. Islam, “Automatic license plate detection in
hazardous condition,” Journal of Visual Communication and Image
we had bounding box co-ordinates of the license plates owing Representation, vol. 36, pp. 172–186, 2016.
to our Mask R-CNN model, we used those co-ordinates to [2] A. C. Roy, M. K. Hossen, and D. Nag, “License plate detection and
calculate the size of the license plates in our samples and character recognition system for commercial vehicles based on mor-
phological approach and template matching,” in 2016 3rd International
found that the best performance was being obtained when the Conference on Electrical Engineering and Information Communication
license plates had dimensions in the ballpark of 150×75 pixels. Technology (ICEEICT). IEEE, 2016, pp. 1–6.
Also, our analysis showed that the black padding had a positive [3] M. A. Uddin, J. B. Joolee, and S. A. Chowdhury, “Bangladeshi vehicle
digital license plate recognition for metropolitan cities using support
effect on license plate recognition despite HB-ALPR not being vector machine,” in Proc. International Conference on Advanced Infor-
trained with such images. Lastly, we found that the RGB color mation and Communication Technology, 2016.
scheme worked best in most of the cases. [4] M. Z. Abedin, A. C. Nath, P. Dhar, K. Deb, and M. S. Hossain, “License
plate recognition system based on contour properties and deep learning
So we decided to deskew the detected license plates to the model,” in 2017 IEEE Region 10 Humanitarian Technology Conference
size 150×75 pixels and then to add black padding to make the (R10-HTC). IEEE, 2017, pp. 590–593.
entire image of size 400 × 400 pixels. The deskewed versions [5] R. Laroca, E. Severo, L. A. Zanlorensi, L. S. Oliveira, G. R. Gonçalves,
W. R. Schwartz, and D. Menotti, “A robust real-time automatic license
of the images in Fig. 1 are shown in Fig. 5. plate recognition based on the yolo detector,” in 2018 International Joint
Conference on Neural Networks (IJCNN). IEEE, 2018, pp. 1–10.
IV. R ESULTS [6] S. M. Silva and C. R. Jung, “Real-time brazilian license plate detection
To understand the impact of our work, we constructed a and recognition using deep convolutional neural networks,” in 2017 30th
SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).
test dataset with 25 challenging normal images, 25 ip-skewed IEEE, 2017, pp. 55–62.
images, and 25 vp-skewed images for a total of 75 images. [7] Z. Selmi, M. B. Halima, and A. M. Alimi, “Deep learning system
As shown in Table I, the recognition accuracy for chal- for automatic license plate detection and recognition,” in 2017 14th
lenging normal, ip-skewed, and vp-skewed images went up IAPR International Conference on Document Analysis and Recognition
(ICDAR), vol. 1. IEEE, 2017, pp. 1132–1138.
by 45%, 86%, and 25% respectively after our pre-processing [8] H. Li, P. Wang, and C. Shen, “Towards end-to-end car license plates
work. In addition, the detection % of license plates (defined detection and recognition with deep neural networks,” ArXiv, vol.
by the percentage of license plates detected successfully) of abs/1709.08828, 2017.
[9] S. Montazzolli Silva and C. Rosito Jung, “License plate detection and
HL-ALPR became almost perfect. recognition in unconstrained scenarios,” in Proceedings of the European
Conference on Computer Vision (ECCV), 2018, pp. 580–596.
V. C ONCLUSION AND F UTURE W ORK [10] A. Dutta, A. Gupta, and A. Zissermann, “VGG image annotator (VIA),”
By pre-processing the input images of Bangla ALPR sys- http://www.robots.ox.ac.uk/ vgg/software/via/, 2016, version: 2.0.6, Ac-
cessed: Mar. 20, 2019.
tems, we have demonstrated how the performance of such [11] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in
systems can be improved for both normal and skewed images Proceedings of the IEEE international conference on computer vision,
despite treating the systems as black boxes. We are currently 2017, pp. 2961–2969.
[12] J. Shi and C. Tomasi, “Good features to track,” Cornell University, Tech.
working on other license plate distortions such as blurriness, Rep., 1993.
fading, low illumination, occlusion et cetera and would be [13] C. G. Harris, M. Stephens et al., “A combined corner and edge detector.”
soon publishing the results of our work for those cases. Citeseer, 1988.

Authorized licensed use limited to: University of Glasgow. Downloaded on June 03,2020 at 22:23:22 UTC from IEEE Xplore. Restrictions apply.