SPA 2022 Rozpozn - Poj - Zabytk - Final
SPA 2022 Rozpozn - Poj - Zabytk - Final
Abstract—In this article, a system for automatic recognition of Automatic vehicle recognition is a subset of an object
Polish historic vehicles is proposed. A unique database of vehicle recognition. It can be divided into the classification of models
images was prepared, including real and scale vehicles. A software and the assessment of their condition. There are various
was developed to automatically classify vehicle models and assess approaches to classification of car makes and their types. As
their condition using artificial neural networks. Training, example, the Higher-Order Singular Value Decomposition
validation and test were performed for various datasets and neural (HOSVD) in the multi-dimensional tensor spaces method
network models. In addition, a web application visualizing vehicle allowed for the car manufacturer vehicle logo recognition [8].
recognition is offered. Neural networks may be used for a such recognition task,
specifically a deep learning. A convolutional neural network
Keywords—Polish historic vehicles; vision system; artificial
neural networks; web application; image recognition; image
(CNN) method with pre-training strategy was used for vehicle
processing manufacturer recognition (VMR) based on logo in [9]. Faster
Region Based Convolutional Neural Networks (Faster-RCNN)
I. INTRODUCTION were used for detection a brand [10, 11]. The Inception-v3 was
used as a pre-trained evaluation network o identify the car
Along with the development of the automotive industry,
manufacturer and type [12].
modern vehicles are continuously designed and produced. They
are more and more automated and environmentally friendly. On Vehicle condition assessment is mainly used in the car
the other hand, we are dealing with historic vehicles. As the insurance processing, and, more specifically, in the damage
years go by, “new old” models join this group. appraisal process. For this purpose deep learning with transfer
learning was used [13, 14]. R-CNN method, statistical analysis
In Poland, the country of the authors of this article, according
and metadata model construction may also be included [15].
to the legislation, a historic vehicle is a vehicle entered in the
However, to the best authors' knowledge, there are no
register of monuments, as well as in the inventory of museum
approaches on the recognition of models and assessment of
objects [1]. Legislative work is underway on a narrow definition
visual condition of historic Polish vehicles.
according to which a historic vehicle should meet the following
conditions jointly: it is entered in the inventory of museum This paper proposes a system for automatic recognition of
objects or in the register of monuments, is at least 30 years old, the selected Polish historic vehicles, having regard to
is no longer produced and was not a subject of fundamental classification of models and assessment of their visual condition.
construction changes [2]. You can find sources that set this age Various CNNs were tested during an implementation of the
limit at 25 or 40 years. system. The tests were conducted on the specially prepared
database of images containing photos of real cars and scale
Historic vehicles are collected and protected, just like other
models.
monuments, and may serve to educate current and future
generations. Polish old vehicles are in the collections of both II. POLISH HISTORIC VEHICLES IMAGE DATABASE
national and private museums. Examples of such museums are:
the Museum of Technology and Transport in Szczecin [3], A database of images for training artificial neural networks
Museum of Automotive and Technology in Otrębusy beside models and for evaluation quality of their operation was
Warsaw [4], Topacz Automotive Museum in Ślęza near prepared. Images were acquired in two ways. The first was to
Wroclaw [5] and National Museum of Agriculture and adapt the so-called web crawler, while the second was to self
Agricultural-Food Industry in Szreniawa near Poznan [6]. photograph the vehicles. Bearing in mind the educational
Particularly noteworthy is the last mentioned museum, where, aspects and increasing possibility of obtaining images, it was
among other Polish vehicles, there is a unique collection of 18 decided to supplement the database containing real vehicles with
Tarpan cars in various production varieties [7]. In the case of images of their scale models. The images were divided into
private individuals, old cars may constitute an investment of training, validation, and test sets in the proportions of
capital as well as be owned for hobby reasons. Fairs and rallies approximately 60, 20 and 20 percent, respectively (a typical
of this type of vehicles are organized, where it is possible to meet proportion for testing systems with CNNs).
many enthusiasts. There are also many models of Polish vehicles
in various scales and with different levels of detail.
The web crawler was implemented in the Python TABLE I. DETAILS OF THE DATABASE OF HISTORIC VEHICLES IMAGES
programming language [16], using the ‘icrawler’ library [17]. It real vehicles scale models
was implemented to download images that were in the public vehicle modeltrain valid train valid total
test total test total
domain. Then, images were selected manually. A special care ing ation ing ation
was taken to ensure that the final image database includes only Fiat 125p 113 38 38 189 32 11 11 54 243
images of those models that had the appropriate number of Fiat 126p 155 52 52 259 27 9 9 45 304
Mikrus MR-300 66 22 22 110 18 6 6 30 140
characteristic features for a given vehicle, so that the vehicle
Nysa 94 32 32 158 37 13 13 63 321
model could be determined. Polonez 154 52 52 258 28 10 10 48 306
Taking photographs of selected Polish historic vehicles was Syrena 71 24 24 119 43 15 15 73 192
Tarpan 151 50 50 251 44 15 15 74 325
possible at the National Museum of Agriculture and
Warszawa 58 19 19 96 24 8 8 40 136
Agricultural-Food Industry in Szreniawa. Several dozen real Żuk 110 36 36 182 13 5 5 23 205
cars and also scale models of vehicles, objects owned by the Non-historic
Museum, have been made available for this work. A part of scale 120 40 40 200 62 21 21 104 304
vehicles
models was owned by one of the authors. The Museum's total 1092 365 365 1822 328 113 113 554 2376
automotive collection consists mainly of Tarpans, but apart from
them, one could see many other vehicles. It is worth adding that
it was possible to take photographs a wide cross-section of the Considering the practical application in vision systems and
wear and tear of the items, which means that not only the the expected effectiveness of this type of a system, the number
restored display models were photographed, but also items in of classes can even be reduced by combining the individual
poor condition, in terms of usability in the automatic vehicle classes with each other. Due to insufficient and unbalanced
condition assessment system. number of images belonging to each of the six classes, it was
decided to also prepare a less detailed assessment using a binary,
The real vehicles were, if possible, photographed from eight i.e. two-valued classification. In the case of two classes, there
positions: front, back, left side, right side, front left side, front are cars in a good or in a poor condition, mainly characterized
right side, rear left side, and rear right side. In addition, the by the occurrence of corrosion on the vehicle body. The details
individual elements of chosen vehicles, such as the radiator of the prepared database of images for the vehicle condition
grille, headlights and mirrors, were registered. As it was assessment are presented in Table II.
mentioned above, the authors also took photos of scale models
of vehicles (designed in two scales 1:24 and 1:43). Like real cars TABLE II. DETAILS OF THE PREPARED DATABASE OF IMAGES FOR
they were photographed from the same eight positions with the VEHICLE CONDITION ASSESSMENT
equal lighting and background conditions. All photos were taken non-Polish historic or
Polish historic vehicles
using an iPhone 11 with two 12MP cameras with an ultra-wide visual condition modern vehicles
total
and wide-angle lens. The original image resolution 4032×3024 of vehicle train valid
test
in train valid
test
in
pixels was scaled down to 2048×1536 pixels. ing ation total ing ation total
good 142 47 47 236 88 29 29 146 382
The choice of vehicles was dictated by the popularity of poor 134 44 44 222 96 32 32 160 382
historic cars and the possibility of obtaining photographs total 276 91 91 458 184 61 61 306 764
showing their images in terms of the availability of input data to
the designed system. Nine models of Polish old vehicles in their Sample example photos showing real vehicles and scale
popular variants were selected: Fiat 125p, Polonez, Warszawa models in terms of model recognition are shown in Fig. 1. Figure
as examples of family cars (also in pick-up variants), Fiat 126p 2 shows illustrative vehicles photos in terms of assessment of
‒ a city car, Mikrus MR-300 ‒ a micro car, Syrena ‒ a small car, their visual condition, i.e. paint and sheet metal condition for two
also in pick-up variant, Nysa, Tarpan (also as a Honker off-road classes.
model), Żuk ‒ vans. An additional class of non-historic vehicles In order to make experiments, pre-processing was performed
was also introduced for vehicles potentially not classified as a on the prepared image database: images were size uniformed
historic car. The details of the prepared database of images for and artificially augmentation in order to increase the training
the vehicle classification are presented in Table I. data. Images were scaled down to the size of 256×256 pixels as
According to appearance and condition, i.e. paint and body, the tested CNN architectures (including Inception V3 and
interior, frame / body, mechanical, and tires, the considered VGG16) prefer such square-sized input images.
historic vehicles, are divided into six condition classes To data augmentation we used: rotation range, width shift,
(numbered from zero to five) [18]. In the zero class there are height shift, brightness range change, shear range, zoom range,
vehicles is in the worst condition, i.e. inoperative. The class one and horizontal flip.
shows signs of severe abuse. Class two indicates signs of
excessive wear and tear. Vehicles in the class three are Examples of the used generated images after the size scaling
characterized by the normal wear and tear, while in the class four and data augmentation are presented in Fig. 3.
there are vehicles in a better than average condition. The last
class five constitutes vehicles in the absolutely excellent
condition [18].
a) b) c)
a) b) Figure 3. Example images generated from the image database using the size
uniformity and augmentation performed for the image from Fig. 1g
a) b)
Figure 6. Screenshot of the prepared application
Figure 5. Examples of the vehicle condition assessment results: (a) correct, (b)
incorrect – vehicle in poor condition was classified as vehicle in good condition
VI. CONCLUDING REMARKS
For educational purposes, in order to prepare the web This article presents a vision system for the automatic
application, a class without vehicles (281 images), called ‘Not a recognition of Polish historic vehicles. During the work, a
car’ has been added to the dataset 1 for the vehicle model database of images was prepared, including real and scale
classification. 169 images from this class were used for training, vehicle images. A software for the automatic classification of
56 images for validation, and 56 images for the test. Metrics vehicle models and condition assessment was prepared using
values for the test using VGG16 network are presented in Table CNNs. Training, validation and tests were performed for
VII. VGG network was used because it gives better results for various datasets and CNNs. In addition, the web application
the vehicle model classification in four datasets (see Tab. III and visualizing vehicle recognition was prepared.
IV), including training and validation on the real vehicles and
With the conducted experiments, for the test on real
the scale models together.
vehicles, a slightly higher accuracy was obtained in the case of
training on real vehicles and scale models, compared to the Database Systems, 2014, vol. 8398, pp. 117-126. doi: 10.1007/978-3-319-
results obtained in training with the real vehicles only. For the 05458-2_13.
test on the scale models, greater accuracy was obtained in the [9] Huang Y., Wu R., Sun Y., Wang W., X. Ding X., “Vehicle Logo
Recognition System Based on Convolutional Neural Networks With a
case of training on real vehicles and scale models, compared to Pretraining Strategy”, IEEE Transactions on Intelligent Transportation
those obtained when the training was performed on scale Systems, 2015, vol. 16, no. 4, pp. 1951-1960, doi:
models only. This means that if the target classification is to 10.1109/TITS.2014.2387069.
take place on scale vehicle models, then in the training and [10] Ansari I., Lee Y., Jeong Y., Shim J., “Recognition of Car Manufacturers
validation phase it is worth supporting the system with images using Faster R-CNN and Perspective Transformation”, Journal of Korea
Multimedia Society, vol. 21, no. 8, August 2018, pp. 888-896, doi:
of real vehicles. In the case of the assessment of the condition https://doi.org/10.9717/kmms.2018.21.8.88.
of vehicles, the accuracy can be slightly improved by adding [11] Kunduraci M., Kahramanli H. (2019). “Vehicle Brand Detection Using
non-Polish historic and modern vehicles to the dataset of Polish Deep Learning Algorithms”, International Journal of Applied Mathematics
historic vehicles. This allows to increase the number of images Electronics and Computers, vol. 7, pp. 70-74, doi: 10.18100/ijamec.578497.
for training and validation. [12] Albini L., Gutoski M., Matheus, Lopes H., “Car Make and Model
Classification with Deep Learning Methods”, Proc. of Congresso Brasileiro
In the future, the image database is planned to be expanded de Inteligência Computacional, 2020, pp. 1-7, doi: 10.21528/CBIC2019-
and some new models to be trained so that the effectiveness of 103.
the system could be improved. Additionally, the web application [13] Patil K., Kulkarni M., Sriraman A., Karande S., “Deep Learning Based Car
may offer also a functionality of the vehicle condition Damage Classification”, Proc. of 16th IEEE International Conference on
Machine Learning and Applications (ICMLA), 2017, pp. 50-54, doi:
assessment. 10.1109/ICMLA.2017.0-179.
[14] Dwivedi, M., Malik H. S., Omkar S. N., Monis E. B., Khanna B., Samal S.
ACKNOWLEDGMENT R., Tiwari A., Rathi A., “Deep Learning-Based Car Damage Classification
The authors of this article would like to thank to the consent and Detection”, In: Chiplunkar, N., Fukao, T. (eds) Advances in Artificial
Intelligence and Data Engineering. Advances in Intelligent Systems and
of the Director of the National Museum of Agriculture and Computing, 2021, vol. 1133. Springer, Singapore.
Agricultural-Food Industry in Szreniawa, Dr. Jan Maćkowiak, https://doi.org/10.1007/978-981-15-3514-7_18.
and courtesy of the museum's curator, Chief Specialist for [15] Poon F., Zhang Y., Roach J., Josephs D., Santerre J., “Modeling and
Motorization of the Countryside, Mr. Gerard Radecki and the Application of Neural Networks for Automotive Damage Appraisals”,
other Museum employees involved in this project. Southern Methodist University (SMU) Data Science Review, 2021.
[16] Python Software Foundation, “Python”, https://www.python.org (accesss:
Images in Figures 1b, e, f, g, i, 2-6 showing historic vehicles 17.06.2022).
were made with permission of the Director of the National [17] Chen K., “Built-in crawlers”, Read the Docs, 2017,
Museum of Agriculture and Agricultural-Food Industry in https://icrawler.readthedocs.io/en/latest/builtin.html (accesss: 17.06.2022).
Szreniawa. The objects are the property of the Museum. The [18] National Auto Auction Association, “NAAA Vehicle Condition Grading
objects in the images in Figures 1a, c, d, h, j come from the Scale”, https://www.naaa.com/standards/vehicle_gradingscale.pdf (accesss:
17.06.2022).
collection of Julian Balcerek and are his property.
[19] Szegedy C., Vanhoucke V., Ioffe S., Shlens J., Wojna Z., “Rethinking the
The research was financed by a research subsidy 0211/SBAD/2022. Inception Architecture for Computer Vision”, 2015, https://arxiv.org
/abs/1512.00567.
REFERENCES [20] Simonyan K., Zisserman A., “Very Deep Convolutional Networks for
Large-Scale Image Recognition”, 2015, https://arxiv.org/abs/1409.1556.
[1] “Prawo o ruchu drogowym” (Polish “Road traffic law”), Dz.U. 1997 Nr 98
[21] Keras, “Adam”, https://keras.io/api/optimizers/adam/ (accesss: 17.06.2022).
poz. 602 z późn. zm., Ustawa z dnia 20 czerwca 1997 r.
[22] Keras, “EarlyStopping”, https://keras.io/api/callbacks/early_stopping/
[2] Web service Prawo drogowe (road law), “Nowa definicja pojazdu
(accesss: 17.06.2022).
historycznego” (“A New definition of a historical vehicle”), 31.05.2021,
https://www.prawodrogowe.pl/informacje/kronika-legislacyjna/nowa- [23] TensorFlow, “tf.keras.layers.GlobalAveragePooling2D”,
definicja-pojazdu-historycznego (access: 17.06.2022). https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAverag
ePooling2D (accesss: 17.06.2022).
[3] Official web page of The Museum of Technology and Transport in Szczecin,
http://en.muzeumtechniki.eu (access: 17.06.2022). [24] Project Jupyter, “Jupyter Notebook”, https://jupyter.org/ (accesss:
17.06.2022).
[4] Official web page of Museum of Automotive and Technology,
http://www.muzeum-motoryzacji.com.pl/gb/ (access: 17.06.2022). [25] “TensorFlow”, https://www.tensorflow.org/ (accesss: 17.06.2022).
[5] Official web page of Topacz Automotive Museum, [26] keras.io, “Keras”, https://keras.io/ (accesss: 17.06.2022).
http://www.muzeum.topacz.pl/ (access: 17.06.2022). [27] opencv.org, “OpenCV”, https://opencv.org/ (accesss: 17.06.2022).
[6] Official web page of National Museum of Agriculture and Agricultural- [28] numpy.org, “NumPy”, https://numpy.org/ (accesss: 17.06.2022).
Food Industry in Szreniawa, https://muzeum-szreniawa.pl (access: [29] JetBrains s.r.o., “PyCharm: The Python IDE for Professional Developers”,
17.06.2022). https://www.jetbrains.com/pycharm/ (accesss: 17.06.2022).
[7] Radecki G., “Museum of the Tarpan car”, https://muzeum- [30] Pallets, “Welcome to Flask”, Flask Documentation,
szreniawa.pl/muzeum_tarpana/ (access: 17.06.2022). https://flask.palletsprojects.com/en/2.1.x/ (accesss: 17.06.2022).
[8] Cyganek B., Wozniak M., “Vehicle Logo Recognition with an Ensemble of
Classifiers”, Proc. of Asian Conference on Intelligent Information and