IoT Based on-the-fly Visual Defect Detection in

Railway Tracks
Noora AlNaimi Uvais Qidwai
Department of Computer Science and Engineering Department of Computer Science and Engineering
College of Engineering College of Engineering
Qatar University Qatar University
Doha, Qatar Doha, Qatar
[email protected] [email protected]

Abstract—Railway transportation requires constant inspec- However, solutions vary in terms of being software-based
tions and immediate maintenance to ensure public safety. Tra- solutions that apply machine vision technologies on recorded
ditional manual inspections are not only time consuming, and videos; or robotic solutions which are automated systems that
expensive, but the accuracy of defect detection is also subjected
to human expertise and efficiency at the time of inspection. Com- are deployed on rail tracks and detect cracks using either
puting and Robotics offer automated IoT based solutions where ultrasonic or IR sensors. Software-based solutions consume
robots could be deployed on rail-tracks and hard to reach areas, time to extract and analyze the images from recorded videos,
and controlled from control rooms to provide faster inspection. and robotic-based solutions are limited in their ability in only
In this paper, a novel automated system based on robotics and detecting cracks using sensors without generating results or
visual inspection is proposed. The system provides local image
processing while inspecting, cloud storage of information that images.
consist of images of the defected railway tracks only, and robot To address these limitations, a novel automated system is
localization within a range of 3-6 inches. The proposed system proposed in this work that consists of a robot which performs
utilizes state of the art Machine Learning system and applies inspection using non-destructive inspection method based on
it on the images obtained from the tracks in order to classify visual inspection with local image processing, cloud storage
them as normal or suspicious. Such locations are then marked
and more careful inspection can be performed by a dedicated of information that will consist of images of defected railway
operator with very few locations to inspect (as opposed to the tracks only, and robot localization within a range of 3 inches.
full track). Local image processing while inspection is a novel inspection
Index Terms—Railtrack inspection, visual inspection, com- technique that will allow for faster inspection in parallel with
puter vision, convolutional neural network. cloud storage of information, which will only receive images
of defected rail tracks.
I. I NTRODUCTION
A. Railway Track Defects
In transport systems, safety and reliability are the main Railway track defects are divided into two main parts:
factors that are always questioned, especially in railway trans- Internal Defects and Surface Defects [2]. These defects may
portation systems. Early inspection systems are crucial to exist in the head, weld or base section of the track. The most
maintain safe rail tracks that will ensure safe journeys. Statis- common defects appearing in rail tracks are known as RCF
tics show that 60% of railway accidents are due to derailment, (Rolling Contact Fatigue) which result from the friction in
and 90% are due to railway cracks [1]. Railway track cracks high-speed railways. Another common set of defects is the one
could be inspected by human personnel; however, this is resulting from the local climate condition and infrastructure
not only time consuming, but also the accuracy is subjective peculiarities. High temperature and humid climate, as in Qatar,
since not all cracks are identifiable by naked eyes. As Qatar causes buckling and heat kinks – also know as sun kinks - in
Rail has launched the first trains in Qatar recently, it is very rail tracks. Defects like broken railway tracks or sun-kinks are
important to look for maintenance systems that suits Qatar’s more crucial than a loose ballast or growth of vegetation.
climate for railway track inspections. This demand requires
inspection systems that will continuously inspect the status of B. Railway Track Inspection Methods
all tracks over Qatar and issue immediate maintenance alerts Railway track inspection methods are either contact-based
to avoid accidents. Over the years, machine-driven inspection which are known as NDT (Non Destructive Testing), or non-
systems proved to offer a solution for faster inspection and contact based methods which is based on analyzing images
maintenance. Such inspection systems are common in their or videos of the rail-track. Some examples of each type is:
ability in finding cracks in the rail tracks, as well as the
crack’s location, which helps the maintenance team to reach
and rectify the crack in lesser time.

978-1-7281-4821-2/20/$31.00 ©2020 IEEE 627

Authorized licensed use limited to: INSTITUTE OF ENGINEERING & MANAGEMENT TRUST. Downloaded on February 27,2025 at 16:20:25 UTC from IEEE Xplore. Restrictions apply.
 • Contact-based (NDT): morphological operation. The camera used in this system can
– Ultrasonic Inspection: this method can detect deep capture 100 frames per second; however, the authors have used
internal defects, but fails to detect surface and near only 10 images per second to achieve fast processing.
surface defects [2], [3], [4], [5]. In [5], the authors suggest an automatic inspection method
– MFL (Magnetic Flux Leakage): this technique can of images captured digital scan line cameras installed on
detect near surface defects such as RFC, but fails to both sides of the track. Binary Image Based Rail (BIBRE)
detect deep internal defects [2]. technique is used to extract the rail section of the tracks
– Eddy Current Inspection: this technique is based on which are enhanced with improvement techniques, and then
magnetic fields; therefore, similar to MFL, this tech- recognize the faults in the rail using Gabor channels. The
nique can detect surface defects, but fails to detect image are preprocessed using the Otsu algorithm, and rail
deep internal defects. To overcome this shortage, sections are detected using Canny edge detection and Hough
hybrid solutions combining both ultrasonic and eddy change calculations.
current inspections are available [5]. In [9], an intelligent image processing algorithm that de-
– Acoustic Emission Inspection: this method is com- tects RCF (Rolling Contact Fatigue) is proposed. Region
mon with steel rail tracks, where it is used to detect defects are detected in the images using adaptive histogram
crack’s growth and accumulation as well as source equalization, and segmented after detection by an adaptive
of crack localization [2]. threshold method. The authors created a 3D crack growth
• Non-contact-based: model using the COMSOL simulation software, designed by
identified geometrical properties. The algorithm consists of
– Visual Inspection: this technique is the most efficient multiple stages: Preprocessing stage removing noise from the
technique used for surface defects detection. It is image using the median filtering technique. Defect-processing
based on high-speed cameras that capture images stage in which segmentation threshold is done to identify
of the railway tracks to be processed later based on defects in the preprocessed images using the automatic iter-
pattern recognition of the captured images; therefore, ative selection method. And the defect post-processing stage
it is economical and time saving, but requires higher where the images resulting from the previous step are further
computational time [2]. processed to remove any remaining noise and false defects
by morphological operations. The authors have also designed
C. Literature Review
a crack growth detecting algorithm based on LEFM (linear
Visual inspection based systems - which are also known as elastic fracture mechanics). Although this system have a
computer vision - are one of the most effective and important comparatively high accuracy; however, it is a complex and
inspection tools for flexible automated rail monitoring [6]. expensive system.
Computer vision based techniques makes rail-track inspection In [11] [12], the authors have designed a rail defect detecting
possible from grayscale images only without additional sen- system with real time image processing in which the rail as
sors [7]. well as the rail defects are detected. Rails are detected using
Computer vision based applications consist of two main Hough transform, Canny edge detection and morphological
steps: Image acquisition that consists of capturing images methods for feature extractions, and rail defects are detected
using a camera, and Image Processing which consists of using Robert’s edge detection, Laplacian low pass filter and
improving image quality by reducing the noise generated due morphological methods for feature extractions. In this method,
to illumination, climate factor, or shaking camera [8], [9]. the system targets faults in head-checks, scours, fractures and
Computer vision based solution would not interfere with the undulation faults. This method achieved an average accuracy
train schedules; therefore, would not cause any interruption or rate of 85.3%.
delay. Once the image is captured, faults in track are localized All previous methods were based on image processing;
by using detection methods such as Canny Edge detection however, signal based image processing method is presented
method which locates discontinuities in the image. In feature in [4]. The authors proposed a hybrid feature extraction and
extraction, single features do not provide enough information; selection method based on laser ultrasonic detected signals.
therefore, multiple feature based systems are richer in useful The system combines WPT (Wavelet Packet Transform) which
information. preprocesses data by decomposing different frequencies of the
In [10], the authors proposed a computer vision based input signal into different frequency bands that are the feature
method by capturing images of the two neighboring rail tracks sets, KPCA (Kernel Principal Component Analysis) which is
to determine the rail in the image and the distance between used to reduce redundancy among the feature sets, and SVM
the rails. Image are captured by a camera placed on the (Support Vector Machine) which is used to classify the defects
train that captures the current main rail track of the moving in the input signals. This method has achieved an accuracy of
train, and the neighboring rail track. This method detects 98.73%.
pitch fault, contraction and expansion faults by comparing the All mentioned previous works consisted of visual inspection
image’s pixel to a predefined threshold value by using canny method that are performed after capturing a set of images or
edge detection, feature extraction methods and the closing recording videos. In this study, a novel system is proposed

628
Authorized licensed use limited to: INSTITUTE OF ENGINEERING & MANAGEMENT TRUST. Downloaded on February 27,2025 at 16:20:25 UTC from IEEE Xplore. Restrictions apply.
 Fig. 1. The Block diagram of the Proposed System

that detects faults in rail-tracks on the fly while the robot is

performing inspection.
II. P ROPOSED S YSTEM A RCHITECTURE
In this section the details of the proposed solution are
described.
A. Block Diagram
In this work, a novel non-destructive inspection method
based on visual inspection is proposed. The block diagram of
the proposed system is shown in Fig. 1. The system consists Fig. 2. The flowchart of the proposed system
of three main parts:
• The robot deployed on the rail-track that will capture
images from both sides of the track.
• Local image processing module embedded in the robot.
• Cloud communication and storage of the defected rail-
track sections.
This paper focuses on the local image processing module that
is detailed in the following section.
III. I MAGE P ROCESSING FOR D EFECT D ETECTION
In this section the details of the local image processing
module are described.
A. Technique
Most of the visual inspection solutions are based on image
processing that is done after capturing the whole images of the
rail-track, or after recording a video that is sent to the cloud or
stored on local devices. That is, image processing is done at a
control station away from the rail-track. In this work however,
Fig. 3. The network’s architecture
image processing is done on the rail-track while inspection,
and only defected rail-track images are stored and sent to the
cloud. This makes inspection faster and does not congest the image to the cloud to be post-processed later. Post-processing
cloud with unnecessary data. Fig. 2 shows the flowchart of the consists of locating the defect in the image; however, this part
system. is done at a later stage.
B. Algorithms C. Machine learning & Network Architecture
Local image processing is based on classifying captured The designed 2DCNN consists of 24 layers as shown in Fig.
images to normal or abnormal using a 2DCNN (Convolutional 3. The network has 5 convolution layers of size 5 x 5 in which
Neural Network) [13] [14]. Once an image is classified as learnable are increased at every layer, batch normalization
abnormal, its location is mapped and sent along with the layers to standardize the inputs, rectified linear unit (ReLU)

629
Authorized licensed use limited to: INSTITUTE OF ENGINEERING & MANAGEMENT TRUST. Downloaded on February 27,2025 at 16:20:25 UTC from IEEE Xplore. Restrictions apply.
 TABLE I TABLE III
C ONVOLUTION AND MAXPOOL LAYERS ’ DETAILS E VALUATION RESULTS

Layer Activation Shape Activation Size Evaluation Metric Percentage

Image Input 160 x 120 x 1 19,200 Accuracy 97.0%
conv 1 1 160 x 120 x 8 153,600 Sensitivity 88.9%
maxpool 1 1 80 x 60 x 8 38,400 Specificity 98.8%
conv 2 1 80 x 60 x 25 120,000 Precision 94.1%
maxpool 1 2 40 x 30 x 25 30,000 Prevalence 81.8%
conv 1 2 40 x 30 x 50 60,000
maxpool 1 2 20 x 15 x 50 15,000
conv 2 2 20 x 15 x 75 22,500
maxpool 2 2 10 x 8 x 75 6,000 The neural network is trained with a learning rate of 0.02
conv 3 10 x 8 x 100 8000
in 128 epochs. Training has consumed 12 minutes 36 seconds
maxpool 3 10 x 8 x 100 8000
on a single CPU.
To evaluate the performance of the system, evaluation
TABLE II metrics such as Sensitivity, Specificity, Accuracy, Precision
N UMBER OF CLASSES AND IMAGES IN T YPE -I DATASET and Prevalence are used. These evaluation metrics depend on
Dataset Class No. of Images
the confusion matrix parameters which are True Positive (TP),
2*Type-I Normal 46 True Negative (TN), False Positive (FP), and False Negative
Abnormal 203 (FN). The proposed system has achieved an accuracy rate of
97%, FP rate of 1%, FN rate of 2%, TP rate of 16.2%, and TN
rate of 80.8%. TABLE III shows the details of the evaluation
layers as the nonlinear activation function, and max pooling results. Fig. 6 shows the result for some test data.
units of size 5 x 5 and a stride of 2. V. C ONCLUSION
The input image of size 160 x 120 has a total of 19200 Rail-tracks early inspection and immediate maintenance are
pixels. The first convolution layer conv 1 1 has 8 filters. The the main major factors to ensure track’s safety. Rail-track
number of filters is increased in each convolution layer. The defects are internal or surface defects, and most commonly
second convolution layer conv 2 1 has 25 filters, the third results from high-speed friction and climate condition. Avail-
convolution layer conv 1 2 has 50 filters, the fourth convo- able inspection solutions are either contact-based that can
lution layer conv 2 3 has 75 filters, and the fifth convolution be deployed on tracks and controlled from control rooms,
layer conv 3 has 100 filters. Filters in all convolution layers or non contact based solutions which is known as machine
are of size 5 x 5. TABLE I shows the activation details of the vision based solutions that can identify and locate cracks by
convolution and pooling layers of the network. analyzing images of the rail-track.
This paper has introduced a novel automated system for
IV. T ESTS R ESULTS rail-track inspection that integrates robotics with visual in-
In this section the detail of experimental setup and perfor- spection to detect and locate surface defects. The novelty of
mance is presented. thiswork comes from providing an IoT based on-the-fly local

A. Experimental Setup
The proposed method is implemented on MATLAB R2019
and tested on Type-I dataset [3] [8] which consist of normal
and defected rail-tracks grayscale images. Type-I dataset has
67 images captured from express rails. Most of the images in
the dataset consist of different types of challenging defects;
however, the availability of normal railway track images is
very scarce. This small number of available images makes Fig. 4. Type-I Normal samples
training and evaluating the neural network very challenging.
To solve this problem, the dataset was inflamed by cropping
the original images of size 1000 x 160 to multiple images of
size 160 x 120, and each image is rotated in all directions as
well as mirrored. Also, brightness is adjusted in all images.
This step resulted in a larger number of images to better train
the neural network. Images in the dataset are then classified
into two classes: normal and abnormal. TABLE II shows
the number of images in each class. Fig. 4 and Fig. 5 show
some images from Type-I dataset. Fig. 5. Type-I Abnormal samples

630
Authorized licensed use limited to: INSTITUTE OF ENGINEERING & MANAGEMENT TRUST. Downloaded on February 27,2025 at 16:20:25 UTC from IEEE Xplore. Restrictions apply.
 [11] C. Taştimur, M. Karaköse, E. Akın, and İ. Aydın, “Rail defect detection
with real time image processing technique,” in 2016 IEEE 14th Inter-
national Conference on Industrial Informatics (INDIN). IEEE, 2016,
pp. 411–415.
[12] C. Taştimur, E. Akın, M. Karaköse, and İ. Aydın, “Detection of rail faults
using morphological feature extraction based image processing,” in 2015
23nd Signal Processing and Communications Applications Conference
(SIU). IEEE, 2015, pp. 1244–1247.
[13] K. O’Shea and R. Nash, “An introduction to convolutional neural
networks,” arXiv preprint arXiv:1511.08458, 2015.
[14] M. C. Nakhaee, D. Hiemstra, M. Stoelinga, and M. van Noort, “The
recent applications of machine learning in rail track maintenance: A
survey,” in International Conference on Reliability, Safety, and Security
of Railway Systems. Springer, 2019, pp. 91–105.

Fig. 6. System’s result samples

detection while inspection using 2DCNN. While the robot is

inspecting, the captured images are sent to the neural network
for detection. Once any surface defect is detected, it will be
communicated directly to the cloud with the corresponding
location for further inspection later. The proposed system has
achieved an accuracy rate of 97%. In the future, more sensors
will be added to detect internal defects, and the neural network
will be trained on GPUs to speed up the training time and
enhance the accuracy rate.

R EFERENCES
[1] N. Mahfuz, O. A. Dhali, S. Ahmed, and M. Nigar, “Autonomous railway
crack detector robot for bangladesh: Scanobot,” in 2017 IEEE Region
10 Humanitarian Technology Conference (R10-HTC). IEEE, 2017, pp.
524–527.
[2] Q. Li, Z. Zhong, Z. Liang, and Y. Liang, “Rail inspection meets big
data: methods and trends,” in 2015 18th International Conference on
Network-Based Information Systems. IEEE, 2015, pp. 302–308.
[3] Q. Li and S. Ren, “A real-time visual inspection system for discrete
surface defects of rail heads,” IEEE Transactions on Instrumentation
and Measurement, vol. 61, no. 8, pp. 2189–2199, 2012.
[4] Y. Jiang, H. Wang, G. Tian, Q. Yi, J. Zhao, and K. Zhen, “Fast
classification for rail defect depths using a hybrid intelligent method,”
Optik, vol. 180, pp. 455–468, 2019.
[5] M. Kumar, M. S. Murali, M. Saranya, S. Arun, and R. Jayakrishnan,
“A survey on crack detection technique in railway track,” in 2018
Conference on Emerging Devices and Smart Systems (ICEDSS). IEEE,
2018, pp. 269–272.
[6] M. C. Nakhaee, D. Hiemstra, M. Stoelinga, and M. van Noort, “The
recent applications of machine learning in rail track maintenance: A
survey,” in International Conference on Reliability, Safety, and Security
of Railway Systems. Springer, 2019, pp. 91–105.
[7] X. Gibert, V. M. Patel, and R. Chellappa, “Robust fastener detection
for autonomous visual railway track inspection,” in 2015 IEEE Winter
Conference on Applications of Computer Vision. IEEE, 2015, pp. 694–
701.
[8] J. Gan, Q. Li, J. Wang, and H. Yu, “A hierarchical extractor-based visual
rail surface inspection system,” IEEE Sensors Journal, vol. 17, no. 23,
pp. 7935–7944, 2017.
[9] B. Sambo, A. Bevan, and C. Pislaru, “A novel application of image
processing for the detection of rail surface rcf damage and incorporation
in a crack growth model,” 2016.
[10] M. Karakose, O. Yaman, M. Baygin, K. Murat, and E. Akin, “A
new computer vision based method for rail track detection and fault
diagnosis in railways,” International Journal of Mechanical Engineering
and Robotics Research, vol. 6, no. 1, pp. 22–17, 2017.

631
Authorized licensed use limited to: INSTITUTE OF ENGINEERING & MANAGEMENT TRUST. Downloaded on February 27,2025 at 16:20:25 UTC from IEEE Xplore. Restrictions apply.