Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Tracking of Intruder using Video Analytics

Apoorva G O Sneha M
Dept. of Computer Science and Engineering Dept. of Computer Science and Engineering
R V College of Engineering R V College of Engineering
Bangalore, India Bangalore, India

Abstract:- Video analytics is a technology that processes a restricted areas, unattended places and in inflated possibility
digital video signal using a special algorithm to perform a security areas [19]. This is an offence, in the event of
security- related function [12]. There is a need to design an Intrusion Identification system, ”watching is trusting”, that is
automated trespass detection and early warning prediction visual perception and Intrusion detection are favor at the same
tool leveraging state-of-the-art machine learning time. In addition, an advantage over further devices like
techniques. Leveraging video surveillance through security microwave Doppler detector provided by the video-based
cameras [3]. In particular, they adopt a CNN-based deep intrusion detection system, infrared detector, magnetic wall,
learning architecture (Faster-RCNN) as the core vibration detector and etc, considering the growing illegal
component of solution. However, these deep learning- activity rate and the availability of powerful general purpose
based methods, while effective, are known to be processors and security cameras [14]at reasonable rates.
computationally expensive and time consuming, especially
when applied to a large amount of surveillance data [3]. Video digital surveillance intelligent protection that can
Given the sparsity of railroad trespassing activity, design a be used as a multi-purpose security device because systems
dual-stage deep learning architecture composed of an are economical, that is the video-based intrusion detection
inexpensive prefiltering stage for activity detection (IDS) system used [5]. This research and product development
followed by a high fidelity trespass detection stage for work is funded financially by an IMPRINT (Grant No.-
robust classification. The former is responsible for filtering 7794/2016) initiative of the Ministry of Human Resource
out frames that show little to no activity, this way reducing Development. Government of India, Secretariat for Housing
the amount of data to be processed by the later more and Urban Affairs, Government of India. The same platform
compute-intensive stage which adopts state-of-the-art that is used for security for other applications. For different
Faster- RCNN to ensure effective classification of applications, different versions of the IDS were suggested.
trespassing activity [3]. no vehicle entry zone, no parking However, these systems lack optimum online performance,
zone, smart home security [14], etc. such as fast , low execution memory, processing speed and
high accuracy [5].
Keywords:- Intruder Tracking, Trespass Detection, Video
Analytics, Security. The goal of this paper is to describe an AI framework
which can analyze live video feeds in real time to gather
I. INTRODUCTION useful information for safety purposes [1]. Specifically, this
study aims to yield the following deliverable:
The fundamental objective of computer vision is to em-
power computers to reproduce the essential elements of 1) To Design and Implement an object tracking system using
human vision, for example, movement recognition and scene Artificial intelligence algorithms [1].
under- standing [2]. To accomplish the objective of quick 2) Applying algorithm on various data sets to analyze the
motion observation, effective work has been done on visual behavior and performance [1].
object tracking. In visual object tracking, the object of interest 3) Identifying and detecting intruders and producing video
is sur- rounded with the bounding rectangular box and the alarm. Figure 1 shows a conceptual view of the system, in
constraints of this box are then estimated for each incoming which an AI algorithm can send live alerts to designated
video frame [2]. Intelligent security systems focused on video personnel by analyzing and identifying trespassing events
surveillance are commonly used to identify suspected crimes in live CCTV feeds. In addition, trespassing events are also
at an early level. This allows real-time sending of intimations recorded in a trespass event database containing video and
and Enables users to take preventive action to minimize the associated metadata [1].
damage. Malicious entry into vulnerable areas, such as

IJISRT21JUN914 www.ijisrt.com 1268

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
II. LITERATURE REVIEW objects. In consecutive frames, and by the moving object by
weigh up their respective centroid coordinate. The foreground
Major factors affecting video analytics are mentioned objects are known as an unknowing objects if that objects are
below static for a predefined period of time and its size is within the
• Lighting issues [7]. Not enough light causes poor camera predefined align [15].
quality; light in the wrong places can produce shadows,
glare and dark spots. Video analytics focused on Deep Learning for individual
• Detection ranges. Know how many pixels on target the tracking [22]. As people’s assets are growing, security and
system requires, and design the camera coverage surveillance have become a matter of great concern today.
accordingly [7]. Surveillance cameras are common and identifying criminals
• High false alarm rates. Wind in many regions can cause from a vast number of surveillance video frames is a
false alarms, as can sun glare, shadows and the presence of repetitive technique. A new intelligent approach using deep
other nuisances. learning was suggested in order to simplify the search process
• Transmission issues. Weak networks, wireless links and [9]. This technique takes gender, shirt pattern, and spectacle
improperly terminated video cable can create headaches status as input to figure out the entity as a person from the
[7]. video log. The efficiency of this approach achieves an
accuracy of 87 percent in identifying the person in the video
In this paper LiDAR Point Cloud Classification for frame.
Visual Object Detection. Neural Network-based approach to
object detection and tracking is considered for visual detection Real Time Object Detection and Video Monitoring
to aid the processing of the LiDAR point cloud, visual-LiDAR System Tracking System [23]. This presents a device capacity
calibration Not to fully assess the efficiency of this approach of real- time video surveillance by integrating the algorithm of
in publicly available data. object detection tracking in a low end edge computing
environment. One stage detection algorithms such as a single
The Industrial Surveillance Systems Improved shot detector have been developed and you only look once
Unsupervised Shift Detector [17]. A crucial role of intelligent (YOLO) at the expense of some precision and can be used
surveillance systems is to identify an intruder that is entering a for real- time systems. However, high performance hardware
prohibited area [11]. This work involves a detector shift such as a general-purpose graphics processing unit is needed
to segment the trespasser from the background (foreground to achieve excellent object detection efficiency and speed [4].
object). Due to the dual camera sensor like color or IR, light A system that can guarantee success in real time by adjusting
shifts, night, static, and camouflaged foreground artifacts, the to managing the cycle of object detection and tracking in
role suffers from the inherent disadvantages of change different edge computing environments has been proposed.
detectors. Gaussian Mixture Models model the history
(GMM). To overcome the different challenges raised by the Cloud-based environment intrusion detection systems
areas and object, like static foreground object, the adaptive (IDS) and future challenges. Crucial applications for cost-
context model update scheme is suggested. The comparison is effectiveness and quantifiable reasons are being additionally
done on three databases algorithms for the detection of connected to the network. Due to various problems, most IDS
unsupervised transition. structures are incorrect for cloud conditions. Intrusion
Detection Systems and also the various related threat problems
Curb Detection and Tracking Based on Optimization that are present in this paper are discussed in supplies
Frame- work [18] in Low-Resolution 3D Point Clouds. Curb direction.
detection and tracking in urban environments for autonomous
vehicle operation. Detecting curbs in different weather A deep learning method of approach to identification of
conditions because of their robustness and can provide reliable trespassing of object using data from video captured
distance measurement Indicates low performance in surveillance [24]. Railroad trespassing is a hazardous practice
intersection areas and can not differentiate between occlusion with serious risks to safety and security. Due to exceptionally
situation and intersection areas. high levels of patrolling, constant patrolling of suspected
trespassing areas is, however, infeasible. Elevated demands
An Approach using Context Subtraction for Unattended for services and staffing expenses. This raises the need to use
Object Detection by Contour Formation [21]. In public areas, state of the art machine learning to Creation of automatic
any unattended object is usually known as a suspicious object. intruder detection and prediction of warning technique. To
One of the demanding tasks of computer vision is the meet this need, For the Automated Railroad Trespassing
recognition of certain items in public places. Robust object Detection System, we suggest a new architecture uses video
detection system that detects safety violations promptly and surveillance information known as ARTS. The basis of this
reliably improves safety measurements Public locations, approach, we are implementing a CNN based deep learning
Shopping centers, malls, train stations, etc., for example. They architecture caliber of video processing [6]. However, these
are further classified into static objects by these foreground deep learning based technique, while successful, are called

IJISRT21JUN914 www.ijisrt.com 1269

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
computationally high and time taking, specially when apply to III. SYSTEM DESIGN
the high volume of surveillance information. Leveraging the
sparsity of Intruder activities on railway tracks, ARTS Detection of trespassing events in video feeds has many
corresponds to a dual- stage deep learning architecture challenges. There are a wide variety of configurations,
consisting of an inexpensive behavior detection pre-filtering environmental variables, and technical features of live data
stage followed by a deep neural network classification stage of streams watching railroads [1]. An AI built for trespass
high fidelity trespass. The resulting dual stage ARTS detection must have several fundamental performance
architecture is a flexible method capable of trading off qualities. It must accurately identify pedestrians and vehicles
accuracy with calculation time [6]. We show the effectiveness within the frame, unhindered by video artifacts, shadows, and
of our public domain surveillance data strategy, achieving a other distortions. Secondly, the AI must maintain accuracy in
0.87 f1 score when Keeping up with the massive volume of diverse environmental conditions (e.g., rain, snow, day, night,
video, maintaining a reasonable time and precision trade off. and fog). Finally, when analyzing a live video stream, the AI
must be able to process the frames with enough speed to
The fundamental objective of computer vision is to em- maintain a fast response time to possible trespassing events
power computers to reproduce the essential elements of [1].
human vision, for example, movement recognition and scene
under- standing. To accomplish the objective of quick motion To address these challenges a generalized AI framework
observation, effective work has been done on visual object for trespass detection which utilizes the combined techniques
tracking. In visual object tracking, the object of interest is of ROI and Mask R-CNN is proposed (Figure 2). After
surrounded with the bounding rectangular box and the defining the ROI, the Mask R-CNN analyzes frames of the
constraints of this box are then estimated for each incoming live video feed. If an unauthorized person or vehicle enters the
video frame [2]. ROI an alert would be sounded, and relevant trespass data
would be recorded to a database for later review and analysis
A number of research works have been attempted to [1]. A key part of Mask R-CNN performance is the training
improve the performance of the video-based intrusion data- set, which allows it to recognize objects. The COCO
detection system to make it more suitable for real-time data-set, consisting of many labeled images of everyday
applications [5].Video surveillance is among the most evolved scenes built for use in object-recognition research, was utilized
applications of object tracking. It can be defined as process of for this purpose. It was selected because of its depth (330,000
monitoring the area with the help of camera network Images), diversity (80 object categories), and timeliness
employed such that entire area is covered. Any unethical through its continual growth and refinement. In addition, the
activity can be observed inside the control room and COCO data-set includes pre-generated boundaries around
accordingly action can be taken. There are enough scopes to recognized images allowing for better object recognition. By
improve it with respect to the accuracy [5], processing speed providing the Mask R-CNN with this data-set it can recognize
and time latency. Most of the proposed techniques for object people, cars, trains, and other objects within the ROI. If an
detection rely on tracking information to detect drop-off illegal object is detected within the ROI a subroutine of the AI
events [10]. these methods are not well suited in complex will execute two simultaneous commands. First, an alert SMS
scenes involving crowds and large amount of occlusions [2]. text or email is relayed to a pre-determined user [1]. This can
They have used alarm system to provide alert against any be a railroad safety official who can decide of possible
motion detected. Include sensors for recognition of body respiratory actions. Second, a clip of the trespass incident is
temperature for distinguishing between humans and animals. recorded and metadata, for example object detected, time,
location, video file name, and so forth, are stored in a trespass
event database. These metadata are automatically generated by
the AI, demonstrating that context of the image can be
extracted and interpreted. Trespass data can provide valuable
information about hazardous environments and behaviors that
lead to trespassing events, which can inform education,
enforcement, and engineering strategies for trespass
prevention. In addition, the aggregation of these trespass
events has the potential to enhance railroad risk analyses in the
future. The AI frame- work should be trained to verify its
accuracy by having the algorithm analyze a video data-set
with established results [1]. Comparing the results of the data-
set to the known number of trespasses verifies the AI
algorithm’s performance. Additional data-sets, including
varying environmental conditions, should be tested with the
Fig. 1. Conceptual trespasser detection alert system using algorithm to verify its performance under diverse
artificial intelligence circumstances.

IJISRT21JUN914 www.ijisrt.com 1270

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
IV. IMPLEMENTATION

To ensure that the AI algorithm achieved maximum

accuracy a several-step validation plan was enacted [1]. Four
results of the analysis were possible: an illegal trespass occurs,
and a detection is recorded (true positives); no illegal trespass
occurs but a detection is recorded (false positive); a trespass
occurs, and no detection is recorded (false negative); and no
trespass occurs, and no detection is recorded (true
negative).

In the training section, the AI analyzed 129 h of live

video data and reported a conglomeration of correct and
incorrect trespassing identification as compared with ground
truth data acquired by manual review of archival footage [1].
Fig. 2. General AI framework for trespass detection. These mistakes were corrected by improving the algorithm,
and a recording of the live feed was reprocessed with the
In Fig.3, the background subtractor module is needed by updated algorithm to ensure that the false positives and false
all the parallel threads except object detection and tracking. negatives would not occur again, resulting in the algorithm
Hence background subtraction is done first through a achieving 100 percentage accuracy at this point. In the testing
synchronized thread and then the 3 modules commence. phase, two right-of-ways were analyzed with no intermittent
Multiple object tracking with detection, anomaly detection, program modifications. Over 100 live hours of combined right-
trespass detection and tampering detection are parallelly of-way footage was manually reviewed and compared with the
executed in the second level thread. Meanwhile the user results generated by the algorithm. To date, the program was
interface (UI) undertakes communication between these 100 percent accurate (no false negative or false positive)
threads as well. Final thread level combines these results 17 [1].
and video output is rendered in the Graphical User Interface
(GUI). GPU acceleration is utilized for anomaly detection and V. RESULTS AND DISCUSSION
object tracking to speed up the process.
To evaluate the performance of the detection and
tracking system. Here we show limited results for the sake of
brevity. The total number of frames present in the video are
430. Some example frames are shown as follows in order to
prove the efficacy [8].It explains that even if a new intruder
enters or old intruder leaves the restricted zone, the tracking
and association of each detection with the corresponding
tracking does not disturb.

In our work, loss functions provide a static

representation of how our model is performing. Our proposed
algorithm use loss function for optimization; thus, finding the
Fig. 3. Surveillance system architecture best parameters (weights) for our data. YOLO optimizes a
multi-part loss function that is composed of four parts during
training the model [4].

Fig. 4. shows the publishing of live video using Kafka video Fig. 5. shows the detection of object in live video streaming
streaming . with accuracy in percentage.

IJISRT21JUN914 www.ijisrt.com 1271

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig. 6. Restricted zone is selected.

Fig. 10. Graph of YOLO and N-YOLO accuracy according to
tracking intervals

VI. CONCLUSION

The including of Artificial Intelligence to find an answer to

the Computer vision function has out performing the image
processing approach of handling the functions. The selection
of suitable algorithm for a particular task is always a trade-off
between speed and accuracy. For real time application, a
generic, but successful video-based Intrusion Detection
System with a new intrusion detection mechanism are
Fig. 7. shows the Intruder detected in the ROI and classify the Identified where the users can pick the possible intruders from
object. the classes of list for which the model is being educated. In
addition, inside the camera view, the client can draw the
ROI using reference frame of any size and shape. A modern
intrusion detection low-computational algorithm. A real-time
notification is sent to the users via registered email ID when
the intrusion event is occur.

Fig. 8. shows the region which we are selected.

Fig. 11. Time-accuracy trade-off for varying AR

FUTURE WORK

To further validate this framework, the amount of data

reviewed will be increased. This will allow the AI algorithm to
experience more environmental conditions and possibly more
trespassing events [1]. A limitation of the current AI is the
Fig. 9. shows the Intruder detection of object in restricted area inability to differentiate between authorized personnel and
trespassers. Future research will apply transfer learning
techniques to update the AI’s library to recognize authorized
personnel through the identification of their personal

IJISRT21JUN914 www.ijisrt.com 1272

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
protective equipment and other unique features. These Direct Proceeding Computer
techniques will also be used for future research into the Science.10.1016/j.procs.2020.04.212
recognition of debris and non moving objects. [18]. Y.Jung,S.-W.Seoand S.-W.Kim,”Curb Detection and
Tracking in Low- Resolution 3D Point Clouds Based on
REFERENCES Optimization Framework,” in IEEE Transactions on
Intelligent Transportation Systems, vol.21, no.9,
[1]. journals.sagepub.com pp.38933908,Sept.2020,doi:10.1109/TITS.2019.293849
[2]. Aayushi Gautam, Sukhwinder Singh. ”Trends in Video 8.
Object Tracking in Surveillance: ASurvey” 2019 Third [19]. A.ShahbazandK.Jo, ”Enhanced Unsupervised Change
International conference on I- SMAC (IoT in Social, Detector for Industrial Surveillance Systems,” in IEEE
Mobile, Analytics and Cloud) (I-SMAC), 2019 Transactions on Industrial Electronics,
[3]. web.wpi.edu doi:10.1109/TIE.2020.3013747
[4]. Sudan Jha, Changho Seo, Eunmok Yang, Gyanendra [20]. Vignesh Kanna, J.S.Ebenezer Raj, Meena M,”Deep
Prasad Joshi. ”Real time object detection and tracking Learning based video analytics for person
system for video surveillance system”, Multimedia Tools tracking”,Conference:2020 International Conference on
and Applications, 2020 Emerging Trends in Information Technology and
[5]. Rashmiranjan Nayak, Mohini Mohan Behera, Umesh Engineering(icETITE)
Chandra Pati, San- tos Kumar Das.”Video-based Real-time ,DOI:10.1109/icETITE47903.2020.173
Intrusion Detection System using Deep-Learning for Smart [21]. Jha, S.,Seo, C.,Yang, E.et al. Real time object detection
City Applications”, 2019 IEEE International Conference and tracking system for video surveillance system. Multi
on Advanced Networks and Telecommunications Systems med Tools App l80,3981– 39
(ANTS), 2019 96(2021).https://doi.org/10.1007/s11042-020-09749-x
[6]. cps-vo.org [22]. M.Bashir, E.A.Rundensteiner and R.Ahsan, ”A deep
[7]. R COLLING. ”Electronic Security System Integration”, learning approach to trespassing detection using video
Hospital and Healthcare Security, 2010 surveillance data,”2019 IEEE International Conference
[8]. Aasma Shahid, Alina Tayyab, Musfira Mehmood, Rida on Big Data(Big Data),Los Angeles,CA, USA, 2019,pp.
Anum, Abdul Jalil, Ahmad Ali, Haider Ali, Javed Ahmed. 3535-3544, doi:10.1109/Big Data47090.2019.9006426.
”Computer vision based in- truder detection framework [23]. R.Nayak,M.M.Behera,U.C.Patiand S.K.Das,”Video
(CV-IDF)”,2017 2nd International Conference on based Real-time Intrusion Detection System using Deep-
Computer and Communication Systems (ICCCS), 2017 Learning for Smart City Applications,” 2019 IEEE Inter-
[9]. J.S. Vignesh Kanna, S M Ebenezer Raj, M Meena, S national Conference on Advanced Networks and
Meghana, S Mansoor Roomi. ”Deep Learning Based Telecommunications Systems (ANTS), Goa, India,
Video Analytics For Person Tracking”,2020 2019, pp.1- 6,doi:10.1109/ANTS47819.2019.9117960.
International Conference on Emerging Trends in [24]. Zaman A, Ren B, Liu X. Artificial Intelligence Aided
Information Technology and Engineering (ic-ETITE), Auto- mated Detection of Railroad Trespassing.
2020. Transportation Research Record.2019;2673(7):25-
[10]. research.ijcaonline.org 37.doi:10.1177/0361198119846468
[11]. Ajmal Shahbaz, Kang-Hyun Jo. ”Enhanced [25]. Zhipeng Zhang, Chintan Trivedi, and Xiang Liu.
Unsupervised Change Detector for Industrial “Automated detection of grade-crossing-trespassing near
Surveillance Systems”, IEEE Transactions on Indus trial misses based on computer vision analysis of surveillance
Electronics, 2020 video data”.Safety Science, 110:276–285, 2018.
[12]. Submitted to Trident University International [26]. A. Shahid et al., ”Computer vision based intruder
[13]. www.ijircce.com detection frame- work (CV-IDF),” 2017 2nd
[14]. dspace.nitrkl.ac.in International Conference on Computer and
[15]. Neelam Dwivedi, Dushyant Kumar Singh,Dharmender Communication Systems (ICCCS), Krakow,Poland,
Singh Kushwaha. ”An Approach for Unattended Object 2017, pp. 41-45, doi: 10.1109/CCOMS.2017.8075263
Detection through Contour Formation using Background [27]. Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming
Subtraction”, Procedia Computer Science, 2020 He,Bharath Hariha- ran, and Serge Belongie. Feature
[16]. Kalake, W.Wan and L.Hou,”Analysis Based on Recent pyramid networks for object detection.In Proceedings of
Deep Learning Approaches Applied in Real-Time Multi- the IEEE Conference on Computer Vision and Pattern
Object Tracking: A Review,”in IEEE Access,vol. Recognition,pages 2117–2125, 2017.
9,pp.32650-32671,2021, [28]. R.Radha and K.Kathiravan “Prevention of Monkey
[17]. Neelam Dwivedia, Dushyant Kumar Singh, Dharmender Trespassing in Agri- cultural Field Using Application
Singh Kush- waha,”An Approach for Unattended Object Agricultural Specific Flooding Approach in Wireless
Detection through Contour Formation using Background Sensor Network”IEEE International Conference on
Sub-traction”,Third International Conference on Techno- logical Innovations in ICT for Agriculture and
Computing and Network Communications,Science Rural Development,2017.

IJISRT21JUN914 www.ijisrt.com 1273

Volume 6, Issue 6, June – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
[29]. Wahyono and K. Jo, “Cumulative dual foreground
differences for illegally parked vehicles detection,” IEEE
Transactions on Industrial Informatics , vol. 13, no. 5,
pp. 2464–2473,2017.
[30]. Wahyono, A. Filonenko, and K. Jo, “Unattended object
identification for intelligent surveillance systems using
sequence of dual background difference,” IEEE
Transactions on Industrial Informatics, vol. 12, no. 6, pp.
2247–2255, 2016.

IJISRT21JUN914 www.ijisrt.com 1274