International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

The Virtual Soldier: Detecting, Recognizing, Tracing, Informing

Criminals as well as Crimes in Real World
Aditya Tripathi1, Tapan Poojary2, Abhishek Yadav3, Jaya Jeswani4
1-4Department of Information Technology Xavier Institute of Engineering Mumbai, India
----------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - Finding criminals or hunting for people, in a in cybercrime research community for detecting criminal
CCTV video footage, after a crime scene or major attack takes behaviors in social media. Due to inherent diversity and
therefore the low duplicate property of images on social
place, is a time-consuming task. Pattern Analytical System and
media, it brings forth many challenges in image retrieval,
Data Mining techniques are powerful tools that helps in especially in identifying distinct features for a given object-
removing covered up information from a huge dataset to of-interest. The quick and accurate identification of criminal
upgrade exactness and productivity of estimating. activity is paramount to securing any residence. With the
rapid climb of smart cities, the mixing of crime detection
Traditional prediction frameworks are hard to manage the systems seeks to enhance this security. In the past a robust
enormous information and exactness of crime detection. As reliance has been placed on standard video surveillance so
informed to us by cyber cell members, they make multiple as to realize this goal. This often creates a backlog of video
data that has got to be monitored by a supervising official.
members of the department sit with laptops and computers
For large urban areas, this creates an increasingly large
literally to search through the CCTV footage to find and trace workload for supervising officials which leads to an increase
the guilty, as they don't have the automated system for doing in error rate Automatic recognition of violence between
this task with them. This process is both time and labor individuals or crowds in videos in real time is very difficult
intensive. but at the same time, if it is implemented in a right direction
can provide a high accurate result.
In this research paper we have tried to survey the existing
technologies as well as we propose a new system for criminal Our Problem revolves around 4 different types of crimes i.e.
Detection & Recognition using OpenCV and Machine Learning,
which if used by our Crime Agencies would definitely help them 1) Crimes happening by the known criminal in real time.
to find criminals from CCTV footage. The proposed system can
2) Crimes happening by unknown criminal in real time.
not only help find criminals but if used properly on different
sites such as railway stations etc., can also help find missing 3) Crimes like mob lynching which normally happens in
children and people from the CCTV footage available from the crowd
respective site. Existing solutions use traditional face
recognition algorithms which can be troublesome in changing 4) Crimes like stealing of vehicles.
Indian environments especially factors like light, weather and
especially orientation and the second part of this project is 2. LITERATURE SURVEY
detecting the vehicle thefts in real time using OCR and sending
SMS to its owner for emergency. The first paper in this survey is also a survey of existing
technologies as well as a new system for criminal Detection
Key Words: Machine Learning, Prediction, Data Mining, & Recognition using Cloud Computing and Machine
Analytics, Estimation, Pattern, OCR, SMS, Django. Learning, which if used by our Crime Agencies would
definitely help them to find criminals from CCTV footage.
1. INTRODUCTION The proposed system can not only help find criminals but if
used properly on different sites such as railway stations etc,
Circuit Television Cameras (CCTV’s) are widely used to can also help find missing children and people from the
control occurrence of crimes in the surroundings. Although CCTV footage available from the respective site. Existing
CCTVs are deployed at various public and private areas to solutions use traditional face recognition algorithms which
monitor the surroundings there is no improvement in the may be troublesome in changing Indian environments
control of crimes. This is because CCTV requires human especially factors like light, weather and particularly
supervision which can cause human prone errors like orientation. Some CCTV are during a bad place and may get
missing of some important crime events by human while tilted leading to a wild increase in inaccuracy. This research
monitoring numerous screens recorded by CCTVs at same paper proposes to use Microsoft Azure Cognitive services
time and Retrieval of images with object-of-interest from a and Cloud system for implementation of the proposed
vast pool of social media images has been a research interest system. The next phase this research is trying to compare

© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2135
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

this proposed methodology with traditional techniques like without the consideration of weather datasets, on different
HAAR cascade to judge performance of the proposed System, types of crime committed. Their proposed methodology will
as it is important to have a high accuracy, for a project of this enable law enforcement to make effective decisions on
sensitivity [1]. appropriate resource allocation, including backup officers
related to crime type and location [6].
The second paper proposed a real-time violence detector
based on deep-learning methods. The proposed model In this work, an end-to-end deep neural network model for
consists of CNN as a spatial feature extractor and LSTM as the purpose of recognizing violence in videos is proposed.
relation learning method with attention on the three-factor The proposed model uses a pre-trained VGG-16 on ImageNet
(overall generality - accuracy - fast response time). The as spatial feature extractor followed by Long STM (LSTM) as
suggested model achieved 98% accuracy with speed of 131 temporal feature extractor and sequence of fully connected
frames/sec [2]. layers for classification purpose. The achieved accuracy is
near state-of-the-art. Also, they contribute by introducing a
In Third Paper we saw a machine learning based predicting new benchmark called Real- Life Violence Situations which
policing algorithm exploiting the modus operandi features of contains 2000 short videos divided into 1000 violence
a recent crime and the existing prior criminal records. The videos and 1000 non-violence videos. The new benchmark is
proposed model predicts and probabilistically shortlist the used for fine-tuning the proposed models achieving a best
potential suspects who could also be involved during a accuracy of 88.2% [7].
recent crime. Thereby, meaningful leads are produced to aid
the investigation process. In this paper, we take the Punjab This paper proposed deep representation-based model using
province of Pakistan as a case study and apply the proposed concept of transfer learning for violent scenes detection to
model on the real data collected from various police stations. identify aggressive human behaviors. The result reports that
The results show that the identification of potential suspects proposed approach is outperforming state-of-the art
proved vital for the cases which involves criminals who had accuracies by learning most discriminating features
previous criminal record [3]. achieving 99.28% and 99.97% accuracies on Hockey and
Movies datasets respectively, by learning finest features for
Fourth we saw a deep learning model based on 3D the task of violent action recognition in videos [8].
convolutional neural networks, without using hand-crafted
features or RNN architectures exclusively for encoding In this paper we came up with Crime Intension Detection
temporal information. The improved internal designs adopt System that detects crime in real time videos, images and
compact but effective bottleneck units for learning motion alerts the human supervisor to require the required actions.
patterns and leverage the Dense Net architecture to promote To alert the supervisors or nearby police headquarters about
feature reusing and channel interaction, which is proved to the occurrence of crime. We added SMS sending module to
be more capable of capturing spatiotemporal features and our system which sends SMS to concern person whenever
requires relatively fewer parameters. The performance of crimes are detected. The proposed system is implemented
the proposed model is validated on three standard datasets using Pre-trained deep learning model VGGNet19 which
in terms of recognition accuracy compared to other detects gun and knife in the hand of person pointing to some
advanced approaches. Meanwhile, supplementary other person. We also compared the working of two
experiments are carried out to evaluate its effectiveness and different pre-trained models like Google Net InceptionV3 in
efficiency [4]. training. The results obtained with VGG19 are more accurate
in terms of training accuracy. This motivated us to use
It presents different techniques for detection of such VGG19 with little fine tuning to detect crime intention in
activities from the video proposed in the recent years. This videos and pictures to beat the problems with existing
research study reviews various state-of-the-art techniques of approaches with more accuracy. And we made use of Fast
violence detection. In this paper, the methods of detection RCNN and RCNN these algorithms are documented as Faster
are divided into three categories that's supported RCNN this helps us to draw the bounding box over objects in
classification techniques used: violence detection using images like person, gun, knife and some untrained images
traditional machine learning, using Support Vector Machine are marked with N/A. Algorithms help for detection and
(SVM) and using Deep Learning [5] classifications of objects over images [9].
In this work, they employ statistical analysis methods and In next research papers, study is based on the analysis of
machine learning models for predicting differing types of faces, emotions, Ages and genders to identify the suspects.
crimes in ny City, supported 2018 crime datasets. They have Face recognition, emotion, age and gender identifications are
combined weather, and its temporal attributes like cloud implemented using deep learning-based CNN approaches.
cover, lighting and time of day to identify relevance to crime Suit’s identification is based on LeNet architecture. In the
data. They note that weather-related attributes play a implementation phase for the classification purpose, Keras
negligible role in crime forecasting. They have evaluated the deep learning library is used, which is implemented on top of
various performance metrics of crime prediction, with and TensorFlow. IMDb is the dataset used for the whole training
© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2136
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

purpose. Training is performed using in AWS cloud which is The above proposed approach will be implemented in the
more powerful and capable way of training instead of using next phase of this project which will help us superior verify
local machines. Real-time Video and pictures are taken for the performance and high accuracy of this approach with
the experiment. Results of the training and predictions are very low latency. Accuracy is of paramount importance for a
discussed below in brief [10]. project of this sensitivity. Also important is the ability to run
on any machine regardless of the how powerful the
In this paper the aim is to classify each facial image together hardware is. These both lead us to choose the Azure face API.
of the seven facial expressions considered during this study. The proposed methodology when implemented successfully
According to the characteristics of countenance recognition, could definitely be of great help to our Criminal agencies in
a replacement convolution neural network structure is detecting and finding of criminals, also search for missing
meant which uses convolution kernel to extract implicit people.
features and max pooling to scale back the dimensions of the
extracted implicit features. In comparison to AlexNet Proposed Model
network, we will improve the popularity accuracy about 4%
higher on the CK+ countenance database by the help of Batch
Normalization (BN) layer to our network. A facial expression
recognition system is constructed in this paper for the
convenience of application, and the experimental results
show that the system could reach the real-time needs [11].

In this paper, heterogeneous data-based danger detection

(HDDD) mechanism is proposed for reducing crime rates.
The HDDD mechanism may be a mechanism for detecting
dangerous situations (e.g., homicide and violence) supported
heterogeneous data that are data gathered from multiple
devices like television (CCTV) cameras, smartphones, and
wearable devices. The HDDD enables immediate detection of
dangerous situations and then reduces crime rates by
responding to the dangerous situations [12].

In this paper, they present CNN (Convolutional Neural

Network) in the use of detect knife, blood and gun from an
image. Detecting these threatening objects from image can Fig. 1. Overview of System
give us a prediction whether a criminal offense occurred or
not and from where the image is taken. They emphasized on We have divided our project in 4 modules and in the end,
the accuracy of detection so that it hardly gives us wrong it will be a whole system in which all these features will
alert to ensure efficient use of the system. This model uses be implanted and can be directly used by any of the
Rectified Linear Unit (ReLU), Convolutional Layer, fully CCTV Camera.
connected layer and dropout function of CNN to reach a
result for the detection. We use TensorFlow, an open-source Module 1: Known Criminals like e.g., Dawood using
platform to implement CNN to achieve our expected output. HAAR Cascade and TensorFlow:
The proposed model achieves 90.2% accuracy for the tested
dataset [13]. Since we have a lot of images of Dawood and lot of
information to…we can train our data with its face, eyes
3. GOAL AND SCOPE DEFINITION etc. Once the dataset is ready with all the known
criminal’s data, we can pass it to our system which is a
The aim of this paper is to provide a comprehensive system live CCTV Surveillance. So due to this if any particular
for detecting crime and recognizing criminals and sending moment camera recognizes a criminal and it gets
the appropriate information to the police body. matched with our dataset…It will give a message and
alarm to the guard who will be at that moment
As a future scope extension of crime detection and analysis monitoring the surveillance and also one message will go
will be to generate the crime hot-spots that will help in directly to the command center of police with the camera
deployment of police at most likely places of crime for any no. and location of the particular criminal.
given window of time, to allow most effective utilization of
police resources. The developed model will reduce crimes Due to this…if police want to catch the criminal it can go
and will help the crime detection field in many ways that is directly catch or if police want to track the criminal or
from arresting the criminals to reducing the crimes by navigates its whole plan…then it can also put an eye on
carrying out various necessary measures.

© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2137
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

it. and try to recognize its plan…or the reason of coming location of that crime will be directly sent to the nearest
to that location. police station.
All this will be happening at a real time. In today’s
scenario…first the crime is happened and after then police
investigates but with our system in between crime police can
reach to the location at a fast rate.

Fig 2: Image of criminal detected

Fig 4: Live crime detection

Module 4: Vehicle Theft:

Suppose there is a situation wherein someone parks his/her
Car or Bike in the parking lot of their society and in night, a
thief robbed the bike and leaving the societies gate, so our
system will work as an alarm system wherein in our database
we will have all the entries of cars and bikes number plate in
that society with their owner’s name and phone number. so,
our system using the cameras located on the gate will detect
and recognize the number plate and will send a SMS by
saying that your bike/car just leaved the gate.so by this
instead of owner getting understand about the robbery of
bike in morning he/she will get and instant message and can
catch the criminal faster.

Fig 3: Video consisting frame of Criminal detected

Module 2: A new Criminal…which is not in our dataset:

If a crime is done by a normal person then without having
any information about it. Police cannot catch him, in that with
our system since it is capturing face of all the people passing
through it…will try to scan that particular image to all the
official websites, Facebook, Instagram etc.
Since nowadays everyone has a Facebook account or
Instagram account…system can easily find out the name and Fig 5: LPN detecting method
place of the criminal.
Module 3: Live Crime detecting and passing it to the
control room (Mob-Lynching):
Crimes like Mob-Lynching, fights between groups are often
seen in India and all the people standing there they take
videos of crime instead of calling the police. So, in this
scenario our system with help of machine learning and
artificial intelligence will try to recognize the crime scene as a
normal scene or any anomaly or unwanted happening in the
scene. If it detects something uneven the images, videos, and

© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2138
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

REFERENCES

[1]. S.Shirsat, A.Naik, D.Tamse, J.Yadav, P. Shetgaonkar,

S.Aswale, “Proposed System for Criminal Detection and
Recognition on CCTV Data Using Cloud and Machine
Learning”, 2019 International Conference on Vision Towards
Emerging Trends in Communication and Networking
(ViTECoN), ISBN: 978-1-5386-9353-7, DOI:
10.1109/ViTECoN.2019.8899441.

[2]. AMR.Abdali, RFA.Tuma, “Robust Real-Time Violence

Fig 6: License Plate Number detection Detection in Video Using CNN And LSTM”, 2019 2nd
Scientific Conference of Computer Sciences (SCCS),
University of Technology - Iraq, ISBN: 978-1-7281-0761-5,
DOI: 10.1109/SCCS.2019.8852616

[3]. S. Ali, SA. Alvi, Atiq Ur Rehman, “The Usual Suspects:

Machine Learning Based Predictive Policing for Criminal
Identification”, 2019 13th International Conference on Open
Fig 7: The correct number plate detected Source Systems and Technologies (ICOSST), ISBN:978-1-
7281-4613-3, DOI:10.1109/ICOSST48232.2019.9043925.
4. CONCLUSIONS
[4]. J.Li, X.Jiang, T.Sun, K.Xu, “Efficient Violence Detection
In this research, we encountered several face recognition Using 3D Convolutional Neural Networks”, 2019 16th IEEE
algorithms along the way. Some were really accurate while International Conference on Advanced Video and Signal
some were really slow. Our research found that Striking a Based Surveillance (AVSS), ISBN: 978-1-7281-0990-9,DOI:
balance between the Accuracy & Speed is really difficult. 10.1109/AVSS.2019.8909883
Some of the algorithms we came across through means of
[5]. M.Ramzan, A. Abid, H. Khan, S. Awan, A. Ismail, M.
this research are HAAR, Eigen Faces, Cam shift, CNNs, Viola-
Ahmed, M. Ilyas, A. Mahmood, “A Review on state-of-the-art
Jones Algorithm, Gaussian, Eucledian distance, AdaBoost etc.
Violence Detection Techniques”, DOI
The most famous detection algorithm we came across 10.1109/ACCESS.2019.2932114, IEEE Access.
through this research was HAAR. We also came across and
[6]. L.Elluri, V.Mandalapu, N.Roy, “Developing Machine
learned about the Microsoft Azure, which bundle of the cloud
Learning based Predictive Models for Smart Policing”, 2019
computing services that are created by Microsoft which can
IEEE International Conference on Smart Computing
be used for building, testing, as well as deploying, and
(SMARTCOMP), ISBN: 978-1-7281-1689-1, DOI:
managing the applications and services using a global
10.1109/SMARTCOMP.2019.00053.
network of the data centres managed by Microsoft , which
has the various Face API services which is basically cloud [7]. M.Soliman, M.Kamal, M.Nashed, Y.Mostafa, B.Chawky,
based service which provides algorithms for analyzing the D.Khattab, “Violence Recognition from Videos using Deep
human faces found in images and videos. It is fast and
accurate and thus our survey and the research helped us [8]. Learning Techniques”, 2019 IEEE Ninth International
choose & use of Microsoft Azure to take care of our face Conference on Intelligent Computing and Information
recognition need. Systems (ICICIS), ISBN: 978-1-7281-3995-1, DOI:
10.1109/ICICIS46948.2019.9014714.
The above proposed approach will be implemented in the
next phase of this project which will help us superior verify [9]. S.Chackravarthy, S.Schmitt, Li.Yang,” Intelligent Crime
the performance and high accuracy of this approach with Anomaly Detection in Smart Cities using Deep Learning”,
very low latency. Accuracy is of paramount importance for a 2018 IEEE 4th International Conference on Collaboration
project of this sensitivity. Also important is the ability to run and Internet Computing, ISBN: 978-1-5386-9502-9, DOI:
on any machine regardless of the how powerful the 10.1109/CIC.2018.00060.
hardware is . These both lead us to choose the Azure face
API. The proposed methodology when implemented [10].A.Mumtaz, Ab.Sargano, Z.Habib, “Violence Detection in
successfully could definitely be of great help to our Criminal Surveillance Videos with Deep Network using Transfer
agencies in detecting and finding of criminals, also search for Learning”, 2018 2nd European Conference on Electrical
missing people.

© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2139
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 08 Issue: 03 | Mar 2021 www.irjet.net p-ISSN: 2395-0072

Engineering and Computer Science (EECS), ISBN: 978-1-

7281-1929-8, DOI: 10.1109/EECS.2018.00109.

[11]. U. Navalgund, Priyadarshini.K, “Crime Intention

Detection System Using Deep Learning”, 2018 International
Conference on Circuits and Systems in Digital Enterprise
Technology (ICCSDET), ISBN: 978-1-5386-0576-9, DOI:
10.1109/ICCSDET.2018.8821168.

[12]. K.Rasanayagam, K.S.D.D.C, W.A.D.D, S.N.T,

Dr.P.Samarashinge, S.E.R.Siriwardana, “CIS: An Automated
Criminal Identification System”, 2018 IEEE International
Conference on Information and Automation for
Sustainability (ICIAfS), ISBN: 978-1-5386-9418-3, DOI:
10.1109/ICIAFS.2018.8913367.

[13]. X. Chen, X. Yang, M. Wang,Ji. Zou, “Convolution

Neural Network for Automatic Facial Expression
Recognition”, 2017 International Conference on Applied
System Innovation (ICASI), ISBN: 978-1-5090-4897-7, DOI:
10.1109/ICASI.2017.7988558.

© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 2140