0% found this document useful (0 votes)
11 views

2 Mayank_Vatsa

Uploaded by

Vijay K
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
11 views

2 Mayank_Vatsa

Uploaded by

Vijay K
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 39

From Sight to Sound: Generative AI

in Law Enforcement Applications

Mayank Vatsa
Professor and Swarnajayanti Fellow
Fellow, IEEE and IAPR
सृजन, COE on Generative AI
IIT Jodhpur, India

[email protected]
http://iab-rubric.org/
We are stepping into the
world of Foundation Models
Foundation Models

Discriminative Models Generative Models

Language
Vision Models Multimodal Audio Models
Models
We are stepping into the
world of Foundation Models
Foundation Models

Discriminative Models Generative Models

Language
Vision Models Multimodal Audio Models
Models
From Sight to Sound …

Foundation Models

Discriminative Models Generative Models

Language
Vision Models Multimodal Audio Models
Models
What is Generative AI
(or GenAI)?
Generative AI is a category of artificial intelligence models designed
to create new content by learning patterns from existing data.

Images, text, music, videos, and audio

Generative AI represents a signi cant


leap in AI capabilities, offering tools
that enhance creativity, ef ciency, and
innovation across industries. However,
its use also raises ethical questions
regarding authenticity, bias, and misuse,
emphasizing the need for responsible
development and application.

All are generated


fi
fi
How GenAI Can be used in
Law Enforcement?
I will give three examples, how we have helped the
agencies with GenAI research

Sketch to Photo Generation

Face Recognition with Injuries

Deepfake Detection
Example 1:
Sketch to Photo Generation
Sketch to Photo Generation

A recent popular case


of NEET:
Fair colour
Young 30 years
Sketch from
Healthy
witnesses Face is matured
Double chin
Facial attributes Sample skin tone

Task: generate a photo

We followed an iterative process to solve


Sketch to Photo Generation

Input Sketch

Sample Outputs from Iterative Generation


Input Sketch

Selected Output
Sketch to Photo Generation

Input Sketch Refined Generative Actual Photo


Output
Example 2:
Face Recognition with Injuries
Face Recognition with
Injuries
Injured face recognition

Injuries are common in road


accidents, violence, and natural
calamities

Every year, 1.25 million people are


killed and 50 million are injured
in road accidents worldwide

50% to 70% of people surviving


traffic accidents have facial
injuries
We started working on this
problem in 2018 …

Till 2019, no dataset to do research on this problem

IF-V2 dataset - 150 subjects


How existing algorithms
perform?
Rank-1 Rank-5

VGG-Face 4.28 ± 1.9 15.55 ± 2.7

OpenFace 4.75 ± 1.5 19.55 ± 1.3

LCNN-9 46.65 ± 6.4 68.18 ± 3.9

LCNN-29 63.93 ± 3.6 85.48 ± 2.0

Recognizing Injured Faces via SCIFI Loss. IEEE Trans. Biom. Behav. Identity
Sci. 3(1): 112-123 (2021)
Why it is challenging?
Generating Synthetic Injured Faces
to Train Face Recognition Model

(a)

(b)

Real and Synthetic Faces for Training Results on Real Data Results on Synthetic Data
Orissa Train Collision Incident

On 2 June 2023, three trains collided in Balasore


district, in the state of Odisha, India

Over 260 people were killed and 1,000 were injured


in the crash

Image source: Arabinda Mahapatra—AP


One of the hardest case
study

Combined with mobile


data, the gallery set was
matched with the images
of dead bodies

Top-k possible matches


were provided by
multiple matchers
The Impact

We were able to identify over 120 unclaimed dead


victims using our approach in less than 18 hours

Opens a new thread of research with opportunity to


assist government agencies in cases of accidents,
natural calamities and unfortunate events.
Example 3:
MultiLingual Deepfake Detection
Which one of them are real
and fake?
Which one of them are real
and fake?

Instagram and Telegram have several AI profiles


And its easy
IJCB2022 paper
Major problem

Local context - voice, language, skin color:


challenges to existing deepfake detection

Most of the tools are trained on specific data and


English language

We have developed models for image/video and


audio deepfake detection - in local context
Pop Up Quiz

1 2 3 4 5 6

Which ones are real and which ones are fake?


Pop Up Quiz

1 2 3 4 5 6

Indian Irish Arabic Scottish UK US


Accent - Accent Accent Accent Accent Accent
North East
Faking Fluent: Unveiling the Achilles'
Heel of Multilingual Deepfake Detection

Svarah Dataset(Indian Accent English) UK English Dataset(UK Accent English)

Accent Bias in Audio Deepfake Detection


Utilising Motion Data

Motion in the video


can be enhanced to
“catch” irregularities
in doctored samples

Enhancing saccadic motion


http://people.csail.mit.edu/mrub/evm/
Multi-lingual Audio
Deepfake Detection
Created a dataset of 10 Indian languages + English

Trained our model on this dataset

Evaluated on test dataset - over 95% accuracy

Solving real cases in Indian context as well as


international cases
Example Solved Cases

Synthetic

Real samples

Real but AI Enhanced


The Impact

Multiple cases:

A case in a village in India: a


pornographic clip was circulated
which was classified as fake by our
system

Detecting morphed faces in ID


photographs

Detecting deepfakes in social media


GenAI is not a magic wand
- handle with care

Note of caution: we are excited about GenAI and it


is powerful; but it is also not perfect and it shows
some challenging cases
GenAI is not a magic wand…

imagine three young Indians in starbucks coffee shop


GenAI is not a magic wand…

Dalle-3 shows biased behaviour


GenAI is not a magic wand…

During our research on large models, we observe that


models do not understand negation
सृजन: Center for Generative AI

Responsible GenAI
Bias and fairness
Synthetic media detection
Model editing and generalization

Foundational Horizontals Application Verticals


Acknowledgements

#inevitableindia Thank You #innovateIndia

You might also like