Challenges in Information, Communication and Computing Technology – V. Sharmila et al.

(Eds)
© 2025 The Author(s), London, 978-1-032-90166-4
Open Access: www.taylorandfrancis.com, CC BY-NC-ND 4.0 license

Deepfake detection using fine-tuned VGG16 model: A transfer

learning approach

Aditya Vikram Singh, Devshree Moghe & K. Meenakshi

Department of NWC, SRMIST, Kattankulathur, Tamil Nadu, India

ABSTRACT: Surge of deep fake videos, enabled by advancements in AI, present a grave
challenge to the trustworthiness of visual material. To confront this issue, we present a fresh
deep fake detection system that relies on facial embeddings and fine-tuned VGG16-based
models. Through capturing subtle facial features and rigorous training with meticulously
curated datasets, our system contributes in distinguishing authentic content from manipu-
lated ones. Equipped with a robust real-time video processing pipeline, our approach acts as
a proactive shield against the proliferation of false information, thereby preserving the
credibility of digital media. By amalgamating deep learning and transfer learning meth-
odologies, our system provides a reliable strategy to combat the malevolent use of deep fake
technology. Leveraging sophisticated algorithms and evaluation metrics, we aim to mitigate
the risks associated with digital manipulation, ultimately upholding the trustworthiness and
authenticity of visual content in the intricate landscape of the digital age.

Keywords: MTCNN, CNN, VGG16, Inception Res Net V1, Deep fakes, Transfer Learning

1 INTRODUCTION

The rise of deepfakes offers a serious threat to the trustworthiness of visual content. These
advanced machine learning techniques can alter videos by swapping or modifying faces,
raising concerns about their potential to mislead and spread false information. As deepfake
technology becomes more accessible, the need for robust detection methods to combat
widespread deception is growing.Our research introduces a new deep fake detection system
that uses facial embeddings and a fine-tuned VGG16 model. By leveraging detailed facial
characteristics and a well-curated dataset, our system aims to accurately differentiate
between authentic and manipulated faces.
Combining Multi-Task Cascaded Convolutional Neural Networks (MTCNN), Inception
Res Net V1, and a finely tuned VGG16 model, we enhance our ability to detect altered
content, making a significant contribution to deep fake detection. This effort not only
addresses a crucial gap in knowledge but also shows promise in preserving the authenticity of
digital media and rebuilding trust in visual content amidst the challenges of the digital age.

2 LITERATURE SURVEY

With the ever-growing concern about the impact of deep fakes, researchers are developing
various strategies for their detection. Demir et al. [1] focused on extracting features and
assessing temporal consistency, using Support Vector Machines (SVM) or Convolutional
Neural Networks (CNN) to analyze biological signals in facial regions.
Deep learning approaches are increasingly important in deep fake detection, with
DARPA even starting a research program dedicated to this field [2,3]. Studies have explored
methods that leverage temporal, spatial, and spectral inconsistencies using multimodal deep
learning [5]. This approach, along with the work presented in [6], which uses a pair-wise deep

DOI: 10.1201/9781003559085-141 825

This chapter has been made available under a CC BY-NC-ND 4.0 license
learning model on the Celeb A dataset, has shown promising results, with accuracy rates
exceeding 90% in deep fake detection.
Using Convolutional Neural Networks (CNNs) is another popular method, with
advancements like Meso Net incorporating dilated convolutions to capture multiscale
information [9]. These CNN-based approaches often target facial distortions or use high-
pass filters to detect statistical irregularities introduced by GANs [9,10]. While they generally
show high accuracy, the lack of standardized datasets across various methods makes it dif-
ficult to directly compare and identify the best approach [11].
More recent research has focused on improved image classification models for videos,
merging some of the best aspects of existing architectures to create more robust and
adaptable systems [13]. Additionally, deep learning frameworks based on the VGG16 archi-
tecture has achieved impressive accuracy rates exceeding 90% in deep fake detection, with
further advancements integrating CNNs to achieve even higher accuracy of 94% [14,15].
Facial embeddings and fine-tuning deep learning models like VGG16 have proven parti-
cularly effective, with the integration of algorithms like CNN and LSTM showing significant
promise in reducing false positives and improving detection rates [7]. This highlights the
importance of robust feature selection and model optimization in deep learning approaches
for deepfake detection. Despite these advancements, further exploration is needed to address
challenges concerning latency and reliability in deep fake detection systems [10].

3 PROPOSED SYSTEM

Deep fakes pose a challenge to the reliability of digital media, demanding for efficient stra-
tegies for detecting and mitigating deep fakes. Researchers have leveraged some advanced
ML techniques, specifically deep learning, to address this issue. One effective approach
involves using fine-tuned Convolutional Neural Networks (CNNs) and facial embeddings,
which capture essential facial features for classification. These embeddings, derived from the
pretrained InceptionResNetV1 model, enable accurate differentiation between genuine and
deep fake visuals. This approach of using the knowledge gained from one task to enhance
performance on another is known as Transfer Learning, as illustrated in the Figure 1.
The process of deep fake detection involves several steps. First, a diverse dataset of
authentic and deep fake media is collected and prepared. Video frames and facial regions are
extracted to focus on key features for classification. These cropped images are then used to
generate facial embeddings. The fine-tuned VGG16 model, renowned for its ability in image
classification, is utilized for deep fake detection. It involves customizing the top layers and
adding custom fully connected layers on top of the base model. The model is then trained on
the extracted facial embeddings, with labels indicating whether the content is authentic or
deep fake. In addition to the above, this paper presents a unique approach to identify deep
fake videos, combining facial embeddings with a fine-tuned deep learning model designed
specifically for deep fake classification. The core of the system involves adjusting the deep
learning model to classify faces as genuine or manipulated by utilizing facial embeddings
extracted from cropped face areas in video frames. The model undergoes extensive training
on a carefully curated dataset containing both authentic and manipulated facial embeddings,
with its top layers customized to detect subtle indicators of manipulation. Integrated with
video processing techniques, this method enables efficient deep fake detection at both frame
and video levels.
After training, the model was evaluated using a separate test dataset to determine how
well it detects deep fakes. We calculate performance metrics such as precision, accuracy, F1-
score, and recall. To improve the model’s performance and its ability to generalize, we might
use optimization techniques like hyperparameter tuning and regularization. In summary,
using fine-tuned CNN architectures with facial embeddings shows great promise for deep
fake detection. This approach effectively combines deep learning and computer vision
techniques, providing a strong defense against the spread of deepf ake videos and helping to
maintain the integrity of multimedia content in the face of growing digital manipulation.

826
Figure 1. Transfer learning.

4 RESULT AND ANALYSIS

The result report outlines performance of the deepfake detection model on a test dataset,
comprising 450 samples. Table 1 in the report assesses the precision, recall, and F1 score for
each category, as well as the overall accuracy.

Table 1. Result of the fine-tuned model.

Precision Recall F1 Score Support

Fake 0.98 0.95 0.96 225

Real 0.95 0.98 0.96 225
Accuracy 0.96 450
Macro Avg 0.96 0.96 0.96 450
Weighted Avg 0.96 0.96 0.96 450

4.1 Precision
Precision gauges the correctness of positive predictions. For the FAKE category, the cal-
culated precision is 0.98, meaning that 98% of the samples identified as FAKE are truly
FAKE. In contrast, for the REAL category, the precision is 0.95, suggesting near-perfect
accuracy, indicating that 95% of the samples labeled as REAL are accurate.

4.2 Recall
Recall, also referred to as sensitivity, gauges the percentage of true positives accurately
identified by the model. The model attained a recall score of 0.95 for the FAKE category,
indicating that it correctly classified all FAKE instances. Conversely, the recall for the
REAL category stands at 0.98, denoting that 98% of REAL instances were correctly
identified.

4.3 F1 score
The F1-score, which is calculated as the harmonic mean of precision and recall, provides a
balanced evaluation of a classifier’s performance. Specifically, the F1-score of 0.96 for the
FAKE class signifies a strong equilibrium between precision and recall, while the REAL
class’s F1-score of 0.96 indicates a satisfactory balance between precision and recall as well.

827
5 CONCLUSION

In conclusion, we aimed to address the urgent issue of identifying deep fake videos, which
present considerable risks across multiple domains such as media, politics, and cybersecurity.
By using machine learning methods, specifically CNNs, we aim at making a detection sys-
tem that can effectively tell the difference between real and fake videos. Following an
extensive assessment using a varied dataset, our model showcased impressive performance,
attaining an overall accuracy rate of 96% on the test dataset.. Our analysis shows promising
results in the model’s performance, with high precision and recall for both FAKE and
REAL classes, maintaining a balanced F1-score despite a minor recall discrepancy. This
underscores our methodology’s effectiveness in combating deep fake content and mis-
information spread. Our study contributes to deep fake detection progress by offering a
solution applicable across domains. The performance analysis curve depicted in Figure 2
provides a detailed breakdown of the model’s performance.

Figure 2. Performance analysis (Epoch Vs Accuracy).

Figure 3. Confusion matrix.

Figure 3 confusion matrix gives a detailed overview of how well the model is classifying
instances into the correct categories.

828
 In conclusion, our research represents significant advancement in deep fake detection,
providing an improved framework for advanced detection systems to uphold digital integrity
in digital space.

6 LIMITATIONS AND FUTURE SCOPE

Detecting deepfake videos successfully is a notable achievement, yet it’s crucial to recognize
limitations and chart future research paths. Enhancing deep fake detection models involves
addressing pivotal challenges, notably minimizing data bias. Ensuring a diverse, representative,
and high-quality training dataset is essential. Future efforts should prioritize creating a robust
dataset to boost model accuracy. Improving the model’s generalization across diverse deep fake
techniques and environmental conditions is crucial for robustness, necessitating strategies to
enhance adaptability and resilience to new data. Addressing vulnerabilities to adversarial attacks
is equally vital to maintain system trustworthiness, requiring the development of defensive
measures. Enhancing interpretability is key for understanding biases and rectifying errors effec-
tively. Additionally, scalable real-time processing systems are essential for applications needing
immediate responses, urging exploration of innovative algorithms and infrastructure.
In summary, addressing data bias, enhancing generalization, mitigating adversarial
threats, optimizing resource usage, considering ethical implications, improving interpret-
ability, and implementing real-time processing capabilities are pivotal areas for advancing
deep fake detection technology.

REFERENCES

[1] Ciftci, U., Demir, I. and Yin, L. (2020). Fake catcher: Detection of synthetic portrait videos using
biological signals. In IEEE Transactions on Pattern Analysis and Machine Intelligence: 1–1.
[2] Passos Júnior, L., Jodas, D., Costa, K., Souza Jr, L., Rodrigues, D., Del Ser, J., Camacho, D. and
Papa, J. (2023). A Review of Deep Learning-based Approaches for Deep Fake Content Detection.
[3] Semantic Forensics (SemaFor) Proposers Day Integrity Assessment of Visual Med Dr. Matt Turek
Systematizing Confidence in Open Research and Evidence (SCORE).
[4] Lewis, J., Toubal, I. E., Chen, H., Sandesera, V., Lomnitz, M., Hampel-Arias, Z., Prasad, C. and
Palaniappan, K. (2020). Deepfake Video Detection Based on Spatial, Spectral, and Temporal
Inconsistencies using Multi-modal Deep Learning.
[5] Munir, K., Raza, A. and Almutairi, M. (2022). A novel deep learning approach for deep fake image
detection. In Applied Sciences: 12.
[6] Zobaed, S., Rabby, F., Hossain, I., Hossain, E., Hasan, S., Karim, A. and Hasib, K. M. (2021).
Deepfakes: Detecting forged and synthetic media content using machine learning. In Artificial Intelligence
in Cyber Security: Impact and Implications: 177–201. Berlin/Heidelberg, Germany: Springer.
[7] Arshed, M. A., Mumtaz, S., Ibrahim, M., Dewi, C., Tanveer, M. and Ahmed, S. (2024). Multiclass
AIGenerated deepfake face detection using patch-wise deep learning model. In Computers: 13.
[8] Rana, M. S., Murali, B. and Sung, A. H. (2021). Deepfake detection using machine learning algorithms,
IIAIAAI.
[9] Afchar, D., Nozick, V., Yamagishi, J. and Echizen, I. (2018). MesoNet: A Compact Facial Video
Forgery Detection Network.
[10] Shi, W., Jiang, F. and Zhao, D. (2017). Single Image Super Resolution with Dilated Convolution Based
Multi-scale Information Learning Inception Module, CoRR.
[11] Li, Y. and Lyu, S. (2018). Exposing DeepFake Videos By Detecting Face Warping Artifacts.
[12] Mo, H., Chen, B. and Luo, W. (2018). Fake Faces Identification via Convolutional Neural Network.
[13] Khan, S.A., Artusi, A. and Dai, H. (2021). Adversarial Robust Deepfake Media Detection Using Fused
Convolutional Neural Network Predictions.
[14] Lakshmi, K., Divyashree, M., Poornima, M. C., Puthineedi, R. and Tanmayee, R. (2023). An All-
inclusive Study on Identification of Fake Multimedia Content Using Machine Learning approaches: A
Survey. In IARJSET.
[15] Munir, K., Raza, A. and Almutairi, M. (2022). A novel deep learning approach for deep fake image
detection. In Applied Sciences:

829