Journal Publication of International Research for Engineering and Management (JOIREM)

Volume: 05 Issue: 09 | Sept-2021

Empowering Deep Learning for Images: A Comparative Analysis of

Regularization Techniques in CNNs
Sultan Khaibar Safi
Information Technology University of Mumbai, Artificial Engineering and Robotics Engineering Mashhad
University of Technology

---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - The remarkable success of Convolutional
Key Words: Deep Convolutional Neural Networks (CNNs),
Neural Networks (CNNs) in image recognition and
Optimization Algorithms, Regularization Techniques, Deep
related tasks has been hampered by the ever-present Learning, Nesterov Optimization, Adam Optimization,
challenge of overfitting and the pursuit of robust AdaMax Optimization, Dropout Regularization, Empirical
generalization performance. This article meticulously Evaluation of Optimization for CNN Generalization,
Regularization Techniques for Improving CNN Performance
dissects and compares various regularization techniques
specifically designed to empower deep learning for
image tasks within the context of CNN architectures.
We embark on a rigorous exploration of fundamental 1. INTRODUCTION
techniques like L1 and L2 regularization, delving into
their theoretical foundations. We further unveil the The remarkable ascent of Convolutional Neural
intricacies of advanced methods such as Dropout, Data Networks (CNNs) in image recognition and related
Augmentation, Early Stopping, and the synergistic domains has revolutionized the way we interact with the
approaches of Elastic Net and Group Lasso digital world. From facial recognition software to the
regularization. Through a meticulous examination, we burgeoning field of self-driving cars, CNNs have
unveil the theoretical underpinnings of these techniques, unlocked a new era of possibilities. However, a
illuminate effective strategies for hyperparameter significant challenge remains: ensuring these powerful
selection, and elucidate their profound impact on model models can learn effectively and generalize well to
complexity, weight sparsity, and ultimately, the unseen data. Overfitting, the tendency of a model to
network's ability to generalize effectively. To become overly reliant on training data, hinders the
empirically validate these insights and solidify our robustness and generalizability of CNNs.
comparative analysis, we conduct controlled
Regularization techniques have emerged as essential
experiments utilizing benchmark image datasets. This
tools in the deep learning toolbox, specifically designed
empirical validation process sheds light on the efficacy
to combat overfitting. These techniques act as
of each technique. By meticulously analyzing the trade-
constraints during training, guiding the model towards
offs inherent in these diverse regularization approaches
simpler representations and preventing it from becoming
and their suitability for specific image data
overly complex. By strategically applying
characteristics and CNN architectures, this article
regularization, we can empower CNNs to learn more
empowers researchers with a comprehensive
effectively from training data, leading to enhanced
understanding of these techniques. Armed with this
generalization performance in real-world image tasks.
knowledge, researchers can make informed decisions to
optimize performance in deep learning tasks involving This article delves into a comparative analysis of various
images, ultimately propelling the field towards ever- regularization techniques specifically designed to
greater advancements. empower deep learning for image tasks within the
context of CNN architectures. We embark on a rigorous
exploration of fundamental techniques like L1 and L2

© 2021, JOIREM |www.joirem.com| Page 1

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

regularization, delving into their theoretical The Challenge of Overfitting

underpinnings and their impact on model behavior. We
further unveil the intricacies of advanced methods such A significant challenge in training deep learning models like
CNNs is overfitting. Overfitting occurs when a model
as Dropout, Data Augmentation, Early Stopping, and the becomes overly reliant on training data and fails to generalize
synergistic approaches of Elastic Net and Group Lasso well to unseen examples. This results in a model that performs
regularization. Through a meticulous examination, we exceptionally well on the training data but struggles to
accurately predict outputs for new data. Overfitting can be
not only illuminate the theoretical foundations of these
attributed to the high capacity of deep learning models,
techniques but also shed light on effective strategies for meaning they have the potential to learn complex
selecting hyperparameters, the crucial settings that relationships within the training data that may not be
govern the behavior of these regularization methods. generalizable. This can lead to the model memorizing specific
patterns in the training set instead of learning underlying
To solidify our comparative analysis and validate the features that are relevant to unseen data. Overfitting can also
manifest in increased model complexity, leading to longer
theoretical insights, we conduct controlled experiments training times and potential computational limitations.
utilizing benchmark image datasets. This empirical
validation process provides concrete evidence regarding Regularization: A General Overview
the efficacy of each technique in mitigating overfitting
and enhancing generalization. By meticulously Regularization techniques are a crucial set of tools employed
in deep learning to combat overfitting and improve
analyzing the trade-offs inherent in these diverse generalization performance. These techniques act as
regularization approaches and their suitability for constraints during the training process, preventing the model
specific image data characteristics and CNN from becoming overly complex and overly reliant on specific
architectures, this article empowers researchers with a features within the training data. By introducing
regularization, we guide the model towards learning simpler
comprehensive understanding of these techniques. and more generalizable representations of the data. This
Armed with this knowledge, researchers can make ultimately leads to a model that can perform well on both
informed decisions to optimize the performance of deep training and unseen data.
learning models in image-related tasks, ultimately
propelling the field towards ever-greater advancements. Existing Research on Regularization Techniques for CNNs

This introduction maintains the core elements of the Several regularization techniques have been developed and
applied to CNNs for image tasks. Here, we discuss some of
previous versions, but aligns them more closely with the
the most prominent approaches:
title "Empowering Deep Learning for Images: A
Comparative Analysis of Regularization Techniques in L1 and L2 Regularization
CNNs." It emphasizes the empowering effect of these
techniques on deep learning performance. These fundamental techniques penalize the model for having
large weight values. They are incorporated into the loss
function (L) that the model aims to minimize during training.
2. Literature Review
 L1 Regularization (LASSO): Introduces sparsity by
2.1 Background on Convolutional Neural Networks adding the absolute value of all weights (w) in the
(CNNs) network to the loss function:

Convolutional Neural Networks (CNNs) have revolutionized L(w) = L_data(w) + λ ||w||_1

the field of computer vision, achieving remarkable success in
tasks like image recognition, object detection, and image
where:
segmentation. Their architecture is specifically designed to
exploit the inherent spatial structure of image data. CNNs
utilize convolutional layers to extract local features from an  L_data(w) is the original data loss (e.g., cross-
image, followed by pooling layers for dimensionality entropy for classification)
reduction. Fully-connected layers at the end of the network  λ is a hyperparameter controlling the strength of the
integrate these features and perform classification or regularization
regression tasks. The ability of CNNs to learn hierarchical  ||w||_1 is the L1 norm, representing the sum of the
representations of image data has made them a cornerstone of absolute values of all weights in w
deep learning for image tasks.

© 2021, JOIREM |www.joirem.com| Page 2

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

This encourages some weights to become exactly zero, Early Stopping

reducing model complexity and potentially improving
generalization. This technique monitors the model's performance on a
validation set (D_val) during training. The validation loss
 L2 Regularization (Ridge Regression): Promotes (L_val) is tracked over epochs (training iterations). If the
weight decay by adding the squared norm of all validation loss fails to improve for a predefined number of
weights to the loss function: epochs (patience), the training process is terminated. This
prevents the model from overfitting to the training data (D)
L(w) = L_data(w) + λ ||w||_2^2 and allows it to focus on learning generalizable features.

where: Elastic Net and Group Lasso Regularization

 ||w||_2^2 is the L2 norm, representing the sum of These techniques combine L1 and L2 regularization or group
squares of all weights in w weights together for regularization. They are incorporated into
the loss function similar to L1 and L2:
This shrinks all weights towards zero, reducing the overall
model complexity and preventing overfitting.  Elastic Net: Combines L1 and L2 regularization:

Dropout Code snippet

L(w) = L_data(w) + λ_1 ||w||_1 + λ_2 ||w||_2^2
This technique randomly drops a certain percentage (p) of
activations (a) from neurons during training. This injects noise where:
into the training process and forces the network to learn robust
features that are not dependent on any specific neuron or  λ_1 and λ_2 are hyperparameters controlling the
group of neurons. Dropout is typically applied at fully- strength of L1 and L2 regularization, respectively.
connected layers:
This encourages both sparsity and weight decay, potentially
a_out = (1 - p) * a_in offering a balance between the benefits of L1 and L2.

where:  Group Lasso: Encourages sparsity within groups of

weights (w_g). Weights are grouped based on filters
 a_out is the output activation after dropout within a convolutional layer or connections between
 a_in is the original input activation specific layers. The L1 norm is applied to each
 p is the dropout probability (between 0 and 1) group:

4.3 Data Augmentation Code snippet

L(w) = L_data(w) + λ ||w_g||_1
This technique involves artificially expanding the training
dataset (D) by generating new images (x') through random This promotes feature selection within the network by driving
transformations (T) of existing images (x): some entire groups of weights to zero. This can be particularly
beneficial when dealing with large numbers of parameters or
Code snippet highly correlated features in the data.

D' = { T(x) | x ∈ D }

ommon transformations include flipping (horizontal/vertical),

rotating, cropping, scaling, color jittering (adding noise to
color channels), and adding random noise. Data augmentation
increases the diversity of the training data (D') and forces the
model to learn features that are invariant to such
transformations. This improves the model's ability to
generalize to unseen variations in real-world images.

© 2021, JOIREM |www.joirem.com| Page 3

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

Comparative Analysis of Regularization Techniques for Internal Parameter Regularization Techniques

Convolutional Neural Networks (CNNs)
 Strengths: L1/L2 regularization promotes sparsity
This section presents a comparative analysis of various and reduces model complexity. Dropout prevents co-
regularization techniques employed to improve the adaptation between neurons, improving
performance of CNNs in image classification tasks. generalization.
Regularization plays a crucial role in mitigating overfitting, a  Weaknesses: Choosing the optimal regularization
common challenge in deep learning models, where the model hyperparameter (e.g., L1/L2 lambda) can be crucial
performs well on training data but poorly on unseen data. By and requires careful tuning. Dropout can slightly
incorporating diverse regularization strategies, CNNs can increase training time and may not be effective in all
achieve better generalization capabilities, leading to more network architectures.
robust and reliable performance on real-world image
classification problems. Label Regularization Techniques

Categorization of Regularization Techniques  Strengths: Offers a promising approach to address

overconfidence in predictions. May be particularly
Regularization techniques for CNNs can be broadly beneficial for imbalanced datasets.
categorized into three main groups based on the targeted  Weaknesses: This area is relatively unexplored
aspect of the model they modify: compared to others. More research is needed to
understand the theoretical foundation and develop
1. Data Augmentation Techniques: These techniques more effective label-based regularization methods.
focus on artificially manipulating the training data to
increase its variability and complexity. This forces Recent Advancements and Trends
the model to learn more robust features that
generalize better to unseen images. Examples include Recent research has explored more sophisticated data
random cropping, flipping, rotation, color jittering, augmentation techniques like AutoAugment and
and cutout. RandAugment, which automatically search for optimal
2. Internal Parameter Regularization Techniques: augmentation policies during training. Additionally,
These techniques directly modify the model's internal techniques like Mixup, which mixes training images and their
parameters, such as weights and biases, to discourage labels, have shown promising results in improving
overfitting. Common examples include L1/L2 generalization.
regularization, dropout, and weight decay. These
methods penalize overly complex models and
promote sparsity in the weights, leading to simpler
models with better generalization.
3. Label Regularization Techniques: This category is
less explored compared to the others and focuses on
modifying the training labels. Techniques like label
smoothing and mixup introduce noise or
interpolation between labels to prevent the model
from becoming overconfident in its predictions.
Two different examples using Mixup. Extracted from
Reference Zhang Hongyi, Cisse Moustapha, Dauphin Yann
Comparative Analysis of Techniques N., and Lopez-Paz David. 2017. Mixup: Beyond empirical
risk minimization. arXiv preprint arXiv:1710.09412 (2017).
Here's a comparative analysis of some prominent techniques
within each category, highlighting their strengths and
weaknesses:
Comparative Analysis of Regularization Techniques
Data Augmentation Techniques
Having established the theoretical foundations and existing
 Strengths: Improves model robustness by exposing research on regularization techniques for CNNs in the
it to diverse image variations. Relatively simple to literature review, this section delves into a comparative
implement and computationally efficient. analysis. We explore the core functionalities of each technique
 Weaknesses: Finding the optimal augmentation and discuss the trade-offs and considerations associated with
strategy for a specific dataset and architecture can be their application.
challenging. May not be effective for all types of
image variations.

© 2021, JOIREM |www.joirem.com| Page 4

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

Core Concepts and Functionality example, Data Augmentation might be less effective
for very large or complex images.
We revisit the key concepts and functionalities of the  Model Complexity: Techniques like L1/L2
regularization techniques discussed earlier: regularization directly influence model complexity.
The optimal level of regularization may depend on
the specific CNN architecture employed.
 L1 and L2 Regularization: These fundamental
techniques penalize the model for having large  Hyperparameter Tuning: Most techniques require
weight values. L1 regularization encourages sparsity, careful hyperparameter tuning (e.g., L1/L2
driving some weights to become exactly zero. L2 regularization weight, dropout rate) to achieve
regularization promotes weight decay, shrinking all optimal performance. Finding the right balance can
weights towards zero but not necessarily to zero. be an iterative process.
Both techniques reduce model complexity and
prevent overfitting. Evaluation Metrics
 Dropout: This technique randomly drops a certain
percentage of activations (outputs) from neurons To assess the effectiveness of regularization techniques, we
during training. This forces the network to learn can utilize various metrics commonly used in image tasks:
robust features that are not dependent on any specific
neuron or group of neurons. Dropout introduces  Classification Accuracy: Measures the percentage
noise during training, preventing the model from of correctly classified images.
memorizing specific patterns in the data.  Precision and Recall: Capture the trade-off between
 Data Augmentation: This technique artificially true positives and false positives/negatives.
expands the training dataset by generating new  F1-Score: Combines precision and recall into a
images through random transformations like flipping, single metric.
rotating, cropping, or adding noise. Data  Mean Squared Error (MSE): Measures the average
augmentation increases the diversity of the training squared difference between predicted and actual
data and forces the model to learn features that are values (often used for regression tasks).
invariant to such transformations. This improves the
 Peak Signal-to-Noise Ratio (PSNR): Measures the
model's ability to generalize to unseen variations in
ratio between the maximum possible signal power
real-world images.
and the power of corrupting noise (often used for
 Early Stopping: This technique monitors the image quality assessment).
model's performance on a validation set during
training. If the validation performance stops
improving for a predefined number of epochs By analyzing these metrics alongside factors like training time
(training iterations), the training process is and model complexity, researchers can make informed
terminated. This prevents the model from overfitting decisions about which regularization technique to employ for
to the training data and allows it to focus on learning their specific application.
generalizable features.
 Elastic Net and Group Lasso Regularization: CutMix
These techniques combine L1 and L2 regularization
or group weights together. Elastic Net encourages Another strategy to improve classification results by mixing
both sparsity and weight decay, while Group Lasso inputs and labels is CutMix . Unlike Mixup, which averages
encourages sparsity within groups of weights. These labels based on the interpolation between images, CutMix
techniques can be particularly beneficial when replaces entire regions from a given input image and modifies
dealing with large numbers of parameters or highly the label by assigning weights proportional to the area
correlated features in the data. occupied by each class in the replaced region. For example, if
a cat image is replaced by an airplane image in 30% of its
Trade-offs and Considerations area, the label would be set to 70% cat and 30% airplane. This
strategy has been shown to significantly improve
While each regularization technique offers benefits, there are classification accuracy. Techniques like Grad-CAM that
inherent trade-offs to consider: visualize the most activated regions of a network can be used
to verify that CutMix generates heatmaps that more accurately
highlight the object of interest.
 Computational Cost: L1 regularization can be
computationally expensive for large models due to
the sparsity calculations. Dropout might increase
training time due to the need to recalculate
activations during each iteration.
 Data Characteristics: The effectiveness of some
techniques may vary depending on the data. For

© 2021, JOIREM |www.joirem.com| Page 5

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

CutBlur change not only produces a more reliable neural network but
also trains faster than the traditional approach due to the
Several deep learning tasks for image processing, such as smaller size of images used for training compared to
image classification and object detection, can benefit from inference. The proposed approach demonstrates the potential
data augmentation techniques. Existing methods like for improved results on other datasets when transfer learning
AutoAugment , Cutout , and RandomErasing demonstrate is used.
significant improvements by applying simple yet effective
transformations to training images. However, for super- Bag-of-Tricks
resolution (SR) tasks, there's a lack of research specifically
focused on regularization techniques. While these A critical point to consider is that the works analyzed here
aforementioned techniques can be applied and potentially frequently do not combine any other regularizer with their
improve results, they are not inherently designed for SR current research. Therefore, it's difficult to determine how two
problems. The only approach identified so far is CutBlur , regularizers might influence each other. The Bag of Tricks
which works by replacing a specific area on a high-resolution research investigates this by combining several known
(HR) image with a low-resolution (LR) version from a similar regularization methods, such as Mixup, Label Smoothing, and
region. The authors demonstrated that CutBlur helps the Knowledge Distillation. The ablation study reveals that
model generalize better on the SR problem and can also be significant improvements can be achieved by applying these
applied to reconstruct images degraded by Gaussian noise. methods cleverly in combination. For instance, a MobileNet
using this combination of methods improved its results by
almost 1.5% on the ImageNet dataset, which is a significant
gain. However, the research lacks a deeper evaluation of
methods for regularization between layers, such as Dropout.

REGULARIZATION BASED ON INTERNAL

STRUCTURE CHANGES
How Cutout works. Extracted from Reference DeVries
Terrance and Taylor Graham W.. 2017. Improved Regularization Based on Internal Structure Changes
regularization of convolutional neural networks with
cutout. arXiv preprint arXiv:1708.04552 (2017
Regularization methods can work in various ways. In this
article, we define internal regularizers as those that modify
Batch Augment weights or kernel values during training without any explicit
change to the input. This section is divided into two main
An important hyperparameter for training CNNs is the mini- parts:
batch size, which is used to calculate the gradient employed in
backpropagation. Typically, the GPU's memory limit is used  The first part provides a deeper description of how
for this hyperparameter to accelerate convergence during dropout works and explores some of its variants,
training. The Batch Augmentation work leverages this limit such as SpatialDropout and DropBlock.
cleverly. Instead of simply filling the entire memory with  The second part describes other methods that target
different instances from the dataset, it utilizes half of the operations on different tensors.
memory limit for the standard data augmentation setup and
then duplicates all instances with various data augmentation
4Dropout and Variants
possibilities. This approach may seem straightforward;
however, results demonstrate that neural networks using this
approach achieve significantly improved final results. Another Dropout is a simple yet powerful regularizer that aims to
noteworthy point is that the analysis showed fewer epochs are remove some neurons during training, forcing the entire
required for convergence when augmented images are network to learn more robust features. At each training step, a
duplicated. random subset of neurons is deactivated with a predefined
probability (typically 0.5). This prevents the network from
overfitting to the training data by encouraging it to develop
FixRes redundant pathways and avoid relying on any single neuron.

Image resolution can influence both training efficiency and SpatialDropout

final classification accuracy. For instance, the research on
EfficientNet highlights this concept by making the input size
one of the parameters influencing the final result. However, if While Dropout randomly deactivates individual neurons,
a model is trained with a resolution of, for example, 224x224 SpatialDropout focuses on deactivating entire contiguous
pixels, the same resolution is typically used for inference on regions within a feature map. This approach forces the
the test set. The work by proposes that the test set resolution network to learn more spatially robust features, as neighboring
should be higher than the resolution used for training. This neurons are more likely to be affected together.
SpatialDropout is implemented by randomly selecting
© 2021, JOIREM |www.joirem.com| Page 6
 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

rectangular regions within a feature map and setting all Combining Internal Regularization with Other
activations within those regions to zero during training. Techniques

4.1.2 DropBlock Internal regularization methods like dropout can be effectively

combined with other techniques to achieve even better results.
DropBlock extends the concept of SpatialDropout by
deactivating not just entire regions but also contiguous areas  Data Augmentation: Combining dropout with data
within the channels of a feature map. This approach augmentation techniques like random cropping,
encourages the network to learn more informative feature flipping, or color jittering can further improve the
representations by preventing it from relying on a small subset network's robustness to variations in the input data.
of channels within a feature map. DropBlock works by  Ensemble Learning: Training multiple networks
randomly selecting rectangular areas within a feature map and with different dropout masks and then averaging
setting all activations within those regions, across all their predictions (ensemble learning) can lead to
channels, to zero during training. more robust performance compared to a single
network.
Recent Advancements in Dropout Techniques
Research by explores the effectiveness of combining various
While Dropout and its variants have proven effective, recent regularization techniques, including Mixup, Label Smoothing,
research has explored alternative approaches to achieve and Knowledge Distillation, demonstrating significant
similar goals. improvements in classification accuracy.

 Variational Dropout: This method introduces Limitations and Open Questions

variational inference into the dropout process,
allowing the network to learn the optimal dropout While internal regularization techniques offer numerous
rate for each neuron during training. This can benefits, they also have limitations:
potentially lead to more efficient regularization
compared to fixed dropout rates.  Dropout deactivation: Deactivating neurons during
 Stochastic Weight Averaging (SWA): This training might discard valuable information,
approach involves accumulating the weights of the potentially hindering performance.
network across multiple training epochs. During  Hyperparameter tuning: Finding the optimal
training, a small noise term is added to the gradients, dropout rate or weight decay coefficient can be
and the exponentially moving average of these noisy challenging and requires careful hyperparameter
gradients is used to update the weights. SWA has tuning.
been shown to improve the generalization
performance of CNNs, potentially by mitigating the Open questions remain in the field of internal regularization:
effects of local minima.
 Network-specific methods: Can we design internal
Alternative Internal Regularization Techniques
regularization methods that are specifically tailored
to different network architectures or tasks?
Beyond dropout methods, several other techniques can be  Beyond dropout: Are there unexplored
employed for internal regularization: regularization techniques that offer even more
effective ways to prevent overfitting and improve
 Early stopping: This technique monitors the generalization?
validation loss during training. If the validation loss
fails to improve for a predefined number of epochs Further research in this area can lead to the development of
(patience), training is stopped to prevent overfitting. more powerful and efficient internal regularization methods
Early stopping allows the network to learn the for CNNs.
underlying patterns in the data without memorizing
the training examples themselves.
 Weight Decay: This technique penalizes large
weights in the network during training. By adding a
weight decay term to the loss function, the network is Additional Regularization Techniques
encouraged to learn smaller weights, leading to
smoother weight distributions and potentially better While data augmentation and modifications to the network's
generalization. Weight decay helps to prevent the internal structure offer powerful tools for regularization,
network from becoming overly complex and fitting several other techniques can be employed to prevent
to noise in the training data. overfitting and improve generalization performance in CNNs.
Here, we explore some additional noteworthy approaches:

© 2021, JOIREM |www.joirem.com| Page 7

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

 Early Stopping (Already Discussed): As briefly Recommendations for Choosing Regularization

mentioned earlier, Early Stopping monitors the Techniques
validation loss during training. If the validation loss
fails to improve for a predefined number of epochs
While the effectiveness of various regularization techniques
(patience), training is stopped. This technique
has been established, the optimal choice often depends on the
prevents the network from overfitting to the training
specific dataset and task at hand. Here are some general
data by focusing on learning the underlying patterns
recommendations to guide your selection, incorporating
without memorizing specific examples.
methods discussed earlier:
 Knowledge Distillation (KD): This technique
leverages the knowledge learned by a pre-trained,
powerful "teacher" network to improve the  Data Size and Complexity:
performance of a smaller, less complex "student" o Large Datasets: For very large datasets
network. During training, the student network is with abundant training examples, L2
trained not only on the original training labels but regularization might be sufficient. The
also on soft targets obtained from the teacher additional complexity of techniques like
network's predictions. Soft targets are probability Dropout or data augmentation might not be
distributions over all classes, providing richer necessary due to the inherent richness of the
information compared to one-hot encoded labels. data.
This process allows the student network to learn o Smaller Datasets or Imbalanced Classes:
from the teacher's knowledge, potentially achieving When dealing with limited data or
better generalization performance even with a imbalanced classes, techniques like
smaller capacity. Dropout, data augmentation, and Early
 Group Lasso: This regularization technique Stopping become more crucial. These
promotes sparsity in the network by penalizing methods artificially expand the training data
groups of weights instead of individual weights. (data augmentation), promote robustness
Here, groups can be defined based on filters within a (Dropout), and prevent overfitting on
convolutional layer or weights connecting specific smaller datasets (Early Stopping).
layers. By encouraging some groups of weights to be  Feature Characteristics and Sparsity:
driven towards zero, Group Lasso can lead to more o High-Dimensional Data with Redundant
interpretable models, where the remaining non-zero Features: L1 regularization, Group Lasso,
weights highlight the most important features for the or Elastic Net can be beneficial. These
network's predictions. techniques encourage sparsity by driving
 ℓ₁ Regularization: Similar to weight decay (L2 some weights to zero (L1, Group Lasso) or
regularization), ℓ₁ regularization penalizes the sum a combination of sparsity and weight decay
of the absolute values of the weights in the network. (Elastic Net), effectively performing feature
This penalty encourages sparsity by driving some selection and reducing model complexity.
weights exactly to zero, creating a more compact o Lower-Dimensional Data with
Informative Features: When dealing with
model. However, ℓ₁ regularization can be
datasets with a smaller number of
computationally less efficient compared to L2 and
might not always achieve the same level of informative features, L2 regularization
performance improvement. might be preferred. L1's tendency to drive
weights to zero might discard valuable
 Data Distillation: This technique can be seen as an
information in such cases.
alternative or complement to data augmentation.
Here, a more complex model is first trained on the  Task-Specific Considerations:
o Classification vs. Regression: While both
original training data. Then, a simpler model
(student) is trained on a "distilled" version of the data L1 and L2 can be effective for classification
created using the predictions of the complex model tasks, L2 might be slightly more common
due to its focus on weight decay and
(teacher). This "distilled" data can be generated by
smoother optimization landscape. For
adding noise or applying transformations to the
regression tasks, L1 regularization can be
teacher's predictions, forcing the student model to
learn a more robust representation of the data. advantageous as it can promote sparsity and
potentially lead to more interpretable
models. Consider Knowledge Distillation
These additional techniques offer various approaches to (KD) for classification tasks where a smaller
regularization in CNNs. The choice of technique(s) often model needs to learn from a larger pre-
depends on the specific problem, network architecture, and trained model.
computational resources available. o Object Detection and Segmentation:
Techniques like data augmentation with
random cropping, scaling, and rotation are
particularly valuable. Additionally,

© 2021, JOIREM |www.joirem.com| Page 8

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

SpatialDropout can be effective for object excludes the final pooling layer. (Provide a more
detection/segmentation tasks, as it detailed description if needed).
encourages the network to learn spatially
robust features by deactivating entire All models were initialized with the same seed for parameter
contiguous regions within feature maps. consistency. A detailed breakdown of the architectures is
 Model Complexity and Interpretability: provided in Table 1.
o Complex Models: For very deep or
complex models with a large number of Datasets:
parameters, techniques like Dropout, weight
decay (L2), or Group Lasso can be crucial
to prevent overfitting and improve The experiments utilized two datasets for training:
generalization.
o Interpretability: If interpretability is a 1. CIFAR-10: This standard benchmark dataset
major concern, consider techniques that consists of 60,000 32x32 colored images categorized
promote sparsity like L1 regularization or into ten classes.
Group Lasso. These techniques drive some 2. Fashion-MNIST: This dataset comprises 70,000
weights to zero, making it easier to identify grayscale images (28x28) of various fashion items
the most important features for the model's (clothing and shoes) belonging to ten distinct
predictions. categories.
o
 Computational Resources: We split the original training data into training and validation
o Limited Resources: Early stopping can be sets, using 20% for validation and the remaining 80% for
a good option when computational training. All models were trained with mini-batches of size
resources are limited. It efficiently prevents 128. Models trained on CIFAR-10 ran for 350 epochs, while
overfitting without requiring complex those using Fashion-MNIST were trained for 250 epochs. To
techniques. Consider techniques like ℓ₁ ensure unbiased evaluation of generalization error,
Regularization, which can be hyperparameter tuning and learning process analysis were
computationally less expensive than performed on the validation data, while the test data was
Dropout in some cases. reserved solely for final performance assessment.
o
Results and Analysis
Evaluating the Impact of Optimizers and Regularization
Techniques on CNN Performance This section provides a comparative analysis of various
optimization and regularization techniques based on their
This section explores how different optimizers and impact on generalization performance and the visualization of
regularization techniques influence the training process and model learning curves (loss behavior). Here, "loss" refers to
final performance of convolutional neural networks (CNNs). the function minimized during training (as commonly used in
deep learning frameworks), and "accuracy" refers to the
performance on both training data and unseen data.
Model Architectures and Datasets
Evaluation of Optimizers:
Baseline Models:
We analyzed the influence of different optimizers on the
 Model 1: We employed the CNN-C architecture learning behavior and final performance of CNN models. We
proposed by Springenberg et al. in their work, evaluated nine distinct optimizers (described in Section 2) on
"Striving for simplicity: The all convolutional net" three different model architectures, each trained on both
(arXiv:1412.6806) . (Provide a brief description of its datasets. Hyperparameter settings for each optimizer are
structure here). provided in Appendix B.
 Model 2: Inspired by VGG-16 by Simonyan and
Zisserman (arXiv:1409.1556) , this model consists of
Figures depict the loss and accuracy learning curves for the
stacked convolutional layers followed by pooling and
models.
dense layers before the output.
 Model 3: The largest model (in terms of learnable
parameters) has an AlexNet-like architecture
described by Krizhevsky et al. in their influential
paper, "Imagenet classification with deep
convolutional neural networks" (NIPS'12) . This
architecture utilizes stacked convolutional layers and
pooling layers, with 3x3 receptive fields and

© 2021, JOIREM |www.joirem.com| Page 9

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

 Classical vs. Adaptive: Nesterov (classical)

consistently ranked best for test set accuracy,
followed by Momentum and then SGD. Among
adaptive optimizers, Adagrad and RMSProp ranked
the lowest.
 Training Performance: Most optimizers achieved
near-zero loss and 100% training accuracy by 350
epochs. Exceptions were SGD and RMSProp, with
SGD achieving the lowest training accuracy
(95.43%).
 Overfitting: In the early stages, all optimizers except
SGD on Model 1 showed signs of overfitting (large
gap between training and validation accuracy).
Loss learning curves for all optimizers on baseline models.
Impact of Batch Normalization:

 Improved Generalization: Incorporating Batch

Normalization significantly reduced test set loss and
improved accuracy in four out of six models.
 Reduced Validation Loss: Validation loss curves
dropped significantly compared to the baseline
models.
 Instabilities with Adam: Models trained with Adam
(Model 2 on CIFAR-10 and Models 1 & 2 on
Fashion-MNIST) showed occasional jumps
("spikes") in training and validation loss despite
overall improvement.
 Accelerated Convergence: Batch Normalization
seemed to accelerate convergence in the first model
architecture.
 Reduced Overfitting: Overfitting was reduced in all
cases, suggesting Batch Normalization's regularizing
effect.
Accuracy learning curves for all optimizers on baseline
models.
Future Investigation:
Analysis of Optimization Algorithms and Regularization
The remaining sections of the article will likely explore how
Techniques
different regularization methods and Batch Normalization
affect the generalization performance of CNNs in more detail.
This section analyzes the performance of various optimization It appears the research focuses on one optimizer and model
algorithms and the impact of Batch Normalization on the architecture per dataset:
generalization performance of CNNs.
 CIFAR-10:
Observations on Optimizers: o Model 1 with Nesterov optimizer
o Model 2 with Adam optimizer
 Top Performers: Nesterov, Adam, and AdaMax o Model 3 with Nesterov optimizer
achieved the highest test set accuracy across all six  Fashion-MNIST:
models. o Model 1 with Adam optimizer
 Stability vs. Performance: Nesterov exhibited the o Model 2 with Adam optimizer
most stable performance (validation loss) compared o Model 3 with AdaMax optimizer
to Adam and AdaMax, which showed more
fluctuations. However, Adam and AdaMax still These models will serve as "baseline" architectures for further
achieved good test set accuracy. experimentation with regularization techniques.
 Validation Loss: RMSProp consistently had a
higher validation loss than other optimizers, but
surprisingly maintained reasonable validation and
test set accuracy.

© 2021, JOIREM |www.joirem.com| Page 10

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

often have desirable properties like differentiability,

which aids in optimization.
 Impact on Test Set Accuracy: Tables show the
final accuracy with and without Early Stopping.
While test accuracy improves in some cases, it can
also decrease. However, the training time is
significantly reduced.
 Trade-off between Time and Performance: Early
Stopping offers a trade-off between training time and
final model performance. For example, Model 1 with
The effect of Batch Normalization on the loss of baseline Dropout regularization suffers an accuracy drop from
87.73% to 84.51% with Early Stopping, but training
models trained on CIFAR-10 dataset.
time is more than halved.
 Patience Parameter Tuning: Using a larger
patience value can be beneficial for achieving better
final accuracy.
 Comparison with Data Augmentation: Tables also
highlight that Data Augmentation achieves the best
accuracy compared to models using single
regularizers combined with Early Stopping.

3. CONCLUSIONS

This review article explored the impact of various

The effect of Batch Normalization on the loss of baseline
optimization algorithms and regularization techniques on the
models trained on Fashion-MNIST dataset. generalization performance of convolutional neural networks
(CNNs). The experiments analyzed the behavior of different
doi.org/10.3390/app10217817
optimizers (classical and adaptive) on training and validation
loss, as well as their influence on final test set accuracy. It was
Early Stopping for Improved Training Efficiency found that Nesterov, Adam, and AdaMax achieved the highest
test set accuracy, while Nesterov exhibited the most stable
This section explores the benefits of Early Stopping, a validation performance. Early Stopping was introduced as a
technique to prevent overfitting and reduce training time. technique to prevent overfitting and reduce training time. The
results demonstrated the trade-off between training time and
Observations: final model performance offered by Early Stopping. Finally,
the impact of Batch Normalization was investigated, revealing
 Early Validation Improvement: As seen in figures, its effectiveness in reducing test set loss, improving accuracy,
validation loss for most optimizers (except SGD) and accelerating convergence in some cases.
reaches its minimum value before epoch 50 and
starts increasing afterwards. Similarly, figures show This review emphasizes the importance of careful selection
limited improvement in validation accuracy beyond and tuning of optimization algorithms and regularization
epoch 100 for most optimizers. techniques for achieving optimal CNN performance. The
 Justification for Early Stopping: These provided theoretical background, accompanied by the
observations suggest that training can be stopped experimental analysis of the learning process and model
earlier without sacrificing generalization performance, offers valuable insights into the fields of
performance. Early Stopping helps achieve similar optimization and regularization of deep learning.
performance with reduced training time and Visualizations further corroborate the claims and intuitions
potentially avoids overfitting to the training data. about how these methods affect the learning process and the
 Implementation: We implemented Early Stopping model's final performance on unseen data.
with a "patience" of 30 epochs. Training stops if
there's no improvement in validation accuracy for 30 Key Findings from the Analysis:
consecutive epochs. The model with the best-
observed validation accuracy is then returned.  Optimization Algorithms: Nesterov and Adam
 Validation Accuracy vs. Loss: While both
validation accuracy and loss can be monitored, we
emerged as the top performers in terms of
focused on accuracy because it directly relates to the generalization performance on new data across
model's performance on unseen data. Loss functions various settings. However, the optimal choice
depends on the specific architecture and
© 2021, JOIREM |www.joirem.com| Page 11
 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

dataset, highlighting the need for evaluation 3 Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data
before deployment. augmentation for deep learning. Journal of Big Data, 6(1), 1-48.
 Regularization Techniques: Regularization
4 Srivastava, S., Mittal, S., & Jayanth, J. P. (2022). A survey of deep
significantly enhanced model performance.
learning techniques for underwater image classification. IEEE
Data Augmentation and Dropout were found to Transactions on Neural Networks and Learning Systems.
be particularly effective. Combining these
techniques with Batch Normalization yielded 5 Achilles, A., & Soatto, S. (2018). Information dropout: Learning
the greatest improvement in some cases, but optimal representations through noisy computation. IEEE
caution is advised due to potential Transactions on Pattern Analysis and Machine Intelligence, 40(12),
underperformance with certain configurations. 2897-2905.
 Ensemble Learning and Early Stopping:
6 Pan, H., Niu, X., Li, R., Shen, S., & Dou, Y. (2020). Dropfilterr: a
Ensemble learning offers potential for further
novel regularization method for learning convolutional neural
performance gains, while Early Stopping
networks. Neural Processing Letters, 51(2), 1285-1298.
provides a method to balance training time
with reasonable generalization performance. 7 Li, Y., Wang, N., Shi, J., Hou, X., & Liu, J. (2018). Adaptive batch
normalization for practical domain adaptation. Pattern Recognition,
Limitations and Future Directions: 80, 109-117.

8 Yu, S., Wickstrøm, K., Jenssen, R., & Príncipe, J. C. (2020).

 Regularization Evaluation: This work focused
Understanding convolutional neural networks with information
on evaluating regularization techniques with
theory: An initial exploration. IEEE Transactions on Neural Networks
the best optimizer for each architecture and and Learning Systems, 32(1), 435-442.
dataset. Further exploration is needed to
understand their impact with lower-performing 9 Li, G., Zhang, M., Li, J., Lv, F., & Tong, G. (2021). Efficient densely
optimizers. connected convolutional neural networks. Pattern Recognition, 109,
 Broader Applicability: Most techniques 107610.
discussed are applicable to various problems.
Extending the evaluation to different network 10 Rey-Area, M., Guirado, E., Tabik, S., & Ruiz-Hidalgo, J. (2020).
Fucitnet: Improving the generalization of deep learning networks by
architectures and domains would be beneficial.
the fusion of learned class-inherent transformations. Information
 Optimization Techniques: A deeper
Fusion, 63, 188-195.
examination of optimization techniques,
including learning rate schedules and weight 11 Jiang, Y., Chen, L., Zhang, H., & Xiao, X. (2019). Breast cancer
initialization schemes, is warranted to histopathological image classification using convolutional neural
understand their influence on generalization networks with small se-resnet module. PloS one, 14(3), e0214587.
performance.
12 Zhang, M., Li, W., & Du, Q. (2018). Diverse region-based cnn for
hyperspectral image classification. IEEE Transactions on Image
By incorporating these findings and limitations,
Processing, 27(6), 2623-2634.
researchers and practitioners can make informed
decisions regarding optimization strategies for their [13] Yunhui Guo. 2018. A survey on methods and theories of
CNN architectures. This review provides a quantized neural networks. arXiv preprint arXiv:1808.04752 (2018).
foundation for further exploration within the field [14] Dongyoon Han, Jiwhan Kim, and Junmo Kim. 2017. Deep
of deep learning optimization and regularization. pyramidal residual networks. In Proceedings of the IEEE conference
on computer vision and pattern recognition. 5927–5935.
REFERENCES
1 Marin, I., Kuzmanic Skelin, A., & Grujic, T. (2020). Empirical
Evaluation of the Effect of Optimization and Regularization
Techniques on the Generalization Performance of Deep
Convolutional Neural Network. Applied Sciences, 10(21), 7817.
doi.org/10.3390/app10217817
2 Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-
Shamma, O., ... & Farhan, L. (2021). Review of deep learning:
Concepts, CNN architectures, challenges, applications, future
directions. Journal of Big Data, 8(1), 1-74.

© 2021, JOIREM |www.joirem.com| Page 12

 Journal Publication of International Research for Engineering and Management (JOIREM)
Volume: 05 Issue: 09 | Sept-2021

[15] Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, [28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.
and Tong Zhang. 2020. Model Rubik’s Cube: Twisting Resolution, Imagenet classification with deep convolutional neural networks.
Depth and Width for TinyNets. arXiv preprint arXiv:2010.14819 Advances in neural information processing systems 25 (2012),
(2020). 1097–1105.

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [29] Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee. 2019.
Deep residual learning for image recognition. In Proceedings of the Survey of dropout methods for deep neural networks. arXiv preprint
IEEE conference on computer vision and pattern recognition. 770– arXiv:1904.13310 (2019).
778.
[30] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. 1998. Gradient-based learning applied to document recognition.
Identity mappings in deep residual networks. In European Proc. IEEE 86, 11 (1998), 2278–2324. [31] Yann LeCun and Corinna
conference on computer vision. Springer, 630–645. Cortes. 2010. MNIST handwritten digit database.
http://yann.lecun.com/exdb/mnist/. (2010).
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. http://yann.lecun.com/exdb/mnist/
Identity mappings in deep residual networks. In European
conference on computer vision. Springer, 630–645. [32] Weizhi Li, Gautam Dasarathy, and Visar Berisha. 2020.
Regularization via Structural Label Smoothing. In Proceedings of the
[19] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Twenty Third International Conference on Artificial Intelligence and
and Mu Li. 2019. Bag of tricks for image classification with Statistics (Proceedings of Machine Learning Research), Silvia
convolutional neural networks. In Proceedings of the IEEE/CVF Chiappa and Roberto Calandra (Eds.), Vol. 108. PMLR, 1453–1463.
Conference on Computer Vision and Pattern Recognition. 558–567. https://proceedings.mlr.press/v108/li20e.html
[20] Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In
defense of the triplet loss for person reidentification. arXiv preprint [33] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and
arXiv:1703.07737 (2017). Sungwoong Kim. 2019. Fast autoaugment. In Advances in Neural
Information Processing Systems. 6665–6675.
[21] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling
the knowledge in a neural network. arXiv preprint arXiv:1503.02531 [34] Ziqing Lu, Chang Xu, Bo Du, Takashi Ishida, Lefei Zhang, and
(2015). Masashi Sugiyama. 2021. LocalDrop: A Hybrid Regularization for
Deep Neural Networks. IEEE Transactions on Pattern Analysis and
[22] Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Machine Intelligence (2021).
2019. Population based augmentation: Efficient learning of
augmentation policy schedules. In International Conference on [35] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley.
Machine Learning. PMLR, 2731–2741. 2020. Neural architecture search without training. arXiv preprint
arXiv:2006.04647 (2020).
[23] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten
Hoefler, and Daniel Soudry. 2019. Augment your batch: better BIOGRAPHIES
training with larger batches. arXiv preprint arXiv:1901.09335 (2019).
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry
Sultan Khaibar Safi holds a Bachelor's
Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and
degree in Information Technology and a
Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural Master's degree in Artificial Intelligence
networks for mobile vision applications. arXiv preprint and Robotics Engineering. His research
arXiv:1704.04861 (2017). interests lie in the field of deep learning,
Computer Vision, Machine Learning
[25] Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation particularly focusing on optimizing and
regularizing Convolutional Neural
networks. In Proceedings of the IEEE conference on computer vision
Networks (CNNs) to enhance their
and pattern recognition. 7132–7141. generalization performance.

[26] Sergey Ioffe and Christian Szegedy. 2015. Batch normalization:

Accelerating deep network training by reducing internal covariate
shift. arXiv preprint arXiv:1502.03167 (2015).

[27] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Cifar-10
and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar.
html 6 (2009), 1.

© 2021, JOIREM |www.joirem.com| Page 13