21CS743 | DEEP LEARNING | SEARCH CREATORS.

Module-04

Convolutional Networks

Definition of Convolution

• Convolution: A mathematical operation that combines two functions (input signal/image

and filter/kernel) to produce a third function.

• Purpose: Captures important patterns and structures in the input data, crucial for tasks like
image recognition.

2. Mathematical Formulation

Search Creators... Page 1

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

3. Parameters of Convolution

a. Stride

• Definition: The number of pixels the filter moves over the input.

• Types:

o Stride of 1: Filter moves one pixel at a time, resulting in a detailed output.

o Stride of 2: Filter moves two pixels at a time, reducing output size (downsampling).

b.Padding

• Definition: Adding extra pixels around the input image.

• Types:

o Valid Padding: No padding applied; results in a smaller output feature map.

o Same Padding: Padding applied to maintain the same output dimensions as the
input.

4. Significance in Neural Networks

• Application: Used in convolutional layers of CNNs to extract features from images.

• Learning Hierarchical Representations: Stacked convolutional layers enable learning of

complex patterns, essential for image classification and other tasks.

Search Creators... Page 2

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

Purpose of Pooling

• Spatial Size Reduction: Decreases the dimensions of the feature maps.

• Parameter and Computation Reduction: Reduces the number of parameters and

computations in the network.

• Overfitting Control: Helps to control overfitting by providing a form of translational

invariance.

2. Types of Pooling

a. Max Pooling

• Definition: Selects the maximum value from each patch (sub-region) of the feature map.

• Purpose: Captures the most prominent features while reducing spatial dimensions.

b. Average Pooling

• Definition: Takes the average value from each patch of the feature map.

• Purpose: Provides a smooth representation of features, reducing sensitivity to noise.

Search Creators... Page 3

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

3. Operation of Pooling

4. Significance in Neural Networks

• Feature Extraction: Reduces the size of the feature maps while retaining the most relevant
features.

• Efficiency: Decreases computational load, allowing deeper networks to train faster.

• Robustness: Provides a degree of invariance to small translations in the input, making the
model more robust.

Search Creators... Page 4

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

1. Convolution as an Infinitely Strong Prior

• Focus on Local Patterns: Emphasizes the importance of local patterns in the data (e.g.,
edges and textures) over global patterns.

• Effectiveness in CNNs: This locality assumption enhances the effectiveness of

Convolutional Neural Networks (CNNs) for image and video analysis.

2. Pooling as an Infinitely Strong Prior

• Enhances Translational Invariance: Allows the network to recognize objects regardless

of their position within the image.

• Reduces Sensitivity to Position: By downsampling, pooling reduces sensitivity to the

exact location of features, improving generalization.

3. Significance in Neural Networks

• Feature Learning: Both operations prioritize local features, enabling efficient learning of
essential characteristics from input data.

• Improved Generalization: The combination of convolution and pooling enhances the

model's ability to generalize across various input variations.

Search Creators... Page 5

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

Variants of the Basic Convolution Function

1. Dilated Convolutions

• Definition: Introduces spacing (dilation) between kernel elements.

• Wider Context: Allows the model to incorporate a wider context of the input data without
significantly increasing the number of parameters.

• Applications: Useful in tasks where understanding broader spatial relationships is

important, such as in semantic segmentation.

2. Depthwise Separable Convolutions

• Two-Stage Process:

o Depthwise Convolution: Applies a separate convolution for each input channel,

reducing computational complexity.

o Pointwise Convolution: Uses 1x1 convolutions to combine the outputs from the
depthwise convolution.

• Parameter Efficiency: Reduces the number of parameters and computations compared to

standard convolutions while maintaining performance.

• Applications: Commonly used in lightweight models, such as MobileNets, for mobile and
edge devices.

Search Creators... Page 6

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

1. Definition of Structured Outputs

• Structured Outputs: Refers to tasks where the output has a specific structure or spatial
arrangement, such as pixel-wise predictions in image segmentation or keypoint localization
in object detection.

2. Importance in Semantic Segmentation

• Maintaining Spatial Structure: For tasks like semantic segmentation, it’s crucial to
maintain the spatial relationships between pixels in predictions to ensure that the output
accurately represents the original input image.

3. Specialized Networks

• Network Design: Specialized neural network architectures, such as Fully Convolutional

Networks (FCNs), are designed to handle structured outputs by replacing fully connected
layers with convolutional layers, allowing for spatially consistent predictions.

• Skip Connections: Techniques like skip connections (used in U-Net and ResNet) help
preserve high-resolution features from earlier layers, improving the accuracy of the output.

4. Adjusted Loss Functions

• Loss Function Modification: Loss functions may be adjusted to enforce structural

consistency in the predictions. Common approaches include:

o Pixel-wise Loss: Evaluating the loss on a per-pixel basis (e.g., Cross-Entropy Loss
for segmentation).

o Structural Loss: Incorporating penalties for structural deviations, such as Dice

Loss or Intersection over Union (IoU) metrics, which consider the overlap between
predicted and true regions.

Search Creators... Page 7

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

5. Applications

• Use Cases: Structured output networks are widely used in various applications, including:

o Semantic Segmentation: Assigning class labels to each pixel in an image.

o Instance Segmentation: Identifying and segmenting individual object instances

within an image.

o Object Detection: Predicting bounding boxes and class labels for objects in an
image while maintaining spatial relations.

Data Types

1. 2D Images

• Standard Input: The most common input type for CNNs, typically used in image
classification, object detection, and segmentation tasks.

• Format: Represented as height × width × channels (e.g., RGB images have three channels).

Search Creators... Page 8

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

2. 3D Data

• Definition: Includes video processing and volumetric data, such as those found in medical
imaging (e.g., MRI or CT scans).

• Format: Represented as depth × height × width × channels, allowing the network to

capture spatial and temporal information.

• Applications: Useful in tasks like action recognition in videos or analyzing 3D medical

images for diagnosis.

3. 1D Data

• Definition: Consists of sequential data, such as time-series data or audio signals.

• Format: Represented as sequences of data points, often one-dimensional.

• Applications: Used in tasks like speech recognition, audio classification, and analyzing
sensor data from IoT devices.

Efficient Convolution Algorithms

1. Fast Fourier Transform (FFT)

• Definition: A mathematical algorithm that computes the discrete Fourier transform (DFT)
and its inverse, converting signals between time (or spatial) domain and frequency domain.

• Convolution in Frequency Domain:

o Convolution in the time or spatial domain can be transformed into multiplication in

the frequency domain, which is often more computationally efficient for large
kernels.

Search Creators... Page 9

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

• Applications: Commonly used in applications requiring large kernel convolutions, such as

in image processing and signal analysis.

2. Winograd's Algorithms

• Definition: A set of algorithms designed to optimize convolution operations by reducing

the number of multiplications needed.

• Efficiency Improvement:

o Winograd's algorithms work by rearranging the computation of convolution to

minimize redundant calculations.

o They can reduce the complexity of convolution operations, particularly for small
kernels, making them more efficient in terms of computational resources.

• Key Concepts:

o The algorithms break down the convolution operation into smaller components,
allowing for fewer multiplicative operations and leveraging addition and
subtraction instead.

o They are particularly effective in scenarios where computational efficiency is

critical, such as mobile devices or real-time applications.

• Applications: Frequently used in lightweight models and resource-constrained

environments where computational power and memory usage are limited.

Search Creators... Page 10

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

1. Random Feature Maps

• Definition: A technique that uses random projections to map input data into a higher-
dimensional space, facilitating the extraction of features without the need for labels.

• Purpose: Helps to approximate kernel methods, enabling linear models to learn complex
functions.

• Advantages:

o Efficiency: Reduces the computational burden of traditional kernel methods while

retaining useful information.

o Scalability: Suitable for large datasets as it allows for faster training times.

• Applications: Commonly used in tasks where labeled data is scarce, such as clustering and
anomaly detection.

2. Autoencoders

• Definition: A type of neural network designed to learn efficient representations of data

through unsupervised learning by encoding the input into a lower-dimensional space and
then reconstructing it back.

• Structure:

o Encoder: Compresses the input data into a latent representation.

o Decoder: Reconstructs the original input from the latent representation.

• Purpose: Learns to capture important features and structures in the data without
supervision, making it effective for dimensionality reduction and feature extraction.

• Advantages:

o Robustness: Can learn from noisy data and still produce meaningful
representations.

Search Creators... Page 11

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

o Flexibility: Can be adapted for various tasks, including denoising, anomaly

detection, and generative modeling.

• Applications: Used in scenarios such as image compression, data denoising, and

generating new data samples.

3. Facilitation of Unsupervised Learning

• Role in Unsupervised Learning: Both methods enable the extraction of meaningful

features from unlabelled data, facilitating learning in scenarios where obtaining labeled
data is challenging or expensive.

• Enhancing Model Performance: By leveraging these techniques, models can improve

their performance on downstream tasks, such as clustering, classification, or regression,
even in the absence of labels.

Search Creators... Page 12

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

Notable Architectures

1. LeNet-5

• Introduction:

o Developed by Yann LeCun and colleagues in 1998.

o One of the first convolutional networks designed specifically for image recognition
tasks.

• Architecture Details:

o Input Layer: Takes in grayscale images of size 32x32 pixels.

o Convolutional Layer 1:

▪ 6 filters (5x5) with a stride of 1.

▪ Output size: 28x28x6.

o Activation Function: Sigmoid or hyperbolic tangent (tanh).

Search Creators... Page 13

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

o Pooling Layer 1:

▪ Average pooling (subsampling) with a 2x2 filter and a stride of 2.

▪ Output size: 14x14x6.

o Convolutional Layer 2:

▪ 16 filters (5x5).

▪ Output size: 10x10x16.

o Pooling Layer 2:

▪ Average pooling (2x2).

▪ Output size: 5x5x16.

o Fully Connected Layers:

▪ 120 neurons in the first layer.

▪ 84 neurons in the second layer.

▪ Output layer with 10 neurons (for digit classes 0-9).

• Significance:

o Introduced the concept of using convolutional layers for feature extraction followed
by pooling layers for dimensionality reduction.

o Paved the way for modern CNNs, influencing later architectures.

2. AlexNet

• Introduction:

o Developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton in 2012.

o Marked a breakthrough in deep learning by achieving top performance in the

ImageNet competition.

Search Creators... Page 14

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

• Architecture Details:

o Input Layer: Accepts images of size 224x224 pixels (RGB).

o Convolutional Layer 1:

▪ 96 filters (11x11) with a stride of 4.

▪ Output size: 55x55x96.

o Activation Function: ReLU, introduced to improve training speed.

o Pooling Layer 1:

▪ Max pooling (3x3) with a stride of 2.

▪ Output size: 27x27x96.

o Convolutional Layer 2:

▪ 256 filters (5x5).

▪ Output size: 27x27x256.

o Pooling Layer 2:

▪ Max pooling (3x3).

▪ Output size: 13x13x256.

o Convolutional Layer 3:

▪ 384 filters (3x3).

▪ Output size: 13x13x384.

o Convolutional Layer 4:

▪ 384 filters (3x3).

▪ Output size: 13x13x384.

Search Creators... Page 15

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

o Convolutional Layer 5:

▪ 256 filters (3x3).

▪ Output size: 13x13x256.

o Pooling Layer 3:

▪ Max pooling (3x3).

▪ Output size: 6x6x256.

o Fully Connected Layers:

▪ First layer with 4096 neurons.

▪ Second layer with 4096 neurons.

▪ Output layer with 1000 neurons (for 1000 classes).

• Innovative Techniques Introduced:

o ReLU Activation:

▪ Enabled faster convergence during training compared to traditional

activation functions like sigmoid or tanh.

o Dropout:

▪ Regularization method that randomly drops neurons during training to

prevent overfitting, significantly improving generalization.

o Data Augmentation:

▪ Used techniques like image rotation, translation, and flipping to artificially

expand the training dataset and improve robustness.

Search Creators... Page 16

 21CS743 | DEEP LEARNING | SEARCH CREATORS.

o GPU Utilization:

▪ Leveraged parallel processing power of GPUs, enabling training on large

datasets in a reasonable timeframe.

• Significance:

o Established deep learning as a powerful approach for image classification and

sparked widespread research and development in CNN architectures.

o Highlighted the importance of large labeled datasets and robust training techniques
in achieving state-of-the-art performance.

Search Creators... Page 17