Neural Networks

and
Deep Learning
Ahmad Kalhor
Associate Professor .
School of Electrical and Computer Engineering –
University of Tehran
Fall 2022
Contents

Ahmad Kalhor - University of Tehran 2

1. Introduction
• Natural Neural Networks
• Artificial NNs and Applications
• Architectures, Activation Functions and Learning in ANNs
• Mcculloch & Pitz Neuron
2. Fully Connected Neural Networks
• Linear Perceptron, AdaLine and MadaLine
• Multi-Layer Perceptron (MLP)
• Auto-encoders and Restricted Boltzmann Machine
• Deep belief networks
3. Convolutional Neural Networks (CNNs)
• Convolutional Neural Networks (CNNs)
• Developments and Techniques in CNNs
• Some popular Architectures (AlexNet, VGG, ResNet, Inception, MobileNet, DenseNet, and EfficientNet)
4. Region based CNNs
• CNNs for Object Detection (RCNN, Mask RCNN, YOLO,...)
• CNNs for Object Segmentation (UNet, Mask RCNN, ...)
Ahmad Kalhor - University of Tehran 3
5. Recurrent Neural Networks
• Recurrent Neural Network(RNN)
• Long Short Term Memory (LSTM)
• Gated Recurrent Units(GRU)
• Some Extensions
6. Transformers
• Attention Mechanism
• Transformers
• Bidirectional Encoder Representations from Transformers (BERT)
• Vision Based Transformers
7. Variational Auto encoders and Generative Adversarial Networks
• Variational Auto-encoders
• Vector Quantization VAEs.
• Generative Adversarial Networks
• Some Architectures (DCGAN, CGAN, ACGAN,SRGAN , InfoGAN...)
Ahmad Kalhor - University of Tehran 4
 Mini Projects and Exams
Chapters 2 3 4 5 6 7
Mini M. Pr. 1 M. Pr. 2 M. Pr. 3 M. Pr. 4 M. Pr. 5 M. Pr. 5
Projects 10% 10% 10% 10% 10% 10%
60%

Exams Midterm(Chapters2, 3, and 4) Final(Chapters5, 6, and 7 )

40% 20% 20%
Total Score 100%

* A few (optional) mini-projects are designed for extra work (bonus points)

Ahmad Kalhor- University of Tehran 5

 Chapter 1

Introduction

Ahmad Kalhor - University of Tehran 6

 1. Introduction
What is Artificial intelligence?
• AI is “a learning technology”
• Inspiring from the natural intelligence (in human and animals)
• Environment perception (clustering/classification/regression)
• Providing Memory about it
• Interaction with it (Doing complexity tasks)
• Computation intelligence (math-logic)
• Stablished on branches of mathematic:
• Matrices Algebra in representation and in computation
• Analysis in learning
• Statistical and Geometric in interpretation and design

• Widely applicable in our learning activities

• Media, Security and Entertainment
• Cyber physical , hybrid and autonomous systems
• Social Robots and human-robot interaction
• Industrial systems
• Health systems
• Cognitive systems
Ahmad Kalhor- University of Tehran 7
• Finance, and Energy Systems
Artificial NN (ANN) and Deep Learning (DL) in AI

Ahmad Kalhor - University of Tehran 8

A simple definition for Neural Networks

• Neural Networks : A set of simple units (neurons) which are wired together in
layers, in order to make important (desired) outputs against stimulating inputs.

Inputs
Outputs

Ahmad Kalhor - University of Tehran 9

Natural Neural Networks

• NNs have been created naturally in body of animals and plants.

Important questions about animals intelligence
• How do they recognize (cluster and classify) foods, enemies and environment?
• How do they make memories about different events?
• How do they learn to define different mechanisms and acts in their environment?
……..
• How do they become intelligent?

Ahmad Kalhor - University of Tehran 10

(Trying to answer to aforementioned questions)

Biologists and Scientists (about 200-300 years ago) discovered

the brain, nervous system and spinal cord.

(currently) Neuroscientists research about the structure and function of the

brain as the most important part of intelligent.

Ahmad Kalhor - University of Tehran 11

A Biological Neuron (nerve cell)

(1) Dendrites
to receive the weighted electrochemical signals from adjacent neurons
(2) Cell body (soma)
to make a summation on received signals
(3) Nucleus
to generate an impulsive signal by comparing the absorbed signal with a threshold
(4) Axon
to send the generated signal to other adjacent neurons
(5) Synaptic gaps
to assign a weight to each send signal to adjacent neurons

Ahmad Kalhor- University of Tehran 12

Communication between neurons by electrochemical signals:

Ahmad Kalhor- University of Tehran 13

A simple mathematical Model for Biological Neuron

𝑤1
𝑥1 𝑧 = 𝑤1 𝑥1 +𝑤2 𝑥2 +𝑤3 𝑥3

𝑤2
𝑥2 1 𝑧>𝜃
෍ 𝑦=ቊ
𝜃 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑤3 Soma Nucleus
𝑥3

Dendrites Synaptic gaps 𝑐𝑒𝑙𝑙 𝐴𝑥𝑜𝑛

Ahmad Kalhor- University of Tehran 14

 Types of neurons in human nervous system

1. Sensory neurons
Get information about what's going on inside and outside of the body and bring that information into
the central nervous system (CNS) so it can be processed.

2. Interneurons
which are found only in the CNS, connect one neuron to another. Most of interneurons are in the brain.
There are about 100 billiard neurons in the brain.
There are about 10^15 connections among neurons(10000 connections for each neuron on average)

3. Motor neurons
get information from other neurons and convey commands to your muscles, organs and glands.

Ahmad Kalhor- University of Tehran 15

A simple Input-Output Model for human nervous system

Sensory neurons Interneurons Motor neurons

(Input Layer) (Hidden Layers) (Output Layer)

Ahmad Kalhor- University of Tehran 16

 Learning capabilities of NNs in human body

(1) Classification
Localization, Detection and Classification of different objects, faces, voices, smells, and
approximation and prediction different physical variables: distances, temperatures, smoothness,
brightness, and so on…..
(2) Memory
Capability to create memories about different events with long and short dependencies.
Capability to associate sequenced different patterns together.
(3) Complex and difficult tasks/actions
Car driving and parking, Swimming, Playing music,…..
(4) Computational intelligence
Logic, mathematics, Inference

Ahmad Kalhor- University of Tehran 17

 Learning in natural neural network
(some important facts)

1. The learning process in brain is mainly performed by tuning the synaptic gaps. Information
gathered or resulted from environment is coded in synaptic gaps.

2. The communication speed (electrochemical signal transition) for each neuron is low but
Since the communications among neurons are performed in parallel, the processing speed
of the brain is high on average.

3. The learning process in the brain is not disturbed if some parts of the brain are damaged
and hence the robustness of the natural NNs is high (Fault tolerant).

4. Due to high level abstraction and inference in the brain, its generalization in learning is high.

Ahmad Kalhor- University of Tehran 18

Artificial Neural Networks

Inspiring From natural NNs:

From begging of the 20- century, Scientists and engineers have been
interested to design artificial neural networks:

“To make solutions for demanded (challenging) learning problems”

Ahmad Kalhor- University of Tehran 19

 Applications of ANNs

• Classification NNs • Memory NNs

• Classification of objects, images, faces, • Word/Voice/Video Prediction
texts, speeches, signatures, videos and …. • Natural Language Translation
• image captioning/Descriptions
• Disease Diagnosis • Sentiment analysis
• Fault Detection (Industrial) • {Demand, Transaction and Price} Prediction
• Fraud/ Anomaly Detection • Path Planning, Tele Representation in Social robots

• Regression NNs • Mechanism-Based NNs

• Function Approximation(Static/ODE/PDE) • Pattern Generation/sorting/clustering/Noise
cancellation
• Identification • Super-resolution techniques
• Segmentation • Recommender systems
• Simulation • Image Enhancement
• Data Compressing
• Prediction
• Imitation-based systems
• Signal Recovering/ Repairing • Multi-Task Learning

Ahmad Kalhor- University of Tehran 20

 Artificial Neural Networks (Here)

Fully Connected Convolutional Recurrent

Neural Networks Transformers VAEs and GANs
Neural Networks Neural Networks

Classification/Reg. Generative App.

Memory App.
Classification/Reg. Image/Video/ Memory App. Seq. Data/Image
Sequential Data
Distinct Inputs Time Series Seq. Data/Image /Distinct Inputs
Filtering /Scaling
Partitioning or Filtering/Scaling/ Filtering /Scaling Generation/Discri
(through time)
Mapping Partitioning or Partit. or Map. mination/Reconstr
Partit. or Map.
Mapping uction

Sequential Data: Text, Time series, Speech, Video,....

Ahmad Kalhor- University of Tehran 21

 Feed-forward and recurrent neural networks

1- Feedforward Neural Networks

In feedforward networks, messages(data flow) are passed forward only
Feedforward neural network is an artificial neural network wherein connections
between the nodes do not form a cycle.
Each output is a static of function of inputs.

Ahmad Kalhor- University of Tehran 22

2- Recurrent Neural Networks
In recurrent networks, at least in one layer , messages(data flow) are returned to the same or
former layers.
In recurrent neural networks (RNNs), a notion of time is introduced. The input at time step t
depends on an output from time step t − 1.
These networks are suitable to represent dynamic behavior of functions and systems as nonlinear
difference or differential equations .

Ahmad Kalhor- University of Tehran 23

Activation Functions
• Each node(neuron) has an activation function by which it responses to the stimulating inputs.

𝑥 𝑓(∙) 𝑓(𝑥)

Ahmad Kalhor- University of Tehran 24

 Supervised and unsupervised learning
1. Supervised Learning
Supervised learning as the name indicates a presence of supervisor as teacher. Basically supervised
learning is a learning in which we teach or train the ANNs using data which is well labeled that means
some data is already tagged with correct answer.
Applications: Classification/Regression: Function Approximation and Prediction/Recognition

Learning methods
• Error back propagation(Steepest Descends) MSE, MAE, Cross Entropy,…. : Batch based- stochastic point
based-stochastic mini batch based
Gradient (Local search): Descent optimization algorithms:
SGD/SGD+Momentum/Nesterov accelerated gradient/Adagrad/RMSprop/Adam/AdaMax/Nadam/AMSGrad
• Evolutionary based (Global search) : Genetic Algorithm, Particle Swarm Optimization, Ant colony
• Intelligence-based (Global search): Simplex - Simulated Annealing
Ahmad 2- University of Tehran 25
2. Unsupervised Learning
•Unsupervised learning is the training of ANNs using information that is neither classified nor labeled
and allowing the algorithm to act on that information without guidance.
•Instead of explicit targets for the network, there are some statistical or geometric properties for the
suitable output of the network.
•Some examples of unsupervised learning algorithms include K-Means Clustering, Principal
Component Analysis and Hierarchical Clustering.
•Causal Relationship in Regression Problems.
•Applications: Pattern Generation/ Pattern clustering/ Pattern sorting/ Optimization
problems/Control tasks

Ahmad Kalhor- University of Tehran 26

 3. Semi Supervised and Self Supervised Learning
• Semi-supervised learning is an approach to machine learning that
combines a small amount of labeled data with a large amount of
unlabeled data during training. Semi-supervised learning falls between
unsupervised learning (with no labeled training data) and supervised
learning (with only labeled training data).

• Self-supervised learning is a representation learning method where a

supervised task is created out of the unlabeled data. Self-supervised
learning is used to reduce the data labelling cost and leverage the
unlabeled data pool. Some of the popular self-supervised tasks are based
on contrastive learning.

What is the difference between unsupervised and self-supervised?

The only difference is that, unlike unsupervised learning, self-supervised learning does
not perform the grouping and clustering of data, as is the case with unsupervised
learning. This learning type allows machines to examine part of a data example to figure
out the remaining part

Ahmad Kalhor- University of Tehran 27

Mcculloch & Pitz Neuron
Warren MuCulloch (neuroscientist) and Walter Pitts (logician) 1943

y_in = 2𝑥1 + 2𝑥2 − 𝑥3

𝜃: 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑

Ahmad Kalhor- University of Tehran 28

Some Logic Functions by M&P neurons
𝜽=𝟐

AND

OR

AND Not

Ahmad Kalhor- University of Tehran 29

Two Applications

• XOR 𝜽=𝟐

Ahmad Kalhor- University of Tehran 30

Two Applications

• Hot and Cold

Ahmad Kalhor- University of Tehran 31

End of Chapter 1

Thank you

Ahmad Kalhor- University of Tehran 32