Machine Learning

Prof. Sergei Gleyzer Lecture

PH451, PH551
March 9, 2023
 Final Projects
• Semester long activity (40% of grade)
• Team activity
• Self-designed project in consultation with
instructor
• Has to include an ML component
• Can pick any topic you feel passionate
about
• Graduate students – can pick something
from your specific field or research area

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 2

 Final Projects
• Pre-proposal 5% due 03/21
• Proposal 10%
• Project Outline and Demo 25%
• Presentation 10%
• Peer Evaluation 10%
• Final Project Submission (write-up,
code) 40%

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 3

 Final Project Teams
• New team assignments
• New teams are only for final projects
• continue with existing teams for all
remaining exercises and activities
• Please start to discuss project ideas
• Pre-Proposals due on 03/21

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 4

 Pre-Proposals
• Should establish:
• Your team
• 3 possible ideas for your project in your
order of preference
• Pick potential problems that
• You have an idea how to solve
• Relevant to someone
• Not yet solved (or provide a different
solution)

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 5

Google Summer of Code 2024
https://summerofcode.withgoogle.com/

• Machine Learning for Science

(ML4SCI): ml4sci.org

• HumanAI: humanai.foundation

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 6

 Convolutional Networks

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 7

introduction
Convolutional Networks

Neural Network (NN) Deep NN Convolutional NN

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 8

 Convolutional Networks
[x1 x2 x3…xi]T
waveform heights

[x11 x12…x1n x21 x22…]T

pixel intensities

Feature learning

Sergei Gleyzer
03/19/2024 9
PH451/PH551 Lecture
 Convolution Example

Exploit structure, neighboring pixel dependence

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 10

 Convolutional Nets
• Emerged from computer vision
• Inspired by visual cortex of the brain
• Simple cells that respond to environment (edges)
• Complex cells with more response invariance

• Neocognitron (1980)
• Synthesis, pooling over
inputs
• Early model that inspired CNNs

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 11

 Convolutional Nets
Convolutional Neural Networks:
• Began with image and sequence-based
problems in computer vision
• Images (2D)
• CNN’s learn features with simple structures
• Filters: repeatedly applied
• Unsupervised learning during first stage

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 12

 Convolutional NN

Feedforward structure, spatially arranged units:

• 2-D feature maps – result of a convolution performed on the
previous layer

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 13

 Convolutional NN

Input (Image) à Convolution à Activation (Non-Linear)

à Spatial Pooling à Feature Maps

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 14

 Convolution Example

Same convolutional filter (set of weights) is applied to each element

• An element in a single 2-D location can only receive input from
elements in similar location from previous layers (locality)
• Same weights for each feature map (and different across maps)
• Exploit structure, neighboring pixel dependence
03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 15
 Filters
Convolutional Neural Networks:
• Began with image and sequence-based problems in
computer vision
• Images (2D)
• CNN’s learn features with simple structures
• Filters: repeatedly applied
• Unsupervised learning during first stage
• Jet images and evolution

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 16

 Pooling
Down-sampling: shrink the size of the feature
map
• Lower resolution that still contains important
information

• Usually added after convolution and non-linearity

(i.e. activation like ReLU) have been applied
• take the average (average-pooling) or maximum
activity (max pooling) to represent the whole area
• Filter size is smaller than original feature map
• Helps model invariance to small local translations
03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 17
 Training
CNNs compute the stacked sequence
of layers
• usually ending with a Fully-Connected Layer
• The FCN is the same as a regular neural
networks

• Train with back-propagation

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 18

Some Well-known Architectures
• LeNet5 (1990s)
• Early CNN used to read digits

• Y. LeCun et al., 1998

• Average pooling, sigmoid, trained on MNIST

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 19

 AlexNet
AlexNet (2012)
• Similar to LeNet but bigger and deeper model
(8 layers, 60M params)

• ReLU activations, max pooling, dropout and

data augmentation trained on GPUs on
ImageNet
• Krizhevsky et al., 2012 (Imagenet 2012 winner)
03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 20
 Data Augmentation
• Useful technique for increasing
training dataset size
• Apply rotations, shifts and re-sizing to
make as many realistic training images
as possible
• Helps in training and to reduce overfitting

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 21

 GoogleNet
• Szegedy et al., 2014
• Much deeper than previous CNNs
• Inception modules

• Multiple kernels stacked at same level

• Concatenated along the depth dimension
• Serve to capture information along the depth dimension
across scales, bottlenecks to reduce dimensionality and
behave like multi-dimensional layers

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 22

 VGGNet
• Simonyan and Zisserman, 2014
• Stacked smaller kernel-sized filters (3x3)

• 16 layers: 2/3 convolutional, 1 max pool and

repeat

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 23

 ResNet
• He et al., 2015 (Imagenet 2015 winner)
• Residual network with skip connections

• 152 layers similar to VGG with skip connections (gated

units) and batch normalization
• Residual learning: h(x) – x
• Helps propagate your signal across the whole network

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 24

 Outline
• Sequential Data
• Recurrent Neural Networks

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 25

 Sequential Data
Text Image

Time Series

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 26

 Sequential Data
• Properties:
• Elements occur in a particular order
• May depend on other elements

• Examples:
• Sentences
• Images
• Radio Waves
• Temperature

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 27

 Some Applications
• Input:
• Fixed size
• Output
• Sequence

• Example: image captioning

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 28

 Some Applications
• Input:
• Sequence
• Output
• Fixed Size

• Example: Sentiment Analysis

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 29

 Some Applications
• Input:
• Sequence
• Output
• Sequence

• Example: Google Translate

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 30

 Recurrent Neural Networks
• Extensions of deep neural networks to
directed graphs and sequences
• Rumelhalt, Hinton, Williams (1986)
• Dynamic behavior in the time domain
• Introduce ideas of memory, feedback loops
to accommodate sequential data
• Key idea: capture information from the past
in a hidden state

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 31

 RNN vs MLP
MLP RNN neuron (unrolled)

RNN layer (unrolled)

No loops

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 32

 Basic RNN introduction

Recurrent neural network

Advantages:
• Weights are shared
across layers
• Uses previous
hidden state
• Weights of each
layer are not learned • A form of “memory” 6

independently
• Train with backpropagation
(through time)

03/19/2024 Sergei Gleyzer PH451/PH551 Lecture 33