ECBM E4040

Neural Networks and Deep

Learning
Micah Goldblum
Acknowledgement for slides - Abhinav Shrivastava (UMD)
People

TBD TBD
Micah Goldblum (TA) (TA)
(Instructor)
Contents

• Machine Learning Basics • RNNs

• Neural Network Basics • Transformers
• ConvNets • Applications of RNNs & Transformers
• Architectures • Advanced Topics
• Applications of ConvNets • Image Generative Models
• Deep Reinforcement Learning
Graded Assignments
• Coding assignments (40% of grade)
• Quizzes (30% of grade) - pop quizzes held at the end of lecture, bring
your own sheet of paper and writing utensil
• Exams (30% of grade) - midterm + final
Coding Assignments (tentative)

1. Background/fundamentals, Simple Neural Networks (images, text)

2. ConvNet + Vision Application (segmentation)
3. Transformer + Language Application (e.g., miniGPT)
4. Generative Models (e.g., VAE/GAN)
Coding Assignments (tentative)

1. Background/fundamentals, Simple Neural Networks (images, text)

2. ConvNet + Vision Application (segmentation)
3. Transformer + Language Application (e.g., miniGPT)
4. Generative Models (e.g., VAE/GAN)
Prerequisites
• Must be comfortable with:
• Probability
• Statistics
• Mathematical functions (logarithms, differentiation, etc.)
• Linear algebra (vectors, matrices)
• We will do lots of coding in Python!
Resources

• Designed to be self-contained
• No textbook
• Lots of extra resources in the syllabus
Resources
Why this course?
Human-Centered Artificial Intelligence – Stanford University
AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

AI Index Report 2018

Is it useful?
Successes of deep learning
 Vision: 15-20 years ago
Reading Digits
Face
Mail Sorting Detection

Image credits: MNIST dataset, CMU PIE dataset, USPS, SONY

 Vision: today
Task: Image Classification

Figures: Fergus, He
 Vision: today
Task: Object Recognition in Internet Images

Figure: He
 Vision: today
Task: Object Recognition in Medical Imaging

Figure: Matterport github

 Vision: today
Task: 3D from LiDAR

Figure: Esri, Matterport

 Vision: today

AI beats human pathologists at

detecting cancer
Technology behind Snapchat lenses

Facebook accessibility tools for the

visually impaired NeRF Explosion 2020
Slide credit: Lazebnik
 Vision: today

Beijing bets on facial recognition in a big drive for total surveillance – Washington Post, 1/8/2018
Slide credit: Lazebnik
 Vision
• Deep video portraits

• “A quiet wager has taken hold among researchers who study artificial intelligence techniques and the
societal impacts of such technologies. They’re betting whether or not someone will create a so-called
Deepfake video about a political candidate that receives more than 2 million views before getting
debunked by the end of 2018” – IEEE Spectrum, 6/22/2018

Slide credit: Lazebnik

 Vision: today
Task: Image Captioning

Figure from “DenseCap: Fully Convolutional Localization Networks for Dense Captioning”, by Johnson et al.
 Speech and Language: a few decades ago
Task: Machine Translation
1954: Georgetown-IBM experiment
• Completely automatic translation of more than sixty Russian sentences
into English
• Only six grammar rules, 250 vocabulary words, restricted to organic chemistry
• Promised that machine translation would be solved in three to five years
(press release)

Early approaches were extremely brittle

• “The spirit is willing but the flesh is weak.” →
“The vodka is strong but the meat is rotten.”

1966: Automatic Language Processing Advisory Committee (ALPAC) report: MT has failed

Slide credit: Lazebnik

 Speech and Language: today

Google Translate App

• Translate between 103 languages by typing
• Offline: Translate 52 languages when you have no Internet
• Instant camera translation: Use your camera to translate
text instantly in 30 languages
• Camera Mode: Take pictures of text for higher-quality
translations in 37 languages
• Conversation Mode: Two-way instant speech translation in
32 languages
https://www.skype.com/en/features/skype-translator/
https://play.google.com/store/apps/details?id=com.google.a
ndroid.apps.translate&hl=en

See also: The Great AI Awakening (New York Times Magazine)

Slide credit: Lazebnik
 Speech and Language: today
Task: Speech-to-text

Slide credit: Grisel, Ollion, Lelarge

 Text: today

https://chat.openai.com/chat
 Games: a few decades ago
1952-1959: Arthur Samuel programmed a
digital computer to learn to play checkers
Implemented search, pruned using a scoring
function

1960: Donald Michie built a “machine” out of

304 matchboxes that could learn to play tic-
tac-toe

Slide credit: Lazebnik

 Games: a few decades ago
“In 1959 Arthur Samuel published a paper titled
‘Some Studies in Machine Learning Using the
Game of Checkers’, the first time the phrase
‘Machine Learning’ was used”

“Donald Michie’s description of reinforcement

learning comes from 1961, and is the first use of
the term reinforcement learning when applied to
a machine process … There have been some
developments in reinforcement learning since
1961, but only in details”

Slide credit: Lazebnik

Rodney Brooks essay, 8/28/2017
 Games: today
2013: DeepMind uses deep reinforcement
learning to beat humans at some Atari games

2016: DeepMind’s AlphaGo system beats Go

grandmaster Lee Sedol 4-1
2017: AlphaZero learns to play Go and chess
from scratch
Slide credit: Lazebnik
 Games: today

AlphaStar – Grandmaster-level StarCraft II

Imitation learning -> Self play

Photo credits: DeepMind, StarCraft

 Games: today

Photo credits: DeepMind, StarCraft

See stream, inside story, article
 Robots: a few decades ago
First robot to break commands into step-by-step plans
Planning
Performed search to find feasible/efficient paths
Combined NLP, vision

Shakey the robot (1972)

Slide credit: Lazebnik

 Robots: today
End-to-end training of deep visuomotor policies

Directly train
function from
image input to
motor torques.

Slide credit: Lazebnik

Overview video, training video
 Self-driving Cars: all aspects coming together

Deep learning crucial for the global success of automotive

autonomy – Automotive World, 6/26/2018
Slide credit: Lazebnik, Image Source: Nvidia
 New breakthroughs every week

Credit: https://news.artnet.com/art-world/stable-diffusion-version-2-ai- Credit: https://www.jousefmurad.com/ai/a-primer-on-stable-diffusion/,

generator-censorship-2220814 https://cdm.link/2022/08/stable-diffusion-the-slick-generative-ai-tool-just-
launched-and-went-live-on-github/
 New breakthroughs every week

Credit: https://github.com/features/copilot
 New breakthroughs every week

Credit: https://deepmind.com/blog/article/alphafold-a- Credit: https://www.deepmind.com/blog/accelerating-fusion-

solution-to-a-50-year-old-grand-challenge-in-biology science-through-learned-plasma-control
 Potential
Societal
Impact

McKinsey Global Institute’s

“Notes from the AI frontier applying AI for social good”

Source: McKinsey Global Institute analysis

 Bottlenecks limiting the use of AI for societal good

McKinsey Global Institute’s “Notes from the AI frontier applying AI for social good”
Source: McKinsey Global Institute analysis
McKinsey Global Institute’s “Notes from the AI frontier applying AI for social good”
Source: McKinsey Global Institute analysis
https://www.aimaps.ai/ https://today.umd.edu/where-are-the-new-ai-jobs-just-ask-ai
History of Neural Networks
 Neurons

Image credit: Wikipedia

A brief history of Neural Networks
1943: McCulloch and Pitts neurons

Neuroscientist + logician

Biological neurons -> abstract math

Neurons take in many inputs

Can be "quiet" or "fire"

 A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron

Perceptron

Perceptron image credit: Kurenkov, Krähenbühl

 Perceptron: Implementation

𝑇
𝑓 𝑥 = ቊ. 1 𝑖𝑓 𝑤 𝑥 > 0
. 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Image from: https://jtsulliv.github.io/perceptron/

Slide credit: Lazebnik
 Early 1950s: Hubel & Wiesel (Nobel 1981)

Introduce stimulus to cat, detect neurons firing off

Different patterns activate different neurons
Studied the development of visual system, first 3 months especially influential
Cat experiment video
Multi-layer Perceptron

Image credit: Kurenkov, Krähenbühl

Multi-layer Perceptron

Image credit: Kurenkov, Krähenbühl

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book

Discuss limitations of perceptrons, XOR,

credited with move from perceptrons​ to
symbolic AI

Slide credit: Lazebnik

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book
….AI Winter…. disappointment, less grant money, etc.

Slide credit: Lazebnik

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book
….AI Winter….
1980s: Fukushima’s Neocognitron

Slide credit: Lazebnik

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book
….AI Winter….
1980s: Fukushima’s Neocognitron
1986: Back-propagation [earlier works in 1960s-1980s, gained recognition in 1986]

Slide credit: Lazebnik

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book
1980s: Fukushima’s Neocognitron
1986: Back-propagation
1989: Convolutional neural networks (LeNet)

Slide credit: Lazebnik, Image credit: LeCun

A brief history of Neural Networks
1943: McCulloch and Pitts neurons
1958: Rosenblatt’s perceptron
1969: Minsky and Papert Perceptrons book
1980s: Fukushima’s Neocognitron
1986: Back-propagation
1989: Convolutional neural networks (LeNet)
2012: AlexNet
For much, much, much more detail, see
• Schmidhuber’s historical overview
• Deep learning, by LeCun, Bengio, Hinton
Slide credit: Lazebnik
Why now?
Why does it work now, and not in 1990s?
Why now?
• Neural networks + decades of machine learning research
 Why now?
• Neural networks + decades of machine learning research
• CPUs/GPUs/storage developed for other purposes

Image Credit: Nvidia, Google. Slide credit: Lelarge

 30 days of = x days of

Slide credit: Krähenbühl

 Why now?
• Neural networks + decades of Machine learning research
• CPUs/GPUs/storage developed for other purposes

Slide credit: Lelarge

 Why now?
• Neural networks + decades of Machine learning research
• CPUs/GPUs/storage developed for other purposes
• Lots of data from “the internet”

Slide credit: Lelarge

 Why now?
• Neural networks + decades of Machine learning research
• CPUs/GPUs/storage developed for other purposes
• Lots of data from “the internet”
• Tools and culture of collaborative and reproducible science
• Resources and efforts from large corporations

Slide credit: Lelarge

In this class
What all will we study?
 ML basics, linear classifiers Multilayer neural networks, backpropagation

Convolutional networks and applications Recurrent networks and applications

Transformers Generative Models Deep reinforcement learning

Course Inspiration: Resources, Slide credit: Lazebnik

Next class
ML basics, linear classifiers