Lecture 1: Introduction

Xuming He
SIST, ShanghaiTech
Fall, 2020

9/7/2020 Xuming He – CS 280 Deep Learning 1

 Outline
 Course logistics
 Overall objective

 Grading policy

 Pre-requisite / Syllabus

 Introduction to deep learning

 Machine learning review
 Artificial neurons

9/7/2020 Xuming He – CS 280 Deep Learning 2

 Course objectives
 Learning to use deep networks
 How to write from scratch, debug and train neural networks

 Toolboxes commonly used in practice

 Understanding deep models

 Key concepts and principles

 State of the art

 Some new topics from research field

 Focusing on vision-related problems

9/7/2020 Xuming He – CS 280 Deep Learning 3

 Syllabus & Schedule
 Piazza:
 piazza.com/shanghaitech.edu.cn/fall2020/cs280
 The schedule for the latter half of the semester may vary a bit
 Part I: Basic neural networks (1~1.5 weeks by Prof He)
 Linear models
 Multiple layer networks
 Gradient descent and BP
 Part II: Convolutional neural networks
 Part III: Recurrent neural networks
 Part IV: Generative neural networks
 Part V: Advanced Topics

9/7/2020 Xuming He – CS 280 Deep Learning 4

 Syllabus & Schedule
 Piazza:
 piazza.com/shanghaitech.edu.cn/fall2020/cs280
 The schedule for the latter half of the semester may vary a bit
 Part I: Basic neural networks (1~1.5 weeks)
 Part II: Convolutional neural networks (4 weeks by Prof He)
 CNN basics
 Understanding CNN
 CNN in Vision
 Part III: Recurrent neural networks
 Part IV: Generative neural networks
 Part V: Advanced Topics

9/7/2020 Xuming He – CS 280 Deep Learning 5

 Syllabus & Schedule
 Piazza:
 piazza.com/shanghaitech.edu.cn/fall2020/cs280
 The schedule for the latter half of the semester may vary a bit
 Part I: Basic neural networks (1~1.5 weeks)
 Part II: Convolutional neural networks (4 weeks)
 Part III: Recurrent neural networks (3 weeks by Prof Xu)
 LSTM, GRU
 Attention modeling
 RNN in Vision/NLP
 Transformer and Graph Neural Networks
 Part IV: Generative neural networks
 Part V: Advanced Topics

9/7/2020 Xuming He – CS 280 Deep Learning 6

 Syllabus & Schedule
 Piazza:
 piazza.com/shanghaitech.edu.cn/fall2020/cs280
 The schedule for the latter half of the semester may vary a bit
 Part I: Basic neural networks (1~1.5 weeks)
 Part II: Convolutional neural networks (4 weeks)
 Part III: Recurrent neural networks (3 weeks)
 Part IV: Generative neural networks (2 weeks by Prof Xu)
 Variational Auto Encoder (VAE)
 Generative deep nets (GAN)
 Part V: Advanced Topics (2 weeks)
 Note: no lectures in the following weeks
 Nov 9 ~ Nov 16 (CVPR)

9/7/2020 Xuming He – CS 280 Deep Learning 7

 Reference books and materials
 Deep learning:
 http://www.deeplearningbook.org/

 https://d2l.ai/

 Online deep learning courses:

 Stanford: CS230, CS231n
 CMU: 11-785
 MIT: 6.S191

 Additional reading materials on Piazza

 Survey papers, tutorials, etc.

9/7/2020 Xuming He – CS 280 Deep Learning 8

 Instructor and TAs
 Instructor: Prof Xuming He and Prof Lan Xu
 [email protected] ; [email protected]
 SIST 1A-304D ; 1C-203D

 TAs:
 Haozhe Wang, Qiuyue Wang, Guoxing Sun, Yannan He, Quan
Meng, Yinwenqi Jiang

 Office hours: To be announced on Piazza

 We will use Piazza as the main communication platform

9/7/2020 Xuming He – CS 280 Deep Learning 9

 Grading policy
 4 Problem sets: 10% x 4 = 40%
 Write-up problem sets + Programming tasks
 Final course project: 40% (+10%)
 Proposal
 Final report (Conference format)
 Presentation
 Bonus points for novel results: 10%
 10 Quizzes (in class): 2% x 10 = 20%
 Late policy
 A total of 7 free late (calendar) days to use, but no more than 4 late days can be
used on any single assignment.
 After that, 25% off per day late
 Does not apply to Final course project/Quizzes
 Collaboration policy
 Project team: 3~5 students
 Grading according to each member’s contribution

9/7/2020 Xuming He – CS 280 Deep Learning 10

 Administrative Stuff
 Plagiarism
 All assignments must be done individually
 You may not look at solutions from any other source
 You may not share solutions with any other students
 Plagiarism detection software will be used on all the programming
assignments
 You may discuss together or help another student but you cannot
give the exact solution

 Plagiarism punishment
 When one student copies from another student, both students
are responsible
 Zero point on the assignment or exam in question
 Repeated violation will result in an F grade for this course as well
as further discipline at the school/university level
 Pre-requisite
 Proficiency in Python
 All class assignments will be in Python (and use numpy)
 A Python tutorial available on Piazza
 Calculus, Linear Algebra, Probability and Statistics
 Undergrad course level
 Equivalent knowledge of Andrew Ng’s CS229 (Machine
Learning)
 Formulating cost functions
 Taking derivatives
 Performing optimization with gradient descent
 Will be evaluated in next quiz (Wednesday)

9/7/2020 Xuming He – CS 280 Deep Learning 12

 Outline
 Course logistics
 Introduction to deep learning
 What & Why deep learning?

 Machine Learning review

 Artificial neurons

Acknowledgement: Bhiksha Raj@CMU’s course notes

9/7/2020 Xuming He – CS 280 Deep Learning 13
 Introduction
 Our goal: Build intelligent algorithms to make sense of data
 Example: Recognizing objects in images

red panda (Ailurus fulgens)

 Example: Predicting what would happen next

Vondrick et al. CVPR2016

9/7/2020 Xuming He – CS 280 Deep Learning 14

 Introduction
 Our goal: Build intelligent algorithms to make sense of data
 Example: Recognizing objects in images

 Example: Predicting what would happen next

9/7/2020 Xuming He – CS 280 Deep Learning 15

 Introduction
 A broad range of real-world applications
 Speech recognition
 Input: sound wave → Output: transcript
 Language translation
 Input: text in language A (Eng) → Output: text in language B (Chs)
 Image classification
 Input: images → Output: image category (cat, dog, car, house, etc.)
 Autonomous driving
 Input: sensory inputs → Output: actions (straight, left, right, stop, etc.)

 Main challenges: difficult to manually design the algorithms

9/7/2020 16
 A data-driven approach
 Each task as a mapping function (or a model)
Mapping function
Input data Expected output

 input data: images

 expected output: object or action names

 Building such mapping functions from data

Mapping function

red panda (Ailurus fulgens)

9/7/2020 Xuming He – CS 280 Deep Learning 17
 A data-driven approach
 Building a mapping function (model)

 x: input data
 y: expected output
 : parameters to be estimated

 Learning the model from data

 Given a dataset
 Find the ‘best’ parameter , such that

 And it can be generalized to unseen input data

9/7/2020 18
 What is deep learning?
 Using deep neural networks as the mapping function

 Model: Deep neural networks

 A family of parametric models
 Consisting of many ‘simple’ computational units
 Constructing a multi-layer representation of input

Image from Jeff Clune’s Deep Learning Overview

9/7/2020 19
 What is deep learning?
 Using deep neural networks as the mapping function

 Learning: Parameter estimation from data

 Parameters: connection weights between units
 Formulated as an optimization problem
 Efficient algorithms for handling large-scale models & datasets

9/7/2020 20
 Why deep networks?
 Inspiration from visual cortex

9/7/2020 21
 Why deep networks?
 A deep architecture can represent certain functions
(exponentially) more compactly
 Learning a rich representation of input data

9/7/2020 22
 Recent success with DL
Recent
 Somesuccess with
recent neuralwith
success networks
neural networks
Steel drum
The Im ag e C lassification C halleng e:
1,000 ob ject classes
1,431,167 im ag es

Recent success with neural networks

Recent success with neural network
Russakovsky et al. arX iv, 2014

• Some recent successes with neural networks

Fei-Fei Li & Justin Jo h nso n & S erena Y eung Lecture 1 - 24 4/4/2017

– A bit of hyperbole, but still..

9/7/2020 23
 Summary: Why deep learning?
 One of the major thrust areas recently in various pattern
recognition, prediction and data analysis
 Efficient representation of data and computation
 Other key factors: large datasets and hardware

 The state of the art in many problems

 Often exceeding previous benchmarks by large margins
 Achieve better performances than human for certain “complex”
tasks.

 But also somewhat controversial …

 Lack of theoretical understanding
 Sometimes difficult to make it work in practice

9/7/2020 24
 Is it alchemy?

9/7/2020 Xuming He – CS 280 Deep Learning 25

 Questions to ask
 Understanding neural networks
 What is different from traditional ML methods?

 How it works for specific problems?

 Why get great performance?

 Future development
 Its limitation and weakness?

 After more than 10 years, what is on-going or next?

 The road to general-purpose AI?

9/7/2020 Xuming He – CS 280 Deep Learning 26

 Outline
 Course logistics
 Introduction to deep learning
 Machine learning review
 Math review

 Supervised learning

 Artificial neurons

Acknowledgement: Hugo Larochelle’s, Mehryar Mohri@NYU’s & Yingyu

Liang@Princeton’s course notes
9/7/2020 Xuming He – CS 280 Deep Learning 27
 Math review – Calculus
 Gradient

9/7/2020 Xuming He – CS 280 Deep Learning 28

 Math review – Calculus
 Local and global minima
 Necessary condition

 Sufficient condition
 Hessian is positive definite

9/7/2020 Xuming He – CS 280 Deep Learning 29

 Math review – Probability
 Factorization

9/7/2020 Xuming He – CS 280 Deep Learning 30

 Math review – Probability
 Common distributions

9/7/2020 Xuming He – CS 280 Deep Learning 31

 Math review – Statistics
 Monte Carlo estimation

 Maximum likelihood

 Independent and identically distributed

9/7/2020 Xuming He – CS 280 Deep Learning 32

 ML tasks
 Classification: assign a category to each item (e.g.,
document classification)
 Regression: predict a real value for each item (e.g.,
prediction of stock values, economic variables)
 Ranking: order items according to some criterion (e.g.,
relevant web pages returned by a search engine)
 Clustering: partition data into ‘homogenous’ regions
(e.g., analysis of very large data sets)
 Dimensionality reduction: find lower-dimensional
manifold preserving some properties of the data

9/7/2020 Xuming He – CS 280 Deep Learning 33

 Standard learning scenarios
 Unsupervised learning: no labeled data
 Supervised learning: uses labeled data for prediction on
unseen points
 Semi-supervised learning: uses labeled and unlabeled
data for prediction on unseen points
 Reinforcement learning: uses reward to learn prediction
on action policies.
 …

9/7/2020 Xuming He – CS 280 Deep Learning 34

 Supervised learning
 Task formulation

9/7/2020 Xuming He – CS 280 Deep Learning 35

 Supervised learning
 Task formulation

9/7/2020 Xuming He – CS 280 Deep Learning 36

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 37

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 38

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 39

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 40

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 41

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 42

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 43

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 44

 Learning problem
 Problem setup

9/7/2020 Xuming He – CS 280 Deep Learning 45

 Learning as iterative optimization
 Gradient descent

9/7/2020 Xuming He – CS 280 Deep Learning 46

 Learning as iterative optimization
 Stochastic gradient descent (SGD)

9/7/2020 Xuming He – CS 280 Deep Learning 47

 Supervised learning pipeline
 Three steps

 Datasets & hyper-parameters

 Hyper-parameter: a parameter of a model that is not trained
(specified before training)

9/7/2020 Xuming He – CS 280 Deep Learning 48

 Generalization
 Model selection for better generalization
 Capacity: flexibility of a model
 Underfitting: state of model which could improve generalization
with more training or capacity
 Overfitting: state of model which could improve generalization
with less training or capacity
 Model Selection: process of choosing the best hyper-parameters
on validation set

9/7/2020 Xuming He – CS 280 Deep Learning 49

 Generalization
 Training/Validation curves

9/7/2020 Xuming He – CS 280 Deep Learning 50

 Questions
 Generalization
 Interaction between training set size/capacity/training time and
training error/generalization error
 If capacity increases:
 Training error will ?
 Generalization error will ?
 If training time increases:
 Training error will ?
 Generalization error will ?
 If training set size increases:
 Generalization error will ?
 Gap between the training and generalization error will ?

9/7/2020 Xuming He – CS 280 Deep Learning 51

 Outline
 Course logistics
 Introduction to deep learning
 Machine learning review
 Artificial neurons
 Math model

 Perceptron algorithm

Acknowledgement: Hugo Larochelle’s, Mehryar Mohri@NYU’s & Yingyu

Liang@Princeton’s course notes
9/7/2020 Xuming He – CS 280 Deep Learning 52
 Artificial Neuron
 Biological inspiration

https://www.youtube.com/watch?v=m0rHZ_RDdyQ
9/7/2020 Xuming He – CS 280 Deep Learning 53
 Artificial Neuron
 Biological inspiration

9/7/2020 Xuming He – CS 280 Deep Learning 54

 Mathematical model of a neuron

9/7/2020 55
 Activation functions

9/7/2020 Xuming He – CS 280 Deep Learning 56

 Capacity of single neuron
 Sigmoid activation function

9/7/2020 Xuming He – CS 280 Deep Learning 57

 What a single neuron does?
 A neuron (perceptron) fires if its input is within a specific
angle of its weight
 If the input pattern matches the weight pattern closely enough

9/7/2020 Xuming He – CS 280 Deep Learning 58

 Single neuron as a linear classifier
 Binary classification

9/7/2020 Xuming He – CS 280 Deep Learning 59

 How do we determine the weights?
 Learning problem

9/7/2020 Xuming He – CS 280 Deep Learning 60

 Linear classification
 Learning problem: simple approach

• Drawback: Sensitive to “outliers”

9/7/2020 Xuming He – CS 280 Deep Learning 61

 1D Example
 Compare two predictors

9/7/2020 Xuming He – CS 280 Deep Learning 62

 Perceptron algorithm
 Learn a single neuron for binary classification

https://towardsdatascience.com/perceptron-explanation-implementation-and-a-visual-example-3c8e76b4e2d1

9/7/2020 Xuming He – CS 280 Deep Learning 63

 Perceptron algorithm
 Learn a single neuron for binary classification

 Task formulation

9/7/2020 Xuming He – CS 280 Deep Learning 64

 Perceptron algorithm
 Algorithm outline

9/7/2020 Xuming He – CS 280 Deep Learning 65

 Perceptron algorithm
 Intuition: correct the current mistake

9/7/2020 Xuming He – CS 280 Deep Learning 66

 Perceptron algorithm
 The Perceptron theorem

9/7/2020 Xuming He – CS 280 Deep Learning 67

Hyperplane Distance
 Perceptron algorithm
 The Perceptron theorem: proof

9/7/2020 Xuming He – CS 280 Deep Learning 69

 Perceptron algorithm
 The Perceptron theorem: proof

9/7/2020 Xuming He – CS 280 Deep Learning 70

 Perceptron algorithm
 The Perceptron theorem: proof intuition

9/7/2020 Xuming He – CS 280 Deep Learning 71

 Perceptron algorithm
 The Perceptron theorem: proof

9/7/2020 Xuming He – CS 280 Deep Learning 72

 Perceptron algorithm
 The Perceptron theorem

9/7/2020 Xuming He – CS 280 Deep Learning 73

 Perceptron Learning problem
 What loss function is minimized?

9/7/2020 Xuming He – CS 280 Deep Learning 74

 Perceptron algorithm
 What loss function is minimized?

9/7/2020 Xuming He – CS 280 Deep Learning 75

 Perceptron algorithm
 What loss function is minimized?

9/7/2020 Xuming He – CS 280 Deep Learning 76

 Summary
 Introduction to deep learning
 Course logistics
 Review of basic math & ML
 Artificial neurons

 Next time
 Basic neural networks
 First Quiz on prerequisite

9/7/2020 Xuming He – CS 280 Deep Learning 77