SlideShare a Scribd company logo

Icml2012 tutorial representation_learning

Z
zukun

This document provides an outline and overview of Yoshua Bengio's 2012 tutorial on representation learning. The key points covered include: 1) The tutorial will cover motivations for representation learning, algorithms such as probabilistic models and auto-encoders, and analysis and practical issues. 2) Representation learning aims to automatically learn good representations of data rather than relying on handcrafted features. Learning representations can help address challenges like exploiting unlabeled data and the curse of dimensionality. 3) Deep learning algorithms attempt to learn multiple levels of increasingly complex representations, with the goal of developing more abstract, disentangled representations that generalize beyond local patterns in the data.

1 of 133
Downloaded 85 times
1
Representa)on  Learning       Yoshua  Bengio     ICML  2012  Tutorial   June  26th  2012,  Edinburgh,  Scotland        
2
Outline of the Tutorial 1.  Mo>va>ons  and  Scope   1.  Feature  /  Representa>on  learning   2.  Distributed  representa>ons   3.  Exploi>ng  unlabeled  data   4.  Deep  representa>ons   5.  Mul>-­‐task  /  Transfer  learning   6.  Invariance  vs  Disentangling   2.  Algorithms   1.  Probabilis>c  models  and  RBM  variants   2.  Auto-­‐encoder  variants  (sparse,  denoising,  contrac>ve)   3.  Explaining  away,  sparse  coding  and  Predic>ve  Sparse  Decomposi>on   4.  Deep  variants   3.  Analysis,  Issues  and  Prac>ce   1.  Tips,  tricks  and  hyper-­‐parameters   2.  Par>>on  func>on  gradient   3.  Inference   4.  Mixing  between  modes   5.  Geometry  and  probabilis>c  Interpreta>ons  of  auto-­‐encoders   6.  Open  ques>ons   See  (Bengio,  Courville  &  Vincent  2012)     “Unsupervised  Feature  Learning  and  Deep  Learning:  A  Review  and  New  Perspec>ves”   And  http://www.iro.umontreal.ca/~bengioy/talks/deep-learning-tutorial-2012.html  for  a   detailed  list  of  references.  
3
Ultimate Goals •  AI   •  Needs  knowledge   •  Needs  learning   •  Needs  generalizing  where  probability  mass   concentrates   •  Needs  ways  to  fight  the  curse  of  dimensionality   •  Needs  disentangling  the  underlying  explanatory  factors   (“making  sense  of  the  data”)   3  
4
Representing data •  In  prac>ce  ML  very  sensi>ve  to  choice  of  data  representa>on   à  feature  engineering  (where  most  effort  is  spent)   à (beber)  feature  learning  (this  talk):        automa>cally  learn  good  representa>ons     •  Probabilis>c  models:   •  Good  representa>on  =  captures  posterior  distribu,on  of   underlying  explanatory  factors  of  observed  input   •  Good  features  are  useful  to  explain  varia>ons   4  
5
Deep Representation Learning Deep  learning  algorithms  abempt  to  learn  mul>ple  levels  of  representa>on  of   increasing  complexity/abstrac>on     When  the  number  of  levels  can  be  data-­‐ selected,  this  is  a  deep  architecture       5  
6
A Good Old Deep Architecture   Op>onal  Output  layer   Here  predic>ng  a  supervised  target     Hidden  layers   These  learn  more  abstract   representa>ons  as  you  head  up     Input  layer   This  has  raw  sensory  inputs  (roughly)   6  
7
What We Are Fighting Against: The Curse ofDimensionality      To  generalize  locally,   need  representa>ve   examples  for  all   relevant  varia>ons!     Classical  solu>on:  hope   for  a  smooth  enough   target  func>on,  or   make  it  smooth  by   handcrafing  features  
8
Easy Learning * * = example (x,y) * y * true unknown function * * * * * * * * * * learned function: prediction = f(x) x
9
Local Smoothness Prior: Locally Capture the Variations * = training example y * true function: unknown prediction learnt = interpolated f(x) * * test point x x *
10
Real Data Are on Highly Curved Manifolds 10  
11
Not Dimensionality so much as Number of Variations (Bengio, Delalleau & Le Roux 2007) •  Theorem:  Gaussian  kernel  machines  need  at  least  k  examples   to  learn  a  func>on  that  has  2k  zero-­‐crossings  along  some  line             •  Theorem:  For  a  Gaussian  kernel  machine  to  learn  some   maximally  varying  func>ons    over  d  inputs  requires  O(2d)   examples    
12
Is there any hope to generalize non-locally? Yes! Need more priors! 12  
13
Part  1   Six Good Reasons to Explore Representation Learning 13  
14
#1 Learning features, not just handcrafting them Most  ML  systems  use  very  carefully  hand-­‐designed   features  and  representa>ons   Many  prac>>oners  are  very  experienced  –  and  good  –  at  such   feature  design  (or  kernel  design)   In  this  world,  “machine  learning”  reduces  mostly  to  linear   models  (including  CRFs)  and  nearest-­‐neighbor-­‐like  features/ models  (including  n-­‐grams,  kernel  SVMs,  etc.)     Hand-­‐cra7ing  features  is  )me-­‐consuming,  bri<le,  incomplete   14  
15
How can we automatically learn good features? Claim:  to  approach  AI,  need  to  move  scope  of  ML  beyond   hand-­‐crafed  features  and  simple  models   Humans  develop  representa>ons  and  abstrac>ons  to   enable  problem-­‐solving  and  reasoning;  our  computers   should  do  the    same   Handcrafed  features  can  be  combined  with  learned   features,  or  new  more  abstract  features  learned  on  top   of  handcrafed  features   15  
16
#2 The need for distributed representations •  Clustering,  Nearest-­‐ Clustering   Neighbors,  RBF  SVMs,  local   non-­‐parametric  density   es>ma>on  &  predic>on,   decision  trees,  etc.   •  Parameters  for  each   dis>nguishable  region   •  #  dis>nguishable  regions   linear  in  #  parameters   16  
17
#2 The need for distributed representations Mul>-­‐   Clustering   •  Factor  models,  PCA,  RBMs,   Neural  Nets,  Sparse  Coding,   Deep  Learning,  etc.   •  Each  parameter  influences   many  regions,  not  just  local   neighbors   •  #  dis>nguishable  regions   grows  almost  exponen>ally   C1   C2   C3   with  #  parameters   •  GENERALIZE  NON-­‐LOCALLY   TO  NEVER-­‐SEEN  REGIONS   input   17  
18
#2 The need for distributed representations Mul>-­‐   Clustering   Clustering   Learning  a  set  of  features  that  are  not  mutually  exclusive   can  be  exponen>ally  more  sta>s>cally  efficient  than   nearest-­‐neighbor-­‐like  or  clustering-­‐like  models   18  
19
#3 Unsupervised feature learning Today,  most  prac>cal  ML  applica>ons  require  (lots  of)   labeled  training  data   But  almost  all  data  is  unlabeled   The  brain  needs  to  learn  about  1014  synap>c  strengths   …  in  about  109  seconds   Labels  cannot  possibly  provide  enough  informa>on   Most  informa>on  acquired  in  an  unsupervised  fashion   19  
20
#3 How do humans generalize from very few examples? •  They  transfer  knowledge  from  previous  learning:   •  Representa>ons   •  Explanatory  factors   •  Previous  learning  from:  unlabeled  data                    +  labels  for  other  tasks   •  Prior:  shared  underlying  explanatory  factors,  in   par)cular  between  P(x)  and  P(Y|x)     20    
21
#3 Sharing Statistical Strength by Semi-Supervised Learning •  Hypothesis:  P(x)  shares  structure  with  P(y|x)   purely   semi-­‐   supervised   supervised   21  
22
#4 Learning multiple levels of representation There  is  theore>cal  and  empirical  evidence  in  favor  of   mul>ple  levels  of  representa>on    Exponen)al  gain  for  some  families  of  func)ons   Biologically  inspired  learning   Brain  has  a  deep  architecture   Cortex  seems  to  have  a     generic  learning  algorithm     Humans  first  learn  simpler     concepts  and  then  compose     them  to  more  complex  ones   22    
23
#4 Sharing Components in a Deep Architecture Polynomial  expressed  with  shared  components:  advantage  of   depth  may  grow  exponen>ally       Sum-­‐product   network  
24
#4 Learning multiple levels of representation (Lee,  Largman,  Pham  &  Ng,  NIPS  2009)   (Lee,  Grosse,  Ranganath  &  Ng,  ICML  2009)     Successive  model  layers  learn  deeper  intermediate  representa>ons     High-­‐level   Layer  3   linguis>c  representa>ons   Parts  combine   to  form  objects   Layer  2   Layer  1   24   Prior:  underlying  factors  &  concepts  compactly  expressed  w/  mul)ple  levels  of  abstrac)on    
25
#4 Handling the compositionality of human language and thought zt-­‐1   zt   zt+1   •  Human  languages,  ideas,  and   ar>facts  are  composed  from   simpler  components   xt-­‐1   xt   xt+1   •  Recursion:  the  same   operator  (same  parameters)   is  applied  repeatedly  on   different  states/components   of  the  computa>on   •  Result  afer  unfolding  =  deep   (Bobou  2011,  Socher  et  al  2011)   representa>ons   25  
26
#5 Multi-Task Learning task 1 task 2 task 3 •  Generalizing  beber  to  new   output y1 output y2 output y3 tasks  is  crucial  to  approach  AI   Task  A   Task  B   Task  C   •  Deep  architectures  learn  good   intermediate  representa>ons   that  can  be  shared  across  tasks   •  Good  representa>ons  that   disentangle  underlying  factors   of  varia>on  make  sense  for   raw input x many  tasks  because  each  task   concerns  a  subset  of  the  factors   26  
27
#5 Sharing Statistical Strength task 1 task 2 task 3 •  Mul>ple  levels  of  latent   output y1 output y2 output y3 variables  also  allow   Task  A   Task  B   Task  C   combinatorial  sharing  of   sta>s>cal  strength:   intermediate  levels  can  also   be  seen  as  sub-­‐tasks   •  E.g.  dic>onary,  with   intermediate  concepts  re-­‐ used  across  many  defini>ons   raw input x Prior:  some  shared  underlying  explanatory  factors  between  tasks       27  
28
#5 Combining Multiple Sources of Evidence with Shared Representations person   url   event   •  Tradi>onal  ML:  data  =  matrix   url   words   history   •  Rela>onal  learning:  mul>ple  sources,   different  tuples  of  variables   •  Share  representa>ons  of  same  types   across  data  sources   •  Shared  learned  representa>ons  help   event   url   person   propagate  informa>on  among  data   history   words   url   sources:  e.g.,  WordNet,  XWN,   Wikipedia,  FreeBase,  ImageNet… (Bordes  et  al  AISTATS  2012)   P(person,url,event)   P(url,words,history)   28  
29
#5 Different object types represented in same space Google:   S.  Bengio,  J.   Weston  &  N.   Usunier   (IJCAI  2011,   NIPS’2010,   JMLR  2010,   MLJ  2010)  
30
#6 Invariance and Disentangling •  Invariant  features   •  Which  invariances?   •  Alterna>ve:  learning  to  disentangle  factors   •  Good  disentangling  à      avoid  the  curse  of  dimensionality   30  
31
#6 Emergence of Disentangling •  (Goodfellow  et  al.  2009):  sparse  auto-­‐encoders  trained   on  images     •  some  higher-­‐level  features  more  invariant  to   geometric  factors  of  varia>on     •  (Glorot  et  al.  2011):  sparse  rec>fied  denoising  auto-­‐ encoders  trained  on  bags  of  words  for  sen>ment   analysis   •  different  features  specialize  on  different  aspects   (domain,  sen>ment)   31   WHY?  
32
#6 Sparse Representations •  Just  add  a  penalty  on  learned  representa>on   •  Informa>on  disentangling  (compare  to  dense  compression)   •  More  likely  to  be  linearly  separable  (high-­‐dimensional  space)   •  Locally  low-­‐dimensional  representa>on  =  local  chart   •  Hi-­‐dim.  sparse  =  efficient  variable  size  representa>on                  =  data  structure   Few  bits  of  informa>on                                                        Many  bits  of  informa>on   Prior:  only  few  concepts  and  a<ributes  relevant  per  example     32  
33
Bypassing the curse We  need  to  build  composi>onality  into  our  ML  models     Just  as  human  languages  exploit  composi>onality  to  give   representa>ons  and  meanings  to  complex  ideas   Exploi>ng  composi>onality  gives  an  exponen>al  gain  in   representa>onal  power   Distributed  representa>ons  /  embeddings:  feature  learning   Deep  architecture:  mul>ple  levels  of  feature  learning   Prior:  composi>onality  is  useful  to  describe  the   world  around  us  efficiently   33    
34
Bypassing the curse by sharing statistical strength •  Besides  very  fast  GPU-­‐enabled  predictors,  the  main  advantage   of  representa>on  learning  is  sta>s>cal:  poten>al  to  learn  from   less  labeled  examples  because  of  sharing  of  sta>s>cal  strength:   •  Unsupervised  pre-­‐training  and  semi-­‐supervised  training   •  Mul>-­‐task  learning   •  Mul>-­‐data  sharing,  learning  about  symbolic  objects  and  their   rela>ons   34  
35
Why now? Despite  prior  inves>ga>on  and  understanding  of  many  of  the   algorithmic  techniques  …   Before  2006  training  deep  architectures  was  unsuccessful   (except  for  convolu>onal  neural  nets  when  used  by  people  who  speak  French)   What  has  changed?   •  New  methods  for  unsupervised  pre-­‐training  have  been   developed  (variants  of  Restricted  Boltzmann  Machines  =   RBMs,  regularized  autoencoders,  sparse  coding,  etc.)   •  Beber  understanding  of  these  methods   •  Successful  real-­‐world  applica>ons,  winning  challenges  and   bea>ng  SOTAs  in  various  areas   35  
36
Major Breakthrough in 2006 •  Ability  to  train  deep  architectures  by   using  layer-­‐wise  unsupervised   learning,  whereas  previous  purely   supervised  abempts  had  failed   •  Unsupervised  feature  learners:   •  RBMs   •  Auto-­‐encoder  variants   Bengio Montréal •  Sparse  coding  variants   Toronto Hinton Le Cun New York 36  
37
Unsupervised and Transfer Learning Challenge + Transfer Learning Challenge: Deep Learning 1st Place NIPS’2011   Raw  data   Transfer   Learning   1  layer   2  layers   Challenge     Paper:   ICML’2012   ICML’2011   workshop  on   Unsup.  &   Transfer  Learning   3  layers   4  layers  
38
More Successful Applications •  Microsof  uses  DL  for  speech  rec.  service  (audio  video  indexing),  based  on   Hinton/Toronto’s  DBNs  (Mohamed  et  al  2011)   •  Google  uses  DL  in  its  Google  Goggles  service,  using  Ng/Stanford  DL  systems   •  NYT  today  talks  about  these:  http://www.nytimes.com/2012/06/26/technology/ in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=1 •  Substan>ally  bea>ng  SOTA  in  language  modeling  (perplexity  from  140  to  102   on  Broadcast  News)  for  speech  recogni>on  (WSJ  WER  from  16.9%  to  14.4%)   (Mikolov  et  al  2011)  and  transla>on  (+1.8  BLEU)  (Schwenk  2012)   •  SENNA:  Unsup.  pre-­‐training  +  mul>-­‐task  DL  reaches  SOTA  on  POS,  NER,  SRL,   chunking,  parsing,  with  >10x  beber  speed  &  memory  (Collobert  et  al  2011)   •  Recursive  nets  surpass  SOTA  in  paraphrasing  (Socher  et  al  2011)   •  Denoising  AEs  substan>ally  beat  SOTA  in  sen>ment  analysis  (Glorot  et  al  2011)   •  Contrac>ve  AEs  SOTA  in  knowledge-­‐free  MNIST  (.8%  err)  (Rifai  et  al  NIPS  2011)   •  Le  Cun/NYU’s  stacked  PSDs  most  accurate  &  fastest  in  pedestrian  detec>on   and  DL  in  top  2  winning  entries  of  German  road  sign  recogni>on  compe>>on     38  
39
39  
40
Part  2   Representation Learning Algorithms 40  
41
A neural network = running several logistic regressions at the same time If  we  feed  a  vector  of  inputs  through  a  bunch  of  logis>c  regression   func>ons,  then  we  get  a  vector  of  outputs   But  we  don’t  have  to  decide   ahead  of  >me  what  variables   these  logis>c  regressions  are   trying  to  predict!   41  
42
A neural network = running several logistic regressions at the same time …  which  we  can  feed  into  another  logis>c  regression  func>on   and  it  is  the  training   criterion  that  will   decide  what  those   intermediate  binary   target  variables  should   be,  so  as  to  make  a   good  job  of  predic>ng   the  targets  for  the  next   layer,  etc.   42  
43
A neural network = running several logistic regressions at the same time •  Before  we  know  it,  we  have  a  mul>layer  neural  network….   How to do unsupervised training? 43  
44
PCA code= latent features h = Linear Manifold = Linear Auto-Encoder … … = Linear Gaussian Factors input reconstruction input  x,  0-­‐mean   Linear  manifold   features=code=h(x)=W  x   reconstruc>on(x)=WT  h(x)  =  WT  W  x   W  =  principal  eigen-­‐basis  of  Cov(X)   reconstruc>on(x)   reconstruc>on  error  vector   x   Probabilis>c  interpreta>ons:   1.  Gaussian  with  full   covariance  WT  W+λI   2.  Latent  marginally  iid   Gaussian  factors  h  with       x  =  WT  h  +  noise   44  
45
Directed Factor Models •  P(h)  factorizes  into  P(h1)  P(h2)…   h1 h2 h3 h4 h5 •  Different  priors:   •  PCA:  P(hi)  is  Gaussian   W3   W1   •  ICA:  P(hi)  is  non-­‐parametric   W5   •  Sparse  coding:  P(hi)  is  concentrated  near  0   •  Likelihood  is  typically  Gaussian  x  |  h     x1 x2          with  mean  given  by  WT  h   •  Inference  procedures  (predic>ng  h,  given  x)  differ   •  Sparse  h:  x  is  explained  by  the  weighted  addi>on  of  selected   filters  hi   x   W1   W3   W5   h1   h3   h5                               =  .9  x                        +  .8  x                      +  .7  x   45  
46
Stacking Single-Layer Learners •  PCA  is  great  but  can’t  be  stacked  into  deeper  more  abstract   representa>ons  (linear  x  linear  =  linear)   •  One  of  the  big  ideas  from  Hinton  et  al.  2006:  layer-­‐wise   unsupervised  feature  learning   Stacking Restricted Boltzmann Machines (RBM) à Deep Belief Network (DBN) 46  
47
Effective deep learning became possible through unsupervised pre-training [Erhan  et  al.,  JMLR  2010]   (with  RBMs  and  Denoising  Auto-­‐Encoders)   Purely  supervised  neural  net   With  unsupervised  pre-­‐training   47  
48
Layer-wise Unsupervised Learning input … 48  
49
Layer-Wise Unsupervised Pre-training features … input … 49  
50
Layer-Wise Unsupervised Pre-training ? reconstruction … input = … of input features … input … 50  
51
Layer-Wise Unsupervised Pre-training features … input … 51  
52
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 52  
53
Layer-Wise Unsupervised Pre-training Layer-wise Unsupervised Learning ? reconstruction … … = of features More abstract … features features … input … 53  
54
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 54  
55
Layer-wise Unsupervised Learning Even more abstract features … More abstract … features features … input … 55  
56
Supervised Fine-Tuning Output Target f(X) six ? = Y two! Even more abstract features … More abstract … features features … input … •  Addi>onal  hypothesis:  features  good  for  P(x)  good  for  P(y|x)   56  
57
Restricted Boltzmann Machines 57  
58
Undirected Models: the Restricted Boltzmann Machine [Hinton  et  al  2006]   •  Probabilis>c  model  of  the  joint  distribu>on  of   h1 h2 h3 the  observed  variables  (inputs  alone  or  inputs   and  targets)  x   •  Latent  (hidden)  variables  h  model  high-­‐order   dependencies   •  Inference  is  easy,  P(h|x)  factorizes   x1 x2 •  See  Bengio  (2009)  detailed  monograph/review:        “Learning  Deep  Architectures  for  AI”.   •  See  Hinton  (2010)            “A  prac,cal  guide  to  training  Restricted  Boltzmann  Machines”  
59
Boltzmann Machines & MRFs •  Boltzmann  machines:        (Hinton  84)     •  Markov  Random  Fields:     Sof  constraint  /  probabilis>c  statement           ¡  More    nteres>ng  with  latent  variables!   i                                                                                                                                                                                                                                                                                                                                                                            
60
Restricted Boltzmann Machine (RBM) •  A  popular  building   block  for  deep   architectures   hidden   •  Bipar)te  undirected   graphical  model   observed
61
Gibbs Sampling in RBMs h1 ~ P(h|x1) h2 ~ P(h|x2) h3 ~ P(h|x3) x1 x2 ~ P(x|h1) x3 ~ P(x|h2) ¡  Easy inference P(h|x)  and  P(x|h)  factorize   ¡  Efficient block Gibbs P(h|x)=  Π  P(hi|x)   sampling xàhàxàh… i  
62
Problems with Gibbs Sampling In  prac>ce,  Gibbs  sampling  does  not  always  mix  well…   RBM trained by CD on MNIST Chains from random state Chains from real digits (Desjardins  et  al  2010)  
63
RBM with (image, label) visible units hidden h U W image y 0 0 1 0 x label y (Larochelle  &  Bengio  2008)  
64
RBMs are Universal Approximators (Le Roux & Bengio 2008) •  Adding  one  hidden  unit  (with  proper  choice  of  parameters)   guarantees  increasing  likelihood     •  With  enough  hidden  units,  can  perfectly  model  any  discrete   distribu>on   •  RBMs  with  variable  #  of  hidden  units  =  non-­‐parametric  
65
RBM Conditionals Factorize
66
RBM Energy Gives Binomial Neurons
67
RBM Free Energy •  Free  Energy  =  equivalent  energy  when  marginalizing       •  Can  be  computed  exactly  and  efficiently  in  RBMs     •  Marginal  likelihood  P(x)  tractable  up  to  par>>on  func>on  Z  
68
Factorization of the Free Energy Let  the  energy  have  the  following  general  form:   Then  
69
Energy-Based Models Gradient
70
Boltzmann Machine Gradient •  Gradient  has  two  components:   positive phase negative phase ¡  In  RBMs,  easy  to  sample  or  sum  over  h|x   ¡  Difficult  part:  sampling  from  P(x),  typically  with  a  Markov  chain  
71
Positive & Negative Samples •  Observed (+) examples push the energy down •  Generated / dream / fantasy (-) samples / particles push the energy up X+ X- Equilibrium:  E[gradient]  =  0  
72
Training RBMs Contras>ve  Divergence:    start  nega>ve  Gibbs  chain  at  observed  x,  run  k   (CD-­‐k)   Gibbs  steps     SML/Persistent  CD:   run  nega>ve  Gibbs  chain  in  background  while   (PCD)    weights  slowly  change   Fast  PCD:   two  sets  of  weights,  one  with  a  large  learning  rate   only  used  for  nega>ve  phase,  quickly  exploring   modes   Herding:   Determinis>c  near-­‐chaos  dynamical  system  defines   both  learning  and  sampling   Tempered  MCMC:   use  higher  temperature  to  escape  modes  
73
Contrastive Divergence Contrastive Divergence (CD-k): start negative phase block Gibbs chain at observed x, run k Gibbs steps (Hinton 2002) h+ ~ P(h|x+) h-~ P(h|x-) Observed x+ k = 2 steps Sampled x- positive phase negative phase push down Free Energy x+ x- push up
74
Persistent CD (PCD) / Stochastic Max. Likelihood (SML) Run  nega>ve  Gibbs  chain  in  background  while  weights  slowly   change  (Younes  1999,  Tieleman  2008):   •    Guarantees  (Younes  1999;  Yuille  2005)     •  If  learning  rate  decreases  in  1/t,          chain  mixes  before  parameters  change  too  much,          chain  stays  converged  when  parameters  change   h+ ~ P(h|x+) previous x- Observed x+ new x- (positive phase)
75
PCD/SML + large learning rate Nega>ve  phase  samples  quickly  push  up  the  energy  of   wherever  they  are  and  quickly  move  to  another  mode   push FreeEnergy down x+ x- push up
76
Some RBM Variants •  Different  energy  func>ons  and  allowed                   values  for  the  hidden  and  visible  units:   •  Hinton  et  al  2006:  binary-­‐binary  RBMs   •  Welling  NIPS’2004:  exponen>al  family  units   •  Ranzato  &  Hinton  CVPR’2010:  Gaussian  RBM  weaknesses  (no   condi>onal  covariance),  propose  mcRBM   •  Ranzato  et  al  NIPS’2010:  mPoT,  similar  energy  func>on   •  Courville  et  al  ICML’2011:  spike-­‐and-­‐slab  RBM     76  
77
Convolutionally Trained Spike & Slab RBMs Samples
78
Training  examples   Generated  samples   ssRBM is not Cheating
79
Auto-Encoders & Variants 79  
80
Auto-Encoders  code=  latent  features   •  MLP  whose  target  output  =  input   •  Reconstruc>on=decoder(encoder(input)),            e  ncoder                      decoder   e.g.    input   …   …      reconstruc>on   •  Probable  inputs  have  small  reconstruc>on  error   because  training  criterion  digs  holes  at  examples   •  With  bobleneck,  code  =  new  coordinate  system   •  Encoder  and  decoder  can  have  1  or  more  layers   •  Training  deep  auto-­‐encoders  notoriously  difficult   80    
81
Stacking Auto-Encoders Auto-­‐encoders  can  be  stacked  successfully  (Bengio  et  al  NIPS’2006)  to  form   highly  non-­‐linear  representa>ons,  which  with  fine-­‐tuning  overperformed   purely  supervised  MLPs     81  
82
Auto-Encoder Variants •  Discrete  inputs:  cross-­‐entropy  or  log-­‐likelihood  reconstruc>on   criterion  (similar  to  used  for  discrete  targets  for  MLPs)   •  Regularized  to  avoid  learning  the  iden>ty  everywhere:   •  Undercomplete  (eg  PCA):    bobleneck  code  smaller  than  input   •  Sparsity:  encourage  hidden  units  to  be  at  or  near  0          [Goodfellow  et  al  2009]   •  Denoising:  predict  true  input  from  corrupted  input          [Vincent  et  al  2008]   •  Contrac>ve:  force  encoder  to  have  small  deriva>ves          [Rifai  et  al  2011]   82  
83
Manifold Learning •  Addi>onal  prior:  examples  concentrate  near  a  lower   dimensional  “manifold”  (region  of  high  density  with  only  few   opera>ons  allowed  which  allow  small  changes  while  staying  on   the  manifold)   83  
84
Denoising Auto-Encoder (Vincent  et  al  2008)   •  Corrupt  the  input   •  Reconstruct  the  uncorrupted  input   Hidden code (representation) KL(reconstruction | raw input) Corrupted input Raw input reconstruction •  Encoder  &  decoder:  any  parametriza>on   •  As  good  or  beber  than  RBMs  for  unsupervised  pre-­‐training  
85
Denoising Auto-Encoder •  Learns  a  vector  field  towards  higher   probability  regions   •  Some  DAEs  correspond  to  a  kind  of   Gaussian  RBM  with  regularized  Score   Corrupted input Matching  (Vincent  2011)   •  But  with  no  par>>on  func>on,  can  measure   training  criterion   Corrupted input
86
Stacked Denoising Auto-Encoders Infinite MNIST
87
Auto-Encoders Learn Salient Variations, like a non-linear PCA •  Minimizing  reconstruc>on  error  forces  to   keep  varia>ons  along  manifold.   •  Regularizer  wants  to  throw  away  all   varia>ons.   •  With  both:  keep  ONLY  sensi>vity  to   varia>ons  ON  the  manifold.   87  
88
Contractive Auto-Encoders (Rifai,  Vincent,  Muller,  Glorot,  Bengio  ICML  2011;  Rifai,  Mesnil,   Vincent,  Bengio,  Dauphin,  Glorot  ECML  2011;  Rifai,  Dauphin,   Vincent,  Bengio,  Muller  NIPS  2011)   Most  hidden  units  saturate:   few  ac>ve  units  represent  the   ac>ve  subspace  (local  chart)   Training  ccontrac>on  in  all   wants   riterion:   cannot  afford  contrac>on  in   direc>ons   manifold  direc>ons    
89
Jacobian’s  spectrum  is  peaked  =   local  low-­‐dimensional   representa>on  /  relevant  factors   89  
90
Contractive Auto-Encoders
91
Input  Point   Tangents   MNIST   91  
92
Input  Point   Tangents   MNIST  Tangents   92  
93
Distributed vs Local (CIFAR-10 unsupervised) Input  Point   Tangents   Local  PCA   Contrac>ve  Auto-­‐Encoder   93  
94
Learned Tangent Prop: the Manifold Tangent Classifier 3  hypotheses:   1.  Semi-­‐supervised  hypothesis  (P(x)  related  to  P(y|x))     2.  Unsupervised  manifold  hypothesis  (data  concentrates  near   low-­‐dim.  manifolds)   3.  Manifold  hypothesis  for  classifica>on  (low  density  between   class  manifolds)   Algorithm:   1.  Es>mate  local  principal  direc>ons  of  varia>on  U(x)  by  CAE   (principal  singular  vectors  of  dh(x)/dx)   2.  Penalize  f(x)=P(y|x)  predictor  by  ||  df/dx  U(x)  ||  
95
Manifold Tangent Classifier Results •  Leading  singular  vectors  on  MNIST,  CIFAR-­‐10,  RCV1:   •  Knowledge-­‐free  MNIST:  0.81%  error     •  Semi-­‐sup.       •  Forest  (500k  examples)    
96
Inference and Explaining Away •  Easy  inference  in  RBMs  and  regularized  Auto-­‐Encoders   •  But  no  explaining  away  (compe>>on  between  causes)   •  (Coates  et  al  2011):  even  when  training  filters  as  RBMs  it  helps   to  perform  addi>onal  explaining  away  (e.g.  plug  them  into  a   Sparse  Coding  inference),  to  obtain  beber-­‐classifying  features   •  RBMs  would  need  lateral  connec>ons  to  achieve  similar  effect   •  Auto-­‐Encoders  would  need  to  have  lateral  recurrent   connec>ons   96  
97
Sparse Coding (Olshausen  et  al  97)   •  Directed  graphical  model:     •  One  of  the  first  unsupervised  feature  learning  algorithms  with   non-­‐linear  feature  extrac>on  (but  linear  decoder)         MAP  inference  recovers  sparse  h  although  P(h|x)  not  concentrated  at  0     •  Linear  decoder,  non-­‐parametric  encoder   •  Sparse  Coding  inference,  convex  opt.  but  expensive   97  
98
Predictive Sparse Decomposition •  Approximate  the  inference  of  sparse  coding  by   an  encoder:   Predic>ve  Sparse  Decomposi>on  (Kavukcuoglu  et  al  2008)   •  Very  successful  applica>ons  in  machine  vision   with  convolu>onal  architectures   98  
99
Predictive Sparse Decomposition •  Stacked  to  form  deep  architectures   •  Alterna>ng  convolu>on,  rec>fica>on,  pooling   •  Tiling:  no  sharing  across  overlapping  filters   •  Group  sparsity  penalty  yields  topographic   maps   99  
100
Deep Variants 100  
101
Stack of RBMs / AEs  Deep MLP •  Encoder  or  P(h|v)  becomes  MLP  layer       h3   ^   y   W3   h2   h3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   101  
102
Stack of RBMs / AEs  Deep Auto-Encoder (Hinton  &  Salakhutdinov  2006)   •  Stack  encoders  /  P(h|x)  into  deep  encoder   •  Stack  decoders  /  P(x|h)  into  deep  decoder   ^   x   ^   WT   1   h1   T   W2   h3   ^   h2   W3   WT   h2   h3   3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   102  
103
Stack of RBMs / AEs  Deep Recurrent Auto-Encoder (Savard  2011)   h3   W3   h2   •  Each  hidden  layer  receives  input  from  below  and   h2   above   W2   h1   •  Halve  the  weights     h1   •  Determinis>c  (mean-­‐field)  recurrent  computa>on   W1   x     h3   W3   T   ½W3   W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   103  
104
Stack of RBMs  Deep Belief Net (Hinton  et  al  2006)   •  Stack  lower  levels  RBMs’  P(x|h)  along  with  top-­‐level  RBM   •  P(x,  h1  ,  h2  ,  h3)  =  P(h2  ,  h3)  P(h1|h2)  P(x  |  h1)   •  Sample:  Gibbs  on  top  RBM,  propagate  down   h3   h2   h1   x   104  
105
Stack of RBMs  Deep Boltzmann Machine (Salakhutdinov  &  Hinton  AISTATS  2009)   •  Halve  the  RBM  weights  because  each  layer  now  has  inputs  from   below  and  from  above   •  Posi>ve  phase:  (mean-­‐field)  varia>onal  inference  =  recurrent  AE   •  Nega>ve  phase:  Gibbs  sampling  (stochas>c  units)   •  train  by  SML/PCD   h3   W3   T   ½W3   ½W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   105  
106
Stack of Auto-Encoders  Deep Generative Auto-Encoder (Rifai  et  al  ICML  2012)   •  MCMC  on  top-­‐level  auto-­‐encoder   •  ht+1  =  encode(decode(ht))+σ  noise   where  noise  is  Normal(0,  d/dh  encode(decode(ht)))   •  Then  determinis>cally  propagate  down  with  decoders     h3   h2   h1   x   106  
107
Sampling from a Regularized Auto-Encoder 107  
108
Sampling from a Regularized Auto-Encoder 108  
109
Sampling from a Regularized Auto-Encoder 109  
110
Sampling from a Regularized Auto-Encoder 110  
111
Sampling from a Regularized Auto-Encoder 111  
112
Part  3   Practice, Issues, Questions 112  
113
Deep Learning Tricks of the Trade •  Y.  Bengio  (2012),  “Prac>cal  Recommenda>ons  for  Gradient-­‐ Based  Training  of  Deep  Architectures”     •  Unsupervised  pre-­‐training   •  Stochas>c  gradient  descent  and  se•ng  learning  rates   •  Main  hyper-­‐parameters   •  Learning  rate  schedule   •  Early  stopping   •  Minibatches   •  Parameter  ini>aliza>on   •  Number  of  hidden  units   •  L1  and  L2  weight  decay   •  Sparsity  regulariza>on   •  Debugging   •  How  to  efficiently  search  for  hyper-­‐parameter  configura>ons   113  
114
Stochastic Gradient Descent (SGD) •  Gradient  descent  uses  total  gradient  over  all  examples  per   update,  SGD  updates  afer  only  1  or  few  examples:   •  L  =  loss  func>on,  zt  =  current  example,  θ  =  parameter  vector,  and   εt  =  learning  rate.   •  Ordinary  gradient  descent  is  a  batch  method,  very  slow,  should   never  be  used.  2nd  order  batch  method  are  being  explored  as  an   alterna>ve  but  SGD  with  selected  learning  schedule  remains  the   method  to  beat.   114  
115
Learning Rates •  Simplest  recipe:  keep  it  fixed  and  use  the  same  for  all   parameters.   •  Collobert  scales  them  by  the  inverse  of  square  root  of  the  fan-­‐in   of  each  neuron   •  Beber  results  can  generally  be  obtained  by  allowing  learning   rates  to  decrease,  typically  in  O(1/t)  because  of  theore>cal   convergence  guarantees,  e.g.,            with  hyper-­‐parameters  ε0  and  τ.   115  
116
Long-Term Dependencies and Clipping Trick •  In  very  deep  networks  such  as  recurrent  networks  (or  possibly   recursive  ones),  the  gradient  is  a  product  of  Jacobian  matrices,   each  associated  with  a  step  in  the  forward  computa>on.  This   can  become  very  small  or  very  large  quickly  [Bengio  et  al  1994],   and  the  locality  assump>on  of  gradient  descent  breaks  down.     •  The  solu>on  first  introduced  by  Mikolov    is  to  clip  gradients   to  a  maximum  value.  Makes  a  big  difference  in  Recurrent    Nets     116  
117
Early Stopping •  Beau>ful  FREE  LUNCH  (no  need  to  launch  many  different   training  runs  for  each  value  of  hyper-­‐parameter  for  #itera>ons)   •  Monitor  valida>on  error  during  training  (afer  visi>ng  #   examples  a  mul>ple  of  valida>on  set  size)   •  Keep  track  of  parameters  with  best  valida>on  error  and  report   them  at  the  end   •  If  error  does  not  improve  enough  (with  some  pa>ence),  stop.   117  
118
Parameter Initialization •  Ini>alize  hidden  layer  biases  to  0  and  output  (or  reconstruc>on)   biases  to  op>mal  value  if  weights  were  0  (e.g.  mean  target  or   inverse  sigmoid  of  mean  target).   •  Ini>alize  weights  ~  Uniform(-­‐r,r),  r  inversely  propor>onal  to  fan-­‐ in  (previous  layer  size)  and  fan-­‐out  (next  layer  size):            for  tanh  units  (and  4x  bigger  for  sigmoid  units)    (Glorot  &  Bengio  AISTATS  2010)   118  
119
Handling Large Output Spaces   •  Auto-­‐encoders  and  RBMs  reconstruct  the  input,  which  is  sparse  and  high-­‐ dimensional;  Language  models  have  huge  output  space.   code= latent features expensive cheap … …     sparse input dense output probabilities   •  (Dauphin  et  al,  ICML  2011)  Reconstruct  the  non-­‐zeros  in     the  input,  and  reconstruct  as  many  randomly  chosen   zeros,  +  importance  weights   categories   •  (Collobert  &  Weston,  ICML  2008)  sample  a  ranking  loss   •  Decompose  output  probabili>es  hierarchically  (Morin   &  Bengio  2005;  Blitzer  et  al  2005;  Mnih  &  Hinton   words  within  each  category   2007,2009;  Mikolov  et  al  2011)   119      
120
Automatic Differentiation •  The  gradient  computa>on  can  be   automa>cally  inferred  from  the  symbolic   expression  of  the  fprop.   •  Makes  it  easier  to  quickly  and  safely  try   new  models.   •  Each  node  type  needs  to  know  how  to   compute  its  output  and  how  to  compute   the  gradient  wrt  its  inputs  given  the   gradient  wrt  its  output.   •  Theano  Library  (python)  does  it   symbolically.  Other  neural  network   packages  (Torch,  Lush)  can  compute   gradients  for  any  given  run-­‐>me  value.   (Bergstra  et  al  SciPy’2010)   120  
121
Random Sampling of Hyperparameters (Bergstra  &  Bengio  2012)   •  Common  approach:  manual  +  grid  search   •  Grid  search  over  hyperparameters:  simple  &  wasteful   •  Random  search:  simple  &  efficient   •  Independently  sample  each  HP,  e.g.  l.rate~exp(U[log(.1),log(.0001)])   •  Each  training  trial  is  iid   •  If  a  HP  is  irrelevant  grid  search  is  wasteful   •  More  convenient:  ok  to  early-­‐stop,  con>nue  further,  etc.   121  
122
Issues and Questions 122  
123
Why is Unsupervised Pre-Training Working So Well? •  Regulariza>on  hypothesis:     •  Unsupervised  component  forces  model  close  to  P(x)   •  Representa>ons  good  for  P(x)  are  good  for  P(y|x)   •  Op>miza>on  hypothesis:   •  Unsupervised  ini>aliza>on  near  beber  local  minimum  of  P(y|x)   •  Can  reach  lower  local  minimum  otherwise  not  achievable  by  random  ini>aliza>on   •  Easier  to  train  each  layer  using  a  layer-­‐local  criterion   (Erhan  et  al  JMLR  2010)  
124
Learning Trajectories in Function Space •  Each  point  a  model  in   func>on  space   •  Color  =  epoch   •  Top:  trajectories  w/o   pre-­‐training   •  Each  trajectory   converges  in  different   local  min.   •  No  overlap  of  regions   with  and  w/o  pre-­‐ training  
125
Dealing with a Partition Function •  Z  =  Σx,h  e-­‐energy(x,h)   •  Intractable  for  most  interes>ng  models   •  MCMC  es>mators  of  its  gradient   •  Noisy  gradient,  can’t  reliably  cover  (spurious)  modes   •  Alterna>ves:   •  Score  matching  (Hyvarinen  2005)   •  Noise-­‐contras>ve  es>ma>on  (Gutmann  &  Hyvarinen  2010)   •  Pseudo-­‐likelihood   •  Ranking  criteria  (wsabie)  to  sample  nega>ve  examples  (Weston  et  al.   2010)   •  Auto-­‐encoders?   125  
126
Dealing with Inference •  P(h|x)  in  general  intractable  (e.g.  non-­‐RBM  Boltzmann  machine)   •  But  explaining  away  is  nice   •  Approxima>ons   •  Varia>onal  approxima>ons,  e.g.  see  Goodfellow  et  al  ICML  2012   (assume  a  unimodal  posterior)   •  MCMC,  but  certainly  not  to  convergence   •  We  would  like  a  model  where  approximate  inference  is  going  to  be  a  good   approxima>on   •  Predic>ve  Sparse  Decomposi>on  does  that   •  Learning  approx.  sparse  decoding    (Gregor  &  LeCun  ICML’2010)   •  Es>ma>ng  E[h|x]  in  a  Boltzmann  with  a  separate  network  (Salakhutdinov  &   Larochelle  AISTATS  2010)   126  
127
For gradient & inference: More difficult to mix with better trained models •  Early  during  training,  density  smeared  out,  mode  bumps  overlap   •  Later  on,  hard  to  cross  empty  voids  between  modes   127  
128
Poor Mixing: Depth to the Rescue •  Deeper  representa>ons  can  yield  some  disentangling   •  Hypotheses:     •  more  abstract/disentangled  representa>on  unfold  manifolds   and  fill  more  the  space   •  can  be  exploited  for  beber  mixing  between  modes   •  E.g.  reverse  video  bit,  class  bits  in  learned  object   representa>ons:  easy  to  Gibbs  sample  between  modes  at   Layer   abstract  level   0   1   2   Points  on  the  interpola>ng  line  between  two  classes,  at  different  levels  of  representa>on   128  
129
Poor Mixing: Depth to the Rescue •  Sampling  from  DBNs  and  stacked  Contras>ve  Auto-­‐Encoders:   1.  MCMC  sample  from  top-­‐level  singler-­‐layer  model   2.  Propagate  top-­‐level  representa>ons  to  input-­‐level  repr.   •  Visits  modes  (classes)  faster   Toronto  Face  Database   h3   h2   h1   x   129   #  classes  visited    
130
What are regularized auto-encoders learning exactly? •  Any  training  criterion  E(X,  θ)  interpretable  as  a  form  of  MAP:   •  JEPADA:  Joint  Energy  in  PArameters  and  Data    (Bengio,  Courville,  Vincent  2012)   This  Z  does  not  depend  on  θ.  If  E(X,  θ)  tractable,  so  is  the  gradient   No  magic;  consider  tradi>onal  directed  model:       Applica>on:  Predic>ve  Sparse  Decomposi>on,  regularized  auto-­‐encoders,  …     130  
131
What are regularized auto-encoders learning exactly? •  Denoising  auto-­‐encoder  is  also  contrac>ve   •  Contrac>ve/denoising  auto-­‐encoders  learn  local  moments   •  r(x)-­‐x      es>mates  the  direc>on  of  E[X|X  in  ball  around  x]   •  Jacobian                  es>mates  Cov(X|X  in  ball  around  x)   •  These  two  also  respec>vely  es>mate  the  score  and  (roughly)  the   Hessian    of  the  density   131  
132
More Open Questions •  What  is  a  good  representa>on?  Disentangling  factors?  Can  we   design  beber  training  criteria  /  setups?   •  Can  we  safely  assume  P(h|x)  to  be  unimodal  or  few-­‐modal?If   not,  is  there  any  alterna>ve  to  explicit  latent  variables?     •  Should  we  have  explicit  explaining  away  or  just  learn  to  produce   good  representa>ons?   •  Should  learned  representa>ons  be  low-­‐dimensional  or  sparse/ saturated  and  high-­‐dimensional?   •  Why  is  it  more  difficult  to  op>mize  deeper  (or  recurrent/ recursive)  architectures?  Does  it  necessarily  get  more  difficult  as   training  progresses?  Can  we  do  beber?   132  
133
The End 133  
Representa)on  Learning       Yoshua  Bengio     ICML  2012  Tutorial   June  26th  2012,  Edinburgh,  Scotland        
Outline of the Tutorial 1.  Mo>va>ons  and  Scope   1.  Feature  /  Representa>on  learning   2.  Distributed  representa>ons   3.  Exploi>ng  unlabeled  data   4.  Deep  representa>ons   5.  Mul>-­‐task  /  Transfer  learning   6.  Invariance  vs  Disentangling   2.  Algorithms   1.  Probabilis>c  models  and  RBM  variants   2.  Auto-­‐encoder  variants  (sparse,  denoising,  contrac>ve)   3.  Explaining  away,  sparse  coding  and  Predic>ve  Sparse  Decomposi>on   4.  Deep  variants   3.  Analysis,  Issues  and  Prac>ce   1.  Tips,  tricks  and  hyper-­‐parameters   2.  Par>>on  func>on  gradient   3.  Inference   4.  Mixing  between  modes   5.  Geometry  and  probabilis>c  Interpreta>ons  of  auto-­‐encoders   6.  Open  ques>ons   See  (Bengio,  Courville  &  Vincent  2012)     “Unsupervised  Feature  Learning  and  Deep  Learning:  A  Review  and  New  Perspec>ves”   And  http://www.iro.umontreal.ca/~bengioy/talks/deep-learning-tutorial-2012.html  for  a   detailed  list  of  references.  
Ultimate Goals •  AI   •  Needs  knowledge   •  Needs  learning   •  Needs  generalizing  where  probability  mass   concentrates   •  Needs  ways  to  fight  the  curse  of  dimensionality   •  Needs  disentangling  the  underlying  explanatory  factors   (“making  sense  of  the  data”)   3  
Representing data •  In  prac>ce  ML  very  sensi>ve  to  choice  of  data  representa>on   à  feature  engineering  (where  most  effort  is  spent)   à (beber)  feature  learning  (this  talk):        automa>cally  learn  good  representa>ons     •  Probabilis>c  models:   •  Good  representa>on  =  captures  posterior  distribu,on  of   underlying  explanatory  factors  of  observed  input   •  Good  features  are  useful  to  explain  varia>ons   4  
Deep Representation Learning Deep  learning  algorithms  abempt  to  learn  mul>ple  levels  of  representa>on  of   increasing  complexity/abstrac>on     When  the  number  of  levels  can  be  data-­‐ selected,  this  is  a  deep  architecture       5  
A Good Old Deep Architecture   Op>onal  Output  layer   Here  predic>ng  a  supervised  target     Hidden  layers   These  learn  more  abstract   representa>ons  as  you  head  up     Input  layer   This  has  raw  sensory  inputs  (roughly)   6  
What We Are Fighting Against: The Curse ofDimensionality      To  generalize  locally,   need  representa>ve   examples  for  all   relevant  varia>ons!     Classical  solu>on:  hope   for  a  smooth  enough   target  func>on,  or   make  it  smooth  by   handcrafing  features  
Easy Learning * * = example (x,y) * y * true unknown function * * * * * * * * * * learned function: prediction = f(x) x
Local Smoothness Prior: Locally Capture the Variations * = training example y * true function: unknown prediction learnt = interpolated f(x) * * test point x x *
Real Data Are on Highly Curved Manifolds 10  
Not Dimensionality so much as Number of Variations (Bengio, Delalleau & Le Roux 2007) •  Theorem:  Gaussian  kernel  machines  need  at  least  k  examples   to  learn  a  func>on  that  has  2k  zero-­‐crossings  along  some  line             •  Theorem:  For  a  Gaussian  kernel  machine  to  learn  some   maximally  varying  func>ons    over  d  inputs  requires  O(2d)   examples    
Is there any hope to generalize non-locally? Yes! Need more priors! 12  
Part  1   Six Good Reasons to Explore Representation Learning 13  
#1 Learning features, not just handcrafting them Most  ML  systems  use  very  carefully  hand-­‐designed   features  and  representa>ons   Many  prac>>oners  are  very  experienced  –  and  good  –  at  such   feature  design  (or  kernel  design)   In  this  world,  “machine  learning”  reduces  mostly  to  linear   models  (including  CRFs)  and  nearest-­‐neighbor-­‐like  features/ models  (including  n-­‐grams,  kernel  SVMs,  etc.)     Hand-­‐cra7ing  features  is  )me-­‐consuming,  bri<le,  incomplete   14  
How can we automatically learn good features? Claim:  to  approach  AI,  need  to  move  scope  of  ML  beyond   hand-­‐crafed  features  and  simple  models   Humans  develop  representa>ons  and  abstrac>ons  to   enable  problem-­‐solving  and  reasoning;  our  computers   should  do  the    same   Handcrafed  features  can  be  combined  with  learned   features,  or  new  more  abstract  features  learned  on  top   of  handcrafed  features   15  
#2 The need for distributed representations •  Clustering,  Nearest-­‐ Clustering   Neighbors,  RBF  SVMs,  local   non-­‐parametric  density   es>ma>on  &  predic>on,   decision  trees,  etc.   •  Parameters  for  each   dis>nguishable  region   •  #  dis>nguishable  regions   linear  in  #  parameters   16  
#2 The need for distributed representations Mul>-­‐   Clustering   •  Factor  models,  PCA,  RBMs,   Neural  Nets,  Sparse  Coding,   Deep  Learning,  etc.   •  Each  parameter  influences   many  regions,  not  just  local   neighbors   •  #  dis>nguishable  regions   grows  almost  exponen>ally   C1   C2   C3   with  #  parameters   •  GENERALIZE  NON-­‐LOCALLY   TO  NEVER-­‐SEEN  REGIONS   input   17  
#2 The need for distributed representations Mul>-­‐   Clustering   Clustering   Learning  a  set  of  features  that  are  not  mutually  exclusive   can  be  exponen>ally  more  sta>s>cally  efficient  than   nearest-­‐neighbor-­‐like  or  clustering-­‐like  models   18  
#3 Unsupervised feature learning Today,  most  prac>cal  ML  applica>ons  require  (lots  of)   labeled  training  data   But  almost  all  data  is  unlabeled   The  brain  needs  to  learn  about  1014  synap>c  strengths   …  in  about  109  seconds   Labels  cannot  possibly  provide  enough  informa>on   Most  informa>on  acquired  in  an  unsupervised  fashion   19  
#3 How do humans generalize from very few examples? •  They  transfer  knowledge  from  previous  learning:   •  Representa>ons   •  Explanatory  factors   •  Previous  learning  from:  unlabeled  data                    +  labels  for  other  tasks   •  Prior:  shared  underlying  explanatory  factors,  in   par)cular  between  P(x)  and  P(Y|x)     20    
#3 Sharing Statistical Strength by Semi-Supervised Learning •  Hypothesis:  P(x)  shares  structure  with  P(y|x)   purely   semi-­‐   supervised   supervised   21  
#4 Learning multiple levels of representation There  is  theore>cal  and  empirical  evidence  in  favor  of   mul>ple  levels  of  representa>on    Exponen)al  gain  for  some  families  of  func)ons   Biologically  inspired  learning   Brain  has  a  deep  architecture   Cortex  seems  to  have  a     generic  learning  algorithm     Humans  first  learn  simpler     concepts  and  then  compose     them  to  more  complex  ones   22    
#4 Sharing Components in a Deep Architecture Polynomial  expressed  with  shared  components:  advantage  of   depth  may  grow  exponen>ally       Sum-­‐product   network  
#4 Learning multiple levels of representation (Lee,  Largman,  Pham  &  Ng,  NIPS  2009)   (Lee,  Grosse,  Ranganath  &  Ng,  ICML  2009)     Successive  model  layers  learn  deeper  intermediate  representa>ons     High-­‐level   Layer  3   linguis>c  representa>ons   Parts  combine   to  form  objects   Layer  2   Layer  1   24   Prior:  underlying  factors  &  concepts  compactly  expressed  w/  mul)ple  levels  of  abstrac)on    
#4 Handling the compositionality of human language and thought zt-­‐1   zt   zt+1   •  Human  languages,  ideas,  and   ar>facts  are  composed  from   simpler  components   xt-­‐1   xt   xt+1   •  Recursion:  the  same   operator  (same  parameters)   is  applied  repeatedly  on   different  states/components   of  the  computa>on   •  Result  afer  unfolding  =  deep   (Bobou  2011,  Socher  et  al  2011)   representa>ons   25  
#5 Multi-Task Learning task 1 task 2 task 3 •  Generalizing  beber  to  new   output y1 output y2 output y3 tasks  is  crucial  to  approach  AI   Task  A   Task  B   Task  C   •  Deep  architectures  learn  good   intermediate  representa>ons   that  can  be  shared  across  tasks   •  Good  representa>ons  that   disentangle  underlying  factors   of  varia>on  make  sense  for   raw input x many  tasks  because  each  task   concerns  a  subset  of  the  factors   26  
#5 Sharing Statistical Strength task 1 task 2 task 3 •  Mul>ple  levels  of  latent   output y1 output y2 output y3 variables  also  allow   Task  A   Task  B   Task  C   combinatorial  sharing  of   sta>s>cal  strength:   intermediate  levels  can  also   be  seen  as  sub-­‐tasks   •  E.g.  dic>onary,  with   intermediate  concepts  re-­‐ used  across  many  defini>ons   raw input x Prior:  some  shared  underlying  explanatory  factors  between  tasks       27  
#5 Combining Multiple Sources of Evidence with Shared Representations person   url   event   •  Tradi>onal  ML:  data  =  matrix   url   words   history   •  Rela>onal  learning:  mul>ple  sources,   different  tuples  of  variables   •  Share  representa>ons  of  same  types   across  data  sources   •  Shared  learned  representa>ons  help   event   url   person   propagate  informa>on  among  data   history   words   url   sources:  e.g.,  WordNet,  XWN,   Wikipedia,  FreeBase,  ImageNet… (Bordes  et  al  AISTATS  2012)   P(person,url,event)   P(url,words,history)   28  
#5 Different object types represented in same space Google:   S.  Bengio,  J.   Weston  &  N.   Usunier   (IJCAI  2011,   NIPS’2010,   JMLR  2010,   MLJ  2010)  
#6 Invariance and Disentangling •  Invariant  features   •  Which  invariances?   •  Alterna>ve:  learning  to  disentangle  factors   •  Good  disentangling  à      avoid  the  curse  of  dimensionality   30  
#6 Emergence of Disentangling •  (Goodfellow  et  al.  2009):  sparse  auto-­‐encoders  trained   on  images     •  some  higher-­‐level  features  more  invariant  to   geometric  factors  of  varia>on     •  (Glorot  et  al.  2011):  sparse  rec>fied  denoising  auto-­‐ encoders  trained  on  bags  of  words  for  sen>ment   analysis   •  different  features  specialize  on  different  aspects   (domain,  sen>ment)   31   WHY?  
#6 Sparse Representations •  Just  add  a  penalty  on  learned  representa>on   •  Informa>on  disentangling  (compare  to  dense  compression)   •  More  likely  to  be  linearly  separable  (high-­‐dimensional  space)   •  Locally  low-­‐dimensional  representa>on  =  local  chart   •  Hi-­‐dim.  sparse  =  efficient  variable  size  representa>on                  =  data  structure   Few  bits  of  informa>on                                                        Many  bits  of  informa>on   Prior:  only  few  concepts  and  a<ributes  relevant  per  example     32  
Bypassing the curse We  need  to  build  composi>onality  into  our  ML  models     Just  as  human  languages  exploit  composi>onality  to  give   representa>ons  and  meanings  to  complex  ideas   Exploi>ng  composi>onality  gives  an  exponen>al  gain  in   representa>onal  power   Distributed  representa>ons  /  embeddings:  feature  learning   Deep  architecture:  mul>ple  levels  of  feature  learning   Prior:  composi>onality  is  useful  to  describe  the   world  around  us  efficiently   33    
Bypassing the curse by sharing statistical strength •  Besides  very  fast  GPU-­‐enabled  predictors,  the  main  advantage   of  representa>on  learning  is  sta>s>cal:  poten>al  to  learn  from   less  labeled  examples  because  of  sharing  of  sta>s>cal  strength:   •  Unsupervised  pre-­‐training  and  semi-­‐supervised  training   •  Mul>-­‐task  learning   •  Mul>-­‐data  sharing,  learning  about  symbolic  objects  and  their   rela>ons   34  
Why now? Despite  prior  inves>ga>on  and  understanding  of  many  of  the   algorithmic  techniques  …   Before  2006  training  deep  architectures  was  unsuccessful   (except  for  convolu>onal  neural  nets  when  used  by  people  who  speak  French)   What  has  changed?   •  New  methods  for  unsupervised  pre-­‐training  have  been   developed  (variants  of  Restricted  Boltzmann  Machines  =   RBMs,  regularized  autoencoders,  sparse  coding,  etc.)   •  Beber  understanding  of  these  methods   •  Successful  real-­‐world  applica>ons,  winning  challenges  and   bea>ng  SOTAs  in  various  areas   35  
Major Breakthrough in 2006 •  Ability  to  train  deep  architectures  by   using  layer-­‐wise  unsupervised   learning,  whereas  previous  purely   supervised  abempts  had  failed   •  Unsupervised  feature  learners:   •  RBMs   •  Auto-­‐encoder  variants   Bengio Montréal •  Sparse  coding  variants   Toronto Hinton Le Cun New York 36  
Unsupervised and Transfer Learning Challenge + Transfer Learning Challenge: Deep Learning 1st Place NIPS’2011   Raw  data   Transfer   Learning   1  layer   2  layers   Challenge     Paper:   ICML’2012   ICML’2011   workshop  on   Unsup.  &   Transfer  Learning   3  layers   4  layers  
More Successful Applications •  Microsof  uses  DL  for  speech  rec.  service  (audio  video  indexing),  based  on   Hinton/Toronto’s  DBNs  (Mohamed  et  al  2011)   •  Google  uses  DL  in  its  Google  Goggles  service,  using  Ng/Stanford  DL  systems   •  NYT  today  talks  about  these:  http://www.nytimes.com/2012/06/26/technology/ in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=1 •  Substan>ally  bea>ng  SOTA  in  language  modeling  (perplexity  from  140  to  102   on  Broadcast  News)  for  speech  recogni>on  (WSJ  WER  from  16.9%  to  14.4%)   (Mikolov  et  al  2011)  and  transla>on  (+1.8  BLEU)  (Schwenk  2012)   •  SENNA:  Unsup.  pre-­‐training  +  mul>-­‐task  DL  reaches  SOTA  on  POS,  NER,  SRL,   chunking,  parsing,  with  >10x  beber  speed  &  memory  (Collobert  et  al  2011)   •  Recursive  nets  surpass  SOTA  in  paraphrasing  (Socher  et  al  2011)   •  Denoising  AEs  substan>ally  beat  SOTA  in  sen>ment  analysis  (Glorot  et  al  2011)   •  Contrac>ve  AEs  SOTA  in  knowledge-­‐free  MNIST  (.8%  err)  (Rifai  et  al  NIPS  2011)   •  Le  Cun/NYU’s  stacked  PSDs  most  accurate  &  fastest  in  pedestrian  detec>on   and  DL  in  top  2  winning  entries  of  German  road  sign  recogni>on  compe>>on     38  
39  
Part  2   Representation Learning Algorithms 40  
A neural network = running several logistic regressions at the same time If  we  feed  a  vector  of  inputs  through  a  bunch  of  logis>c  regression   func>ons,  then  we  get  a  vector  of  outputs   But  we  don’t  have  to  decide   ahead  of  >me  what  variables   these  logis>c  regressions  are   trying  to  predict!   41  
A neural network = running several logistic regressions at the same time …  which  we  can  feed  into  another  logis>c  regression  func>on   and  it  is  the  training   criterion  that  will   decide  what  those   intermediate  binary   target  variables  should   be,  so  as  to  make  a   good  job  of  predic>ng   the  targets  for  the  next   layer,  etc.   42  
A neural network = running several logistic regressions at the same time •  Before  we  know  it,  we  have  a  mul>layer  neural  network….   How to do unsupervised training? 43  
PCA code= latent features h = Linear Manifold = Linear Auto-Encoder … … = Linear Gaussian Factors input reconstruction input  x,  0-­‐mean   Linear  manifold   features=code=h(x)=W  x   reconstruc>on(x)=WT  h(x)  =  WT  W  x   W  =  principal  eigen-­‐basis  of  Cov(X)   reconstruc>on(x)   reconstruc>on  error  vector   x   Probabilis>c  interpreta>ons:   1.  Gaussian  with  full   covariance  WT  W+λI   2.  Latent  marginally  iid   Gaussian  factors  h  with       x  =  WT  h  +  noise   44  
Directed Factor Models •  P(h)  factorizes  into  P(h1)  P(h2)…   h1 h2 h3 h4 h5 •  Different  priors:   •  PCA:  P(hi)  is  Gaussian   W3   W1   •  ICA:  P(hi)  is  non-­‐parametric   W5   •  Sparse  coding:  P(hi)  is  concentrated  near  0   •  Likelihood  is  typically  Gaussian  x  |  h     x1 x2          with  mean  given  by  WT  h   •  Inference  procedures  (predic>ng  h,  given  x)  differ   •  Sparse  h:  x  is  explained  by  the  weighted  addi>on  of  selected   filters  hi   x   W1   W3   W5   h1   h3   h5                               =  .9  x                        +  .8  x                      +  .7  x   45  
Stacking Single-Layer Learners •  PCA  is  great  but  can’t  be  stacked  into  deeper  more  abstract   representa>ons  (linear  x  linear  =  linear)   •  One  of  the  big  ideas  from  Hinton  et  al.  2006:  layer-­‐wise   unsupervised  feature  learning   Stacking Restricted Boltzmann Machines (RBM) à Deep Belief Network (DBN) 46  
Effective deep learning became possible through unsupervised pre-training [Erhan  et  al.,  JMLR  2010]   (with  RBMs  and  Denoising  Auto-­‐Encoders)   Purely  supervised  neural  net   With  unsupervised  pre-­‐training   47  
Layer-wise Unsupervised Learning input … 48  
Layer-Wise Unsupervised Pre-training features … input … 49  
Layer-Wise Unsupervised Pre-training ? reconstruction … input = … of input features … input … 50  
Layer-Wise Unsupervised Pre-training features … input … 51  
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 52  
Layer-Wise Unsupervised Pre-training Layer-wise Unsupervised Learning ? reconstruction … … = of features More abstract … features features … input … 53  
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 54  
Layer-wise Unsupervised Learning Even more abstract features … More abstract … features features … input … 55  
Supervised Fine-Tuning Output Target f(X) six ? = Y two! Even more abstract features … More abstract … features features … input … •  Addi>onal  hypothesis:  features  good  for  P(x)  good  for  P(y|x)   56  
Restricted Boltzmann Machines 57  
Undirected Models: the Restricted Boltzmann Machine [Hinton  et  al  2006]   •  Probabilis>c  model  of  the  joint  distribu>on  of   h1 h2 h3 the  observed  variables  (inputs  alone  or  inputs   and  targets)  x   •  Latent  (hidden)  variables  h  model  high-­‐order   dependencies   •  Inference  is  easy,  P(h|x)  factorizes   x1 x2 •  See  Bengio  (2009)  detailed  monograph/review:        “Learning  Deep  Architectures  for  AI”.   •  See  Hinton  (2010)            “A  prac,cal  guide  to  training  Restricted  Boltzmann  Machines”  
Boltzmann Machines & MRFs •  Boltzmann  machines:        (Hinton  84)     •  Markov  Random  Fields:     Sof  constraint  /  probabilis>c  statement           ¡  More    nteres>ng  with  latent  variables!   i                                                                                                                                                                                                                                                                                                                                                                            
Restricted Boltzmann Machine (RBM) •  A  popular  building   block  for  deep   architectures   hidden   •  Bipar)te  undirected   graphical  model   observed
Gibbs Sampling in RBMs h1 ~ P(h|x1) h2 ~ P(h|x2) h3 ~ P(h|x3) x1 x2 ~ P(x|h1) x3 ~ P(x|h2) ¡  Easy inference P(h|x)  and  P(x|h)  factorize   ¡  Efficient block Gibbs P(h|x)=  Π  P(hi|x)   sampling xàhàxàh… i  
Problems with Gibbs Sampling In  prac>ce,  Gibbs  sampling  does  not  always  mix  well…   RBM trained by CD on MNIST Chains from random state Chains from real digits (Desjardins  et  al  2010)  
RBM with (image, label) visible units hidden h U W image y 0 0 1 0 x label y (Larochelle  &  Bengio  2008)  
RBMs are Universal Approximators (Le Roux & Bengio 2008) •  Adding  one  hidden  unit  (with  proper  choice  of  parameters)   guarantees  increasing  likelihood     •  With  enough  hidden  units,  can  perfectly  model  any  discrete   distribu>on   •  RBMs  with  variable  #  of  hidden  units  =  non-­‐parametric  
RBM Conditionals Factorize
RBM Energy Gives Binomial Neurons
RBM Free Energy •  Free  Energy  =  equivalent  energy  when  marginalizing       •  Can  be  computed  exactly  and  efficiently  in  RBMs     •  Marginal  likelihood  P(x)  tractable  up  to  par>>on  func>on  Z  
Factorization of the Free Energy Let  the  energy  have  the  following  general  form:   Then  
Energy-Based Models Gradient
Boltzmann Machine Gradient •  Gradient  has  two  components:   positive phase negative phase ¡  In  RBMs,  easy  to  sample  or  sum  over  h|x   ¡  Difficult  part:  sampling  from  P(x),  typically  with  a  Markov  chain  
Positive & Negative Samples •  Observed (+) examples push the energy down •  Generated / dream / fantasy (-) samples / particles push the energy up X+ X- Equilibrium:  E[gradient]  =  0  
Training RBMs Contras>ve  Divergence:    start  nega>ve  Gibbs  chain  at  observed  x,  run  k   (CD-­‐k)   Gibbs  steps     SML/Persistent  CD:   run  nega>ve  Gibbs  chain  in  background  while   (PCD)    weights  slowly  change   Fast  PCD:   two  sets  of  weights,  one  with  a  large  learning  rate   only  used  for  nega>ve  phase,  quickly  exploring   modes   Herding:   Determinis>c  near-­‐chaos  dynamical  system  defines   both  learning  and  sampling   Tempered  MCMC:   use  higher  temperature  to  escape  modes  
Contrastive Divergence Contrastive Divergence (CD-k): start negative phase block Gibbs chain at observed x, run k Gibbs steps (Hinton 2002) h+ ~ P(h|x+) h-~ P(h|x-) Observed x+ k = 2 steps Sampled x- positive phase negative phase push down Free Energy x+ x- push up
Persistent CD (PCD) / Stochastic Max. Likelihood (SML) Run  nega>ve  Gibbs  chain  in  background  while  weights  slowly   change  (Younes  1999,  Tieleman  2008):   •    Guarantees  (Younes  1999;  Yuille  2005)     •  If  learning  rate  decreases  in  1/t,          chain  mixes  before  parameters  change  too  much,          chain  stays  converged  when  parameters  change   h+ ~ P(h|x+) previous x- Observed x+ new x- (positive phase)
PCD/SML + large learning rate Nega>ve  phase  samples  quickly  push  up  the  energy  of   wherever  they  are  and  quickly  move  to  another  mode   push FreeEnergy down x+ x- push up
Some RBM Variants •  Different  energy  func>ons  and  allowed                   values  for  the  hidden  and  visible  units:   •  Hinton  et  al  2006:  binary-­‐binary  RBMs   •  Welling  NIPS’2004:  exponen>al  family  units   •  Ranzato  &  Hinton  CVPR’2010:  Gaussian  RBM  weaknesses  (no   condi>onal  covariance),  propose  mcRBM   •  Ranzato  et  al  NIPS’2010:  mPoT,  similar  energy  func>on   •  Courville  et  al  ICML’2011:  spike-­‐and-­‐slab  RBM     76  
Convolutionally Trained Spike & Slab RBMs Samples
Training  examples   Generated  samples   ssRBM is not Cheating
Auto-Encoders & Variants 79  
Auto-Encoders  code=  latent  features   •  MLP  whose  target  output  =  input   •  Reconstruc>on=decoder(encoder(input)),            e  ncoder                      decoder   e.g.    input   …   …      reconstruc>on   •  Probable  inputs  have  small  reconstruc>on  error   because  training  criterion  digs  holes  at  examples   •  With  bobleneck,  code  =  new  coordinate  system   •  Encoder  and  decoder  can  have  1  or  more  layers   •  Training  deep  auto-­‐encoders  notoriously  difficult   80    
Stacking Auto-Encoders Auto-­‐encoders  can  be  stacked  successfully  (Bengio  et  al  NIPS’2006)  to  form   highly  non-­‐linear  representa>ons,  which  with  fine-­‐tuning  overperformed   purely  supervised  MLPs     81  
Auto-Encoder Variants •  Discrete  inputs:  cross-­‐entropy  or  log-­‐likelihood  reconstruc>on   criterion  (similar  to  used  for  discrete  targets  for  MLPs)   •  Regularized  to  avoid  learning  the  iden>ty  everywhere:   •  Undercomplete  (eg  PCA):    bobleneck  code  smaller  than  input   •  Sparsity:  encourage  hidden  units  to  be  at  or  near  0          [Goodfellow  et  al  2009]   •  Denoising:  predict  true  input  from  corrupted  input          [Vincent  et  al  2008]   •  Contrac>ve:  force  encoder  to  have  small  deriva>ves          [Rifai  et  al  2011]   82  
Manifold Learning •  Addi>onal  prior:  examples  concentrate  near  a  lower   dimensional  “manifold”  (region  of  high  density  with  only  few   opera>ons  allowed  which  allow  small  changes  while  staying  on   the  manifold)   83  
Denoising Auto-Encoder (Vincent  et  al  2008)   •  Corrupt  the  input   •  Reconstruct  the  uncorrupted  input   Hidden code (representation) KL(reconstruction | raw input) Corrupted input Raw input reconstruction •  Encoder  &  decoder:  any  parametriza>on   •  As  good  or  beber  than  RBMs  for  unsupervised  pre-­‐training  
Denoising Auto-Encoder •  Learns  a  vector  field  towards  higher   probability  regions   •  Some  DAEs  correspond  to  a  kind  of   Gaussian  RBM  with  regularized  Score   Corrupted input Matching  (Vincent  2011)   •  But  with  no  par>>on  func>on,  can  measure   training  criterion   Corrupted input
Stacked Denoising Auto-Encoders Infinite MNIST
Auto-Encoders Learn Salient Variations, like a non-linear PCA •  Minimizing  reconstruc>on  error  forces  to   keep  varia>ons  along  manifold.   •  Regularizer  wants  to  throw  away  all   varia>ons.   •  With  both:  keep  ONLY  sensi>vity  to   varia>ons  ON  the  manifold.   87  
Contractive Auto-Encoders (Rifai,  Vincent,  Muller,  Glorot,  Bengio  ICML  2011;  Rifai,  Mesnil,   Vincent,  Bengio,  Dauphin,  Glorot  ECML  2011;  Rifai,  Dauphin,   Vincent,  Bengio,  Muller  NIPS  2011)   Most  hidden  units  saturate:   few  ac>ve  units  represent  the   ac>ve  subspace  (local  chart)   Training  ccontrac>on  in  all   wants   riterion:   cannot  afford  contrac>on  in   direc>ons   manifold  direc>ons    
Jacobian’s  spectrum  is  peaked  =   local  low-­‐dimensional   representa>on  /  relevant  factors   89  
Contractive Auto-Encoders
Input  Point   Tangents   MNIST   91  
Input  Point   Tangents   MNIST  Tangents   92  
Distributed vs Local (CIFAR-10 unsupervised) Input  Point   Tangents   Local  PCA   Contrac>ve  Auto-­‐Encoder   93  
Learned Tangent Prop: the Manifold Tangent Classifier 3  hypotheses:   1.  Semi-­‐supervised  hypothesis  (P(x)  related  to  P(y|x))     2.  Unsupervised  manifold  hypothesis  (data  concentrates  near   low-­‐dim.  manifolds)   3.  Manifold  hypothesis  for  classifica>on  (low  density  between   class  manifolds)   Algorithm:   1.  Es>mate  local  principal  direc>ons  of  varia>on  U(x)  by  CAE   (principal  singular  vectors  of  dh(x)/dx)   2.  Penalize  f(x)=P(y|x)  predictor  by  ||  df/dx  U(x)  ||  
Manifold Tangent Classifier Results •  Leading  singular  vectors  on  MNIST,  CIFAR-­‐10,  RCV1:   •  Knowledge-­‐free  MNIST:  0.81%  error     •  Semi-­‐sup.       •  Forest  (500k  examples)    
Inference and Explaining Away •  Easy  inference  in  RBMs  and  regularized  Auto-­‐Encoders   •  But  no  explaining  away  (compe>>on  between  causes)   •  (Coates  et  al  2011):  even  when  training  filters  as  RBMs  it  helps   to  perform  addi>onal  explaining  away  (e.g.  plug  them  into  a   Sparse  Coding  inference),  to  obtain  beber-­‐classifying  features   •  RBMs  would  need  lateral  connec>ons  to  achieve  similar  effect   •  Auto-­‐Encoders  would  need  to  have  lateral  recurrent   connec>ons   96  
Sparse Coding (Olshausen  et  al  97)   •  Directed  graphical  model:     •  One  of  the  first  unsupervised  feature  learning  algorithms  with   non-­‐linear  feature  extrac>on  (but  linear  decoder)         MAP  inference  recovers  sparse  h  although  P(h|x)  not  concentrated  at  0     •  Linear  decoder,  non-­‐parametric  encoder   •  Sparse  Coding  inference,  convex  opt.  but  expensive   97  
Predictive Sparse Decomposition •  Approximate  the  inference  of  sparse  coding  by   an  encoder:   Predic>ve  Sparse  Decomposi>on  (Kavukcuoglu  et  al  2008)   •  Very  successful  applica>ons  in  machine  vision   with  convolu>onal  architectures   98  
Predictive Sparse Decomposition •  Stacked  to  form  deep  architectures   •  Alterna>ng  convolu>on,  rec>fica>on,  pooling   •  Tiling:  no  sharing  across  overlapping  filters   •  Group  sparsity  penalty  yields  topographic   maps   99  
Deep Variants 100  
Stack of RBMs / AEs  Deep MLP •  Encoder  or  P(h|v)  becomes  MLP  layer       h3   ^   y   W3   h2   h3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   101  
Stack of RBMs / AEs  Deep Auto-Encoder (Hinton  &  Salakhutdinov  2006)   •  Stack  encoders  /  P(h|x)  into  deep  encoder   •  Stack  decoders  /  P(x|h)  into  deep  decoder   ^   x   ^   WT   1   h1   T   W2   h3   ^   h2   W3   WT   h2   h3   3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   102  
Stack of RBMs / AEs  Deep Recurrent Auto-Encoder (Savard  2011)   h3   W3   h2   •  Each  hidden  layer  receives  input  from  below  and   h2   above   W2   h1   •  Halve  the  weights     h1   •  Determinis>c  (mean-­‐field)  recurrent  computa>on   W1   x     h3   W3   T   ½W3   W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   103  
Stack of RBMs  Deep Belief Net (Hinton  et  al  2006)   •  Stack  lower  levels  RBMs’  P(x|h)  along  with  top-­‐level  RBM   •  P(x,  h1  ,  h2  ,  h3)  =  P(h2  ,  h3)  P(h1|h2)  P(x  |  h1)   •  Sample:  Gibbs  on  top  RBM,  propagate  down   h3   h2   h1   x   104  
Stack of RBMs  Deep Boltzmann Machine (Salakhutdinov  &  Hinton  AISTATS  2009)   •  Halve  the  RBM  weights  because  each  layer  now  has  inputs  from   below  and  from  above   •  Posi>ve  phase:  (mean-­‐field)  varia>onal  inference  =  recurrent  AE   •  Nega>ve  phase:  Gibbs  sampling  (stochas>c  units)   •  train  by  SML/PCD   h3   W3   T   ½W3   ½W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   105  
Stack of Auto-Encoders  Deep Generative Auto-Encoder (Rifai  et  al  ICML  2012)   •  MCMC  on  top-­‐level  auto-­‐encoder   •  ht+1  =  encode(decode(ht))+σ  noise   where  noise  is  Normal(0,  d/dh  encode(decode(ht)))   •  Then  determinis>cally  propagate  down  with  decoders     h3   h2   h1   x   106  
Sampling from a Regularized Auto-Encoder 107  
Sampling from a Regularized Auto-Encoder 108  
Sampling from a Regularized Auto-Encoder 109  
Sampling from a Regularized Auto-Encoder 110  
Sampling from a Regularized Auto-Encoder 111  
Part  3   Practice, Issues, Questions 112  
Deep Learning Tricks of the Trade •  Y.  Bengio  (2012),  “Prac>cal  Recommenda>ons  for  Gradient-­‐ Based  Training  of  Deep  Architectures”     •  Unsupervised  pre-­‐training   •  Stochas>c  gradient  descent  and  se•ng  learning  rates   •  Main  hyper-­‐parameters   •  Learning  rate  schedule   •  Early  stopping   •  Minibatches   •  Parameter  ini>aliza>on   •  Number  of  hidden  units   •  L1  and  L2  weight  decay   •  Sparsity  regulariza>on   •  Debugging   •  How  to  efficiently  search  for  hyper-­‐parameter  configura>ons   113  
Stochastic Gradient Descent (SGD) •  Gradient  descent  uses  total  gradient  over  all  examples  per   update,  SGD  updates  afer  only  1  or  few  examples:   •  L  =  loss  func>on,  zt  =  current  example,  θ  =  parameter  vector,  and   εt  =  learning  rate.   •  Ordinary  gradient  descent  is  a  batch  method,  very  slow,  should   never  be  used.  2nd  order  batch  method  are  being  explored  as  an   alterna>ve  but  SGD  with  selected  learning  schedule  remains  the   method  to  beat.   114  
Learning Rates •  Simplest  recipe:  keep  it  fixed  and  use  the  same  for  all   parameters.   •  Collobert  scales  them  by  the  inverse  of  square  root  of  the  fan-­‐in   of  each  neuron   •  Beber  results  can  generally  be  obtained  by  allowing  learning   rates  to  decrease,  typically  in  O(1/t)  because  of  theore>cal   convergence  guarantees,  e.g.,            with  hyper-­‐parameters  ε0  and  τ.   115  
Long-Term Dependencies and Clipping Trick •  In  very  deep  networks  such  as  recurrent  networks  (or  possibly   recursive  ones),  the  gradient  is  a  product  of  Jacobian  matrices,   each  associated  with  a  step  in  the  forward  computa>on.  This   can  become  very  small  or  very  large  quickly  [Bengio  et  al  1994],   and  the  locality  assump>on  of  gradient  descent  breaks  down.     •  The  solu>on  first  introduced  by  Mikolov    is  to  clip  gradients   to  a  maximum  value.  Makes  a  big  difference  in  Recurrent    Nets     116  
Early Stopping •  Beau>ful  FREE  LUNCH  (no  need  to  launch  many  different   training  runs  for  each  value  of  hyper-­‐parameter  for  #itera>ons)   •  Monitor  valida>on  error  during  training  (afer  visi>ng  #   examples  a  mul>ple  of  valida>on  set  size)   •  Keep  track  of  parameters  with  best  valida>on  error  and  report   them  at  the  end   •  If  error  does  not  improve  enough  (with  some  pa>ence),  stop.   117  
Parameter Initialization •  Ini>alize  hidden  layer  biases  to  0  and  output  (or  reconstruc>on)   biases  to  op>mal  value  if  weights  were  0  (e.g.  mean  target  or   inverse  sigmoid  of  mean  target).   •  Ini>alize  weights  ~  Uniform(-­‐r,r),  r  inversely  propor>onal  to  fan-­‐ in  (previous  layer  size)  and  fan-­‐out  (next  layer  size):            for  tanh  units  (and  4x  bigger  for  sigmoid  units)    (Glorot  &  Bengio  AISTATS  2010)   118  
Handling Large Output Spaces   •  Auto-­‐encoders  and  RBMs  reconstruct  the  input,  which  is  sparse  and  high-­‐ dimensional;  Language  models  have  huge  output  space.   code= latent features expensive cheap … …     sparse input dense output probabilities   •  (Dauphin  et  al,  ICML  2011)  Reconstruct  the  non-­‐zeros  in     the  input,  and  reconstruct  as  many  randomly  chosen   zeros,  +  importance  weights   categories   •  (Collobert  &  Weston,  ICML  2008)  sample  a  ranking  loss   •  Decompose  output  probabili>es  hierarchically  (Morin   &  Bengio  2005;  Blitzer  et  al  2005;  Mnih  &  Hinton   words  within  each  category   2007,2009;  Mikolov  et  al  2011)   119      
Automatic Differentiation •  The  gradient  computa>on  can  be   automa>cally  inferred  from  the  symbolic   expression  of  the  fprop.   •  Makes  it  easier  to  quickly  and  safely  try   new  models.   •  Each  node  type  needs  to  know  how  to   compute  its  output  and  how  to  compute   the  gradient  wrt  its  inputs  given  the   gradient  wrt  its  output.   •  Theano  Library  (python)  does  it   symbolically.  Other  neural  network   packages  (Torch,  Lush)  can  compute   gradients  for  any  given  run-­‐>me  value.   (Bergstra  et  al  SciPy’2010)   120  
Random Sampling of Hyperparameters (Bergstra  &  Bengio  2012)   •  Common  approach:  manual  +  grid  search   •  Grid  search  over  hyperparameters:  simple  &  wasteful   •  Random  search:  simple  &  efficient   •  Independently  sample  each  HP,  e.g.  l.rate~exp(U[log(.1),log(.0001)])   •  Each  training  trial  is  iid   •  If  a  HP  is  irrelevant  grid  search  is  wasteful   •  More  convenient:  ok  to  early-­‐stop,  con>nue  further,  etc.   121  
Issues and Questions 122  
Why is Unsupervised Pre-Training Working So Well? •  Regulariza>on  hypothesis:     •  Unsupervised  component  forces  model  close  to  P(x)   •  Representa>ons  good  for  P(x)  are  good  for  P(y|x)   •  Op>miza>on  hypothesis:   •  Unsupervised  ini>aliza>on  near  beber  local  minimum  of  P(y|x)   •  Can  reach  lower  local  minimum  otherwise  not  achievable  by  random  ini>aliza>on   •  Easier  to  train  each  layer  using  a  layer-­‐local  criterion   (Erhan  et  al  JMLR  2010)  
Learning Trajectories in Function Space •  Each  point  a  model  in   func>on  space   •  Color  =  epoch   •  Top:  trajectories  w/o   pre-­‐training   •  Each  trajectory   converges  in  different   local  min.   •  No  overlap  of  regions   with  and  w/o  pre-­‐ training  
Dealing with a Partition Function •  Z  =  Σx,h  e-­‐energy(x,h)   •  Intractable  for  most  interes>ng  models   •  MCMC  es>mators  of  its  gradient   •  Noisy  gradient,  can’t  reliably  cover  (spurious)  modes   •  Alterna>ves:   •  Score  matching  (Hyvarinen  2005)   •  Noise-­‐contras>ve  es>ma>on  (Gutmann  &  Hyvarinen  2010)   •  Pseudo-­‐likelihood   •  Ranking  criteria  (wsabie)  to  sample  nega>ve  examples  (Weston  et  al.   2010)   •  Auto-­‐encoders?   125  
Dealing with Inference •  P(h|x)  in  general  intractable  (e.g.  non-­‐RBM  Boltzmann  machine)   •  But  explaining  away  is  nice   •  Approxima>ons   •  Varia>onal  approxima>ons,  e.g.  see  Goodfellow  et  al  ICML  2012   (assume  a  unimodal  posterior)   •  MCMC,  but  certainly  not  to  convergence   •  We  would  like  a  model  where  approximate  inference  is  going  to  be  a  good   approxima>on   •  Predic>ve  Sparse  Decomposi>on  does  that   •  Learning  approx.  sparse  decoding    (Gregor  &  LeCun  ICML’2010)   •  Es>ma>ng  E[h|x]  in  a  Boltzmann  with  a  separate  network  (Salakhutdinov  &   Larochelle  AISTATS  2010)   126  
For gradient & inference: More difficult to mix with better trained models •  Early  during  training,  density  smeared  out,  mode  bumps  overlap   •  Later  on,  hard  to  cross  empty  voids  between  modes   127  
Poor Mixing: Depth to the Rescue •  Deeper  representa>ons  can  yield  some  disentangling   •  Hypotheses:     •  more  abstract/disentangled  representa>on  unfold  manifolds   and  fill  more  the  space   •  can  be  exploited  for  beber  mixing  between  modes   •  E.g.  reverse  video  bit,  class  bits  in  learned  object   representa>ons:  easy  to  Gibbs  sample  between  modes  at   Layer   abstract  level   0   1   2   Points  on  the  interpola>ng  line  between  two  classes,  at  different  levels  of  representa>on   128  
Poor Mixing: Depth to the Rescue •  Sampling  from  DBNs  and  stacked  Contras>ve  Auto-­‐Encoders:   1.  MCMC  sample  from  top-­‐level  singler-­‐layer  model   2.  Propagate  top-­‐level  representa>ons  to  input-­‐level  repr.   •  Visits  modes  (classes)  faster   Toronto  Face  Database   h3   h2   h1   x   129   #  classes  visited    
What are regularized auto-encoders learning exactly? •  Any  training  criterion  E(X,  θ)  interpretable  as  a  form  of  MAP:   •  JEPADA:  Joint  Energy  in  PArameters  and  Data    (Bengio,  Courville,  Vincent  2012)   This  Z  does  not  depend  on  θ.  If  E(X,  θ)  tractable,  so  is  the  gradient   No  magic;  consider  tradi>onal  directed  model:       Applica>on:  Predic>ve  Sparse  Decomposi>on,  regularized  auto-­‐encoders,  …     130  
What are regularized auto-encoders learning exactly? •  Denoising  auto-­‐encoder  is  also  contrac>ve   •  Contrac>ve/denoising  auto-­‐encoders  learn  local  moments   •  r(x)-­‐x      es>mates  the  direc>on  of  E[X|X  in  ball  around  x]   •  Jacobian                  es>mates  Cov(X|X  in  ball  around  x)   •  These  two  also  respec>vely  es>mate  the  score  and  (roughly)  the   Hessian    of  the  density   131  
More Open Questions •  What  is  a  good  representa>on?  Disentangling  factors?  Can  we   design  beber  training  criteria  /  setups?   •  Can  we  safely  assume  P(h|x)  to  be  unimodal  or  few-­‐modal?If   not,  is  there  any  alterna>ve  to  explicit  latent  variables?     •  Should  we  have  explicit  explaining  away  or  just  learn  to  produce   good  representa>ons?   •  Should  learned  representa>ons  be  low-­‐dimensional  or  sparse/ saturated  and  high-­‐dimensional?   •  Why  is  it  more  difficult  to  op>mize  deeper  (or  recurrent/ recursive)  architectures?  Does  it  necessarily  get  more  difficult  as   training  progresses?  Can  we  do  beber?   132  
The End 133  
Ad

Recommended

Semi-Supervised Autoencoders for Predicting Sentiment Distributions（第 5 回 De...
Semi-Supervised Autoencoders for Predicting Sentiment Distributions（第 5 回 De... Semi-Supervised Autoencoders for Predicting Sentiment Distributions（第 5 回 De...
Semi-Supervised Autoencoders for Predicting Sentiment Distributions（第 5 回 De...
Ohsawa Goodfellow
 
Lecture8 - From CBR to IBk
Lecture8 - From CBR to IBkLecture8 - From CBR to IBk
Lecture8 - From CBR to IBk
Albert Orriols-Puig
 
Lecture12 - SVM
Lecture12 - SVMLecture12 - SVM
Lecture12 - SVM
Albert Orriols-Puig
 
Intro to Deep learning - Autoencoders
Intro to Deep learning - Autoencoders Intro to Deep learning - Autoencoders
Intro to Deep learning - Autoencoders
Akash Goel
 
P03 neural networks cvpr2012 deep learning methods for vision
P03 neural networks cvpr2012 deep learning methods for visionP03 neural networks cvpr2012 deep learning methods for vision
P03 neural networks cvpr2012 deep learning methods for vision
zukun
 
Transfer Learning: An overview
Transfer Learning: An overviewTransfer Learning: An overview
Transfer Learning: An overview
jins0618
 
2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learning2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learning
Van Thanh
 
P04 restricted boltzmann machines cvpr2012 deep learning methods for vision
P04 restricted boltzmann machines cvpr2012 deep learning methods for visionP04 restricted boltzmann machines cvpr2012 deep learning methods for vision
P04 restricted boltzmann machines cvpr2012 deep learning methods for vision
zukun
 
Fcv learn yu
Fcv learn yuFcv learn yu
Fcv learn yu
zukun
 
A Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware DetectionA Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware Detection
IJCSIS Research Publications
 
Practical deepllearningv1
Practical deepllearningv1Practical deepllearningv1
Practical deepllearningv1
arthi v
 
Lecture17
Lecture17Lecture17
Lecture17
Albert Orriols-Puig
 
Lecture19
Lecture19Lecture19
Lecture19
Albert Orriols-Puig
 
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
Universitat Politècnica de Catalunya
 
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning NetworksLooking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Dinesh V
 
Machine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative ModelsMachine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative Models
butest
 
A survey on transfer learning
A survey on transfer learningA survey on transfer learning
A survey on transfer learning
azuring
 
Lecture1 - Machine Learning
Lecture1 - Machine LearningLecture1 - Machine Learning
Lecture1 - Machine Learning
Albert Orriols-Puig
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
IJERD Editor
 
Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)
준식 최
 
[ppt]
[ppt][ppt]
[ppt]
butest
 
Unsupervised Feature Learning
Unsupervised Feature LearningUnsupervised Feature Learning
Unsupervised Feature Learning
Amgad Muhammad
 
Data Redundacy
Data RedundacyData Redundacy
Data Redundacy
Poonam Seth
 
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursiveC users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
rokiah64
 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant features
zukun
 
From_seq2seq_to_BERT
From_seq2seq_to_BERTFrom_seq2seq_to_BERT
From_seq2seq_to_BERT
Huali Zhao
 
Simulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter DaelemansSimulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter Daelemans
butest
 
Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!
Roelof Pieters
 
Machine Learning and Inductive Inference
Machine Learning and Inductive InferenceMachine Learning and Inductive Inference
Machine Learning and Inductive Inference
butest
 
A Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine LearningA Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine Learning
nep_test_account
 

More Related Content

What's hot (20)

Fcv learn yu
Fcv learn yuFcv learn yu
Fcv learn yu
zukun
 
A Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware DetectionA Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware Detection
IJCSIS Research Publications
 
Practical deepllearningv1
Practical deepllearningv1Practical deepllearningv1
Practical deepllearningv1
arthi v
 
Lecture17
Lecture17Lecture17
Lecture17
Albert Orriols-Puig
 
Lecture19
Lecture19Lecture19
Lecture19
Albert Orriols-Puig
 
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
Universitat Politècnica de Catalunya
 
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning NetworksLooking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Dinesh V
 
Machine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative ModelsMachine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative Models
butest
 
A survey on transfer learning
A survey on transfer learningA survey on transfer learning
A survey on transfer learning
azuring
 
Lecture1 - Machine Learning
Lecture1 - Machine LearningLecture1 - Machine Learning
Lecture1 - Machine Learning
Albert Orriols-Puig
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
IJERD Editor
 
Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)
준식 최
 
[ppt]
[ppt][ppt]
[ppt]
butest
 
Unsupervised Feature Learning
Unsupervised Feature LearningUnsupervised Feature Learning
Unsupervised Feature Learning
Amgad Muhammad
 
Data Redundacy
Data RedundacyData Redundacy
Data Redundacy
Poonam Seth
 
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursiveC users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
rokiah64
 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant features
zukun
 
From_seq2seq_to_BERT
From_seq2seq_to_BERTFrom_seq2seq_to_BERT
From_seq2seq_to_BERT
Huali Zhao
 
Simulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter DaelemansSimulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter Daelemans
butest
 
Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!
Roelof Pieters
 
Fcv learn yu
Fcv learn yuFcv learn yu
Fcv learn yu
zukun
 
A Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware DetectionA Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware Detection
IJCSIS Research Publications
 
Practical deepllearningv1
Practical deepllearningv1Practical deepllearningv1
Practical deepllearningv1
arthi v
 
Lecture17
Lecture17Lecture17
Lecture17
Albert Orriols-Puig
 
Lecture19
Lecture19Lecture19
Lecture19
Albert Orriols-Puig
 
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
Universitat Politècnica de Catalunya
 
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning NetworksLooking into the Black Box - A Theoretical Insight into Deep Learning Networks
Looking into the Black Box - A Theoretical Insight into Deep Learning Networks
Dinesh V
 
Machine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative ModelsMachine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative Models
butest
 
A survey on transfer learning
A survey on transfer learningA survey on transfer learning
A survey on transfer learning
azuring
 
Lecture1 - Machine Learning
Lecture1 - Machine LearningLecture1 - Machine Learning
Lecture1 - Machine Learning
Albert Orriols-Puig
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
IJERD Editor
 
Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)Paper Summary of Disentangling by Factorising (Factor-VAE)
Paper Summary of Disentangling by Factorising (Factor-VAE)
준식 최
 
[ppt]
[ppt][ppt]
[ppt]
butest
 
Unsupervised Feature Learning
Unsupervised Feature LearningUnsupervised Feature Learning
Unsupervised Feature Learning
Amgad Muhammad
 
Data Redundacy
Data RedundacyData Redundacy
Data Redundacy
Poonam Seth
 
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursiveC users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
C users_mpk7_app_data_local_temp_plugtmp_plugin-week3recursive
rokiah64
 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant features
zukun
 
From_seq2seq_to_BERT
From_seq2seq_to_BERTFrom_seq2seq_to_BERT
From_seq2seq_to_BERT
Huali Zhao
 
Simulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter DaelemansSimulation of Language Acquisition Walter Daelemans
Simulation of Language Acquisition Walter Daelemans
butest
 
Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!
Roelof Pieters
 

Similar to Icml2012 tutorial representation_learning (20)

Machine Learning and Inductive Inference
Machine Learning and Inductive InferenceMachine Learning and Inductive Inference
Machine Learning and Inductive Inference
butest
 
A Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine LearningA Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine Learning
nep_test_account
 
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkkOBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
shesnasuneer
 
Easy to learn deep learning guide - elementry
Easy to learn deep learning guide - elementryEasy to learn deep learning guide - elementry
Easy to learn deep learning guide - elementry
AnjaliSohoni
 
Buku panduan untuk Machine Learning.pdf
Buku panduan untuk Machine Learning.pdfBuku panduan untuk Machine Learning.pdf
Buku panduan untuk Machine Learning.pdf
jrmstudent
 
Feedforward Networks and Deep Learning Module-02.pdf
Feedforward Networks and Deep Learning Module-02.pdfFeedforward Networks and Deep Learning Module-02.pdf
Feedforward Networks and Deep Learning Module-02.pdf
roopashreesv
 
CS229_MachineLearning_notes.pdfkkkkkkkkkk
CS229_MachineLearning_notes.pdfkkkkkkkkkkCS229_MachineLearning_notes.pdfkkkkkkkkkk
CS229_MachineLearning_notes.pdfkkkkkkkkkk
lenhan070903
 
machine learning notes by Andrew Ng and Tengyu Ma
machine learning notes by Andrew Ng and Tengyu Mamachine learning notes by Andrew Ng and Tengyu Ma
machine learning notes by Andrew Ng and Tengyu Ma
Vijayabaskar Uthirapathy
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Unexpected Challenges in Large Scale Machine Learning by Charles Parker
Unexpected Challenges in Large Scale Machine Learning by Charles Parker Unexpected Challenges in Large Scale Machine Learning by Charles Parker
Unexpected Challenges in Large Scale Machine Learning by Charles Parker
BigMine
 
Tsinghua invited talk_zhou_xing_v2r0
Tsinghua invited talk_zhou_xing_v2r0Tsinghua invited talk_zhou_xing_v2r0
Tsinghua invited talk_zhou_xing_v2r0
Joe Xing
 
lec1.ppt
lec1.pptlec1.ppt
lec1.ppt
SVasuKrishna1
 
Lecture 1
Lecture 1Lecture 1
Lecture 1
Aun Akbar
 
Chapter10.pptx
Chapter10.pptxChapter10.pptx
Chapter10.pptx
adnansbp
 
Machine learning and its parameter is discussed here
Machine learning and its parameter is discussed hereMachine learning and its parameter is discussed here
Machine learning and its parameter is discussed here
RevathiSundar4
 
Deep Learning: Towards General Artificial Intelligence
Deep Learning: Towards General Artificial IntelligenceDeep Learning: Towards General Artificial Intelligence
Deep Learning: Towards General Artificial Intelligence
Rukshan Batuwita
 
notes as .ppt
notes as .pptnotes as .ppt
notes as .ppt
butest
 
Machine learning
Machine learningMachine learning
Machine learning
Digvijay Singh
 
Machine Learning and Inductive Inference
Machine Learning and Inductive InferenceMachine Learning and Inductive Inference
Machine Learning and Inductive Inference
butest
 
A Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine LearningA Few Useful Things to Know about Machine Learning
A Few Useful Things to Know about Machine Learning
nep_test_account
 
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkkOBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
OBJECTRECOGNITION1.pptxjjjkkkkjjjjkkkkkkk
shesnasuneer
 
Easy to learn deep learning guide - elementry
Easy to learn deep learning guide - elementryEasy to learn deep learning guide - elementry
Easy to learn deep learning guide - elementry
AnjaliSohoni
 
Buku panduan untuk Machine Learning.pdf
Buku panduan untuk Machine Learning.pdfBuku panduan untuk Machine Learning.pdf
Buku panduan untuk Machine Learning.pdf
jrmstudent
 
Feedforward Networks and Deep Learning Module-02.pdf
Feedforward Networks and Deep Learning Module-02.pdfFeedforward Networks and Deep Learning Module-02.pdf
Feedforward Networks and Deep Learning Module-02.pdf
roopashreesv
 
CS229_MachineLearning_notes.pdfkkkkkkkkkk
CS229_MachineLearning_notes.pdfkkkkkkkkkkCS229_MachineLearning_notes.pdfkkkkkkkkkk
CS229_MachineLearning_notes.pdfkkkkkkkkkk
lenhan070903
 
machine learning notes by Andrew Ng and Tengyu Ma
machine learning notes by Andrew Ng and Tengyu Mamachine learning notes by Andrew Ng and Tengyu Ma
machine learning notes by Andrew Ng and Tengyu Ma
Vijayabaskar Uthirapathy
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Introduction
IntroductionIntroduction
Introduction
butest
 
Unexpected Challenges in Large Scale Machine Learning by Charles Parker
Unexpected Challenges in Large Scale Machine Learning by Charles Parker Unexpected Challenges in Large Scale Machine Learning by Charles Parker
Unexpected Challenges in Large Scale Machine Learning by Charles Parker
BigMine
 
Tsinghua invited talk_zhou_xing_v2r0
Tsinghua invited talk_zhou_xing_v2r0Tsinghua invited talk_zhou_xing_v2r0
Tsinghua invited talk_zhou_xing_v2r0
Joe Xing
 
lec1.ppt
lec1.pptlec1.ppt
lec1.ppt
SVasuKrishna1
 
Lecture 1
Lecture 1Lecture 1
Lecture 1
Aun Akbar
 
Chapter10.pptx
Chapter10.pptxChapter10.pptx
Chapter10.pptx
adnansbp
 
Machine learning and its parameter is discussed here
Machine learning and its parameter is discussed hereMachine learning and its parameter is discussed here
Machine learning and its parameter is discussed here
RevathiSundar4
 
Deep Learning: Towards General Artificial Intelligence
Deep Learning: Towards General Artificial IntelligenceDeep Learning: Towards General Artificial Intelligence
Deep Learning: Towards General Artificial Intelligence
Rukshan Batuwita
 
notes as .ppt
notes as .pptnotes as .ppt
notes as .ppt
butest
 
Machine learning
Machine learningMachine learning
Machine learning
Digvijay Singh
 
Ad

More from zukun (20)

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
zukun
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
zukun
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
zukun
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
zukun
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
zukun
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
zukun
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
zukun
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-software
zukun
 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptors
zukun
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
zukun
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
zukun
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
zukun
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
zukun
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
zukun
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
zukun
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
zukun
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
zukun
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
zukun
 
My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
zukun
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
zukun
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
zukun
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
zukun
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
zukun
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
zukun
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
zukun
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-software
zukun
 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptors
zukun
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
zukun
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
zukun
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
zukun
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
zukun
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
zukun
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
zukun
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
zukun
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
zukun
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
zukun
 
Ad

Icml2012 tutorial representation_learning

  • 1. Representa)on  Learning       Yoshua  Bengio     ICML  2012  Tutorial   June  26th  2012,  Edinburgh,  Scotland        
  • 2. Outline of the Tutorial 1.  Mo>va>ons  and  Scope   1.  Feature  /  Representa>on  learning   2.  Distributed  representa>ons   3.  Exploi>ng  unlabeled  data   4.  Deep  representa>ons   5.  Mul>-­‐task  /  Transfer  learning   6.  Invariance  vs  Disentangling   2.  Algorithms   1.  Probabilis>c  models  and  RBM  variants   2.  Auto-­‐encoder  variants  (sparse,  denoising,  contrac>ve)   3.  Explaining  away,  sparse  coding  and  Predic>ve  Sparse  Decomposi>on   4.  Deep  variants   3.  Analysis,  Issues  and  Prac>ce   1.  Tips,  tricks  and  hyper-­‐parameters   2.  Par>>on  func>on  gradient   3.  Inference   4.  Mixing  between  modes   5.  Geometry  and  probabilis>c  Interpreta>ons  of  auto-­‐encoders   6.  Open  ques>ons   See  (Bengio,  Courville  &  Vincent  2012)     “Unsupervised  Feature  Learning  and  Deep  Learning:  A  Review  and  New  Perspec>ves”   And  http://www.iro.umontreal.ca/~bengioy/talks/deep-learning-tutorial-2012.html  for  a   detailed  list  of  references.  
  • 3. Ultimate Goals •  AI   •  Needs  knowledge   •  Needs  learning   •  Needs  generalizing  where  probability  mass   concentrates   •  Needs  ways  to  fight  the  curse  of  dimensionality   •  Needs  disentangling  the  underlying  explanatory  factors   (“making  sense  of  the  data”)   3  
  • 4. Representing data •  In  prac>ce  ML  very  sensi>ve  to  choice  of  data  representa>on   à  feature  engineering  (where  most  effort  is  spent)   à (beber)  feature  learning  (this  talk):        automa>cally  learn  good  representa>ons     •  Probabilis>c  models:   •  Good  representa>on  =  captures  posterior  distribu,on  of   underlying  explanatory  factors  of  observed  input   •  Good  features  are  useful  to  explain  varia>ons   4  
  • 5. Deep Representation Learning Deep  learning  algorithms  abempt  to  learn  mul>ple  levels  of  representa>on  of   increasing  complexity/abstrac>on     When  the  number  of  levels  can  be  data-­‐ selected,  this  is  a  deep  architecture       5  
  • 6. A Good Old Deep Architecture   Op>onal  Output  layer   Here  predic>ng  a  supervised  target     Hidden  layers   These  learn  more  abstract   representa>ons  as  you  head  up     Input  layer   This  has  raw  sensory  inputs  (roughly)   6  
  • 7. What We Are Fighting Against: The Curse ofDimensionality      To  generalize  locally,   need  representa>ve   examples  for  all   relevant  varia>ons!     Classical  solu>on:  hope   for  a  smooth  enough   target  func>on,  or   make  it  smooth  by   handcrafing  features  
  • 8. Easy Learning * * = example (x,y) * y * true unknown function * * * * * * * * * * learned function: prediction = f(x) x
  • 9. Local Smoothness Prior: Locally Capture the Variations * = training example y * true function: unknown prediction learnt = interpolated f(x) * * test point x x *
  • 10. Real Data Are on Highly Curved Manifolds 10  
  • 11. Not Dimensionality so much as Number of Variations (Bengio, Delalleau & Le Roux 2007) •  Theorem:  Gaussian  kernel  machines  need  at  least  k  examples   to  learn  a  func>on  that  has  2k  zero-­‐crossings  along  some  line             •  Theorem:  For  a  Gaussian  kernel  machine  to  learn  some   maximally  varying  func>ons    over  d  inputs  requires  O(2d)   examples    
  • 12. Is there any hope to generalize non-locally? Yes! Need more priors! 12  
  • 13. Part  1   Six Good Reasons to Explore Representation Learning 13  
  • 14. #1 Learning features, not just handcrafting them Most  ML  systems  use  very  carefully  hand-­‐designed   features  and  representa>ons   Many  prac>>oners  are  very  experienced  –  and  good  –  at  such   feature  design  (or  kernel  design)   In  this  world,  “machine  learning”  reduces  mostly  to  linear   models  (including  CRFs)  and  nearest-­‐neighbor-­‐like  features/ models  (including  n-­‐grams,  kernel  SVMs,  etc.)     Hand-­‐cra7ing  features  is  )me-­‐consuming,  bri<le,  incomplete   14  
  • 15. How can we automatically learn good features? Claim:  to  approach  AI,  need  to  move  scope  of  ML  beyond   hand-­‐crafed  features  and  simple  models   Humans  develop  representa>ons  and  abstrac>ons  to   enable  problem-­‐solving  and  reasoning;  our  computers   should  do  the    same   Handcrafed  features  can  be  combined  with  learned   features,  or  new  more  abstract  features  learned  on  top   of  handcrafed  features   15  
  • 16. #2 The need for distributed representations •  Clustering,  Nearest-­‐ Clustering   Neighbors,  RBF  SVMs,  local   non-­‐parametric  density   es>ma>on  &  predic>on,   decision  trees,  etc.   •  Parameters  for  each   dis>nguishable  region   •  #  dis>nguishable  regions   linear  in  #  parameters   16  
  • 17. #2 The need for distributed representations Mul>-­‐   Clustering   •  Factor  models,  PCA,  RBMs,   Neural  Nets,  Sparse  Coding,   Deep  Learning,  etc.   •  Each  parameter  influences   many  regions,  not  just  local   neighbors   •  #  dis>nguishable  regions   grows  almost  exponen>ally   C1   C2   C3   with  #  parameters   •  GENERALIZE  NON-­‐LOCALLY   TO  NEVER-­‐SEEN  REGIONS   input   17  
  • 18. #2 The need for distributed representations Mul>-­‐   Clustering   Clustering   Learning  a  set  of  features  that  are  not  mutually  exclusive   can  be  exponen>ally  more  sta>s>cally  efficient  than   nearest-­‐neighbor-­‐like  or  clustering-­‐like  models   18  
  • 19. #3 Unsupervised feature learning Today,  most  prac>cal  ML  applica>ons  require  (lots  of)   labeled  training  data   But  almost  all  data  is  unlabeled   The  brain  needs  to  learn  about  1014  synap>c  strengths   …  in  about  109  seconds   Labels  cannot  possibly  provide  enough  informa>on   Most  informa>on  acquired  in  an  unsupervised  fashion   19  
  • 20. #3 How do humans generalize from very few examples? •  They  transfer  knowledge  from  previous  learning:   •  Representa>ons   •  Explanatory  factors   •  Previous  learning  from:  unlabeled  data                    +  labels  for  other  tasks   •  Prior:  shared  underlying  explanatory  factors,  in   par)cular  between  P(x)  and  P(Y|x)     20    
  • 21. #3 Sharing Statistical Strength by Semi-Supervised Learning •  Hypothesis:  P(x)  shares  structure  with  P(y|x)   purely   semi-­‐   supervised   supervised   21  
  • 22. #4 Learning multiple levels of representation There  is  theore>cal  and  empirical  evidence  in  favor  of   mul>ple  levels  of  representa>on    Exponen)al  gain  for  some  families  of  func)ons   Biologically  inspired  learning   Brain  has  a  deep  architecture   Cortex  seems  to  have  a     generic  learning  algorithm     Humans  first  learn  simpler     concepts  and  then  compose     them  to  more  complex  ones   22    
  • 23. #4 Sharing Components in a Deep Architecture Polynomial  expressed  with  shared  components:  advantage  of   depth  may  grow  exponen>ally       Sum-­‐product   network  
  • 24. #4 Learning multiple levels of representation (Lee,  Largman,  Pham  &  Ng,  NIPS  2009)   (Lee,  Grosse,  Ranganath  &  Ng,  ICML  2009)     Successive  model  layers  learn  deeper  intermediate  representa>ons     High-­‐level   Layer  3   linguis>c  representa>ons   Parts  combine   to  form  objects   Layer  2   Layer  1   24   Prior:  underlying  factors  &  concepts  compactly  expressed  w/  mul)ple  levels  of  abstrac)on    
  • 25. #4 Handling the compositionality of human language and thought zt-­‐1   zt   zt+1   •  Human  languages,  ideas,  and   ar>facts  are  composed  from   simpler  components   xt-­‐1   xt   xt+1   •  Recursion:  the  same   operator  (same  parameters)   is  applied  repeatedly  on   different  states/components   of  the  computa>on   •  Result  afer  unfolding  =  deep   (Bobou  2011,  Socher  et  al  2011)   representa>ons   25  
  • 26. #5 Multi-Task Learning task 1 task 2 task 3 •  Generalizing  beber  to  new   output y1 output y2 output y3 tasks  is  crucial  to  approach  AI   Task  A   Task  B   Task  C   •  Deep  architectures  learn  good   intermediate  representa>ons   that  can  be  shared  across  tasks   •  Good  representa>ons  that   disentangle  underlying  factors   of  varia>on  make  sense  for   raw input x many  tasks  because  each  task   concerns  a  subset  of  the  factors   26  
  • 27. #5 Sharing Statistical Strength task 1 task 2 task 3 •  Mul>ple  levels  of  latent   output y1 output y2 output y3 variables  also  allow   Task  A   Task  B   Task  C   combinatorial  sharing  of   sta>s>cal  strength:   intermediate  levels  can  also   be  seen  as  sub-­‐tasks   •  E.g.  dic>onary,  with   intermediate  concepts  re-­‐ used  across  many  defini>ons   raw input x Prior:  some  shared  underlying  explanatory  factors  between  tasks       27  
  • 28. #5 Combining Multiple Sources of Evidence with Shared Representations person   url   event   •  Tradi>onal  ML:  data  =  matrix   url   words   history   •  Rela>onal  learning:  mul>ple  sources,   different  tuples  of  variables   •  Share  representa>ons  of  same  types   across  data  sources   •  Shared  learned  representa>ons  help   event   url   person   propagate  informa>on  among  data   history   words   url   sources:  e.g.,  WordNet,  XWN,   Wikipedia,  FreeBase,  ImageNet… (Bordes  et  al  AISTATS  2012)   P(person,url,event)   P(url,words,history)   28  
  • 29. #5 Different object types represented in same space Google:   S.  Bengio,  J.   Weston  &  N.   Usunier   (IJCAI  2011,   NIPS’2010,   JMLR  2010,   MLJ  2010)  
  • 30. #6 Invariance and Disentangling •  Invariant  features   •  Which  invariances?   •  Alterna>ve:  learning  to  disentangle  factors   •  Good  disentangling  à      avoid  the  curse  of  dimensionality   30  
  • 31. #6 Emergence of Disentangling •  (Goodfellow  et  al.  2009):  sparse  auto-­‐encoders  trained   on  images     •  some  higher-­‐level  features  more  invariant  to   geometric  factors  of  varia>on     •  (Glorot  et  al.  2011):  sparse  rec>fied  denoising  auto-­‐ encoders  trained  on  bags  of  words  for  sen>ment   analysis   •  different  features  specialize  on  different  aspects   (domain,  sen>ment)   31   WHY?  
  • 32. #6 Sparse Representations •  Just  add  a  penalty  on  learned  representa>on   •  Informa>on  disentangling  (compare  to  dense  compression)   •  More  likely  to  be  linearly  separable  (high-­‐dimensional  space)   •  Locally  low-­‐dimensional  representa>on  =  local  chart   •  Hi-­‐dim.  sparse  =  efficient  variable  size  representa>on                  =  data  structure   Few  bits  of  informa>on                                                        Many  bits  of  informa>on   Prior:  only  few  concepts  and  a<ributes  relevant  per  example     32  
  • 33. Bypassing the curse We  need  to  build  composi>onality  into  our  ML  models     Just  as  human  languages  exploit  composi>onality  to  give   representa>ons  and  meanings  to  complex  ideas   Exploi>ng  composi>onality  gives  an  exponen>al  gain  in   representa>onal  power   Distributed  representa>ons  /  embeddings:  feature  learning   Deep  architecture:  mul>ple  levels  of  feature  learning   Prior:  composi>onality  is  useful  to  describe  the   world  around  us  efficiently   33    
  • 34. Bypassing the curse by sharing statistical strength •  Besides  very  fast  GPU-­‐enabled  predictors,  the  main  advantage   of  representa>on  learning  is  sta>s>cal:  poten>al  to  learn  from   less  labeled  examples  because  of  sharing  of  sta>s>cal  strength:   •  Unsupervised  pre-­‐training  and  semi-­‐supervised  training   •  Mul>-­‐task  learning   •  Mul>-­‐data  sharing,  learning  about  symbolic  objects  and  their   rela>ons   34  
  • 35. Why now? Despite  prior  inves>ga>on  and  understanding  of  many  of  the   algorithmic  techniques  …   Before  2006  training  deep  architectures  was  unsuccessful   (except  for  convolu>onal  neural  nets  when  used  by  people  who  speak  French)   What  has  changed?   •  New  methods  for  unsupervised  pre-­‐training  have  been   developed  (variants  of  Restricted  Boltzmann  Machines  =   RBMs,  regularized  autoencoders,  sparse  coding,  etc.)   •  Beber  understanding  of  these  methods   •  Successful  real-­‐world  applica>ons,  winning  challenges  and   bea>ng  SOTAs  in  various  areas   35  
  • 36. Major Breakthrough in 2006 •  Ability  to  train  deep  architectures  by   using  layer-­‐wise  unsupervised   learning,  whereas  previous  purely   supervised  abempts  had  failed   •  Unsupervised  feature  learners:   •  RBMs   •  Auto-­‐encoder  variants   Bengio Montréal •  Sparse  coding  variants   Toronto Hinton Le Cun New York 36  
  • 37. Unsupervised and Transfer Learning Challenge + Transfer Learning Challenge: Deep Learning 1st Place NIPS’2011   Raw  data   Transfer   Learning   1  layer   2  layers   Challenge     Paper:   ICML’2012   ICML’2011   workshop  on   Unsup.  &   Transfer  Learning   3  layers   4  layers  
  • 38. More Successful Applications •  Microsof  uses  DL  for  speech  rec.  service  (audio  video  indexing),  based  on   Hinton/Toronto’s  DBNs  (Mohamed  et  al  2011)   •  Google  uses  DL  in  its  Google  Goggles  service,  using  Ng/Stanford  DL  systems   •  NYT  today  talks  about  these:  http://www.nytimes.com/2012/06/26/technology/ in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=1 •  Substan>ally  bea>ng  SOTA  in  language  modeling  (perplexity  from  140  to  102   on  Broadcast  News)  for  speech  recogni>on  (WSJ  WER  from  16.9%  to  14.4%)   (Mikolov  et  al  2011)  and  transla>on  (+1.8  BLEU)  (Schwenk  2012)   •  SENNA:  Unsup.  pre-­‐training  +  mul>-­‐task  DL  reaches  SOTA  on  POS,  NER,  SRL,   chunking,  parsing,  with  >10x  beber  speed  &  memory  (Collobert  et  al  2011)   •  Recursive  nets  surpass  SOTA  in  paraphrasing  (Socher  et  al  2011)   •  Denoising  AEs  substan>ally  beat  SOTA  in  sen>ment  analysis  (Glorot  et  al  2011)   •  Contrac>ve  AEs  SOTA  in  knowledge-­‐free  MNIST  (.8%  err)  (Rifai  et  al  NIPS  2011)   •  Le  Cun/NYU’s  stacked  PSDs  most  accurate  &  fastest  in  pedestrian  detec>on   and  DL  in  top  2  winning  entries  of  German  road  sign  recogni>on  compe>>on     38  
  • 39. 39  
  • 40. Part  2   Representation Learning Algorithms 40  
  • 41. A neural network = running several logistic regressions at the same time If  we  feed  a  vector  of  inputs  through  a  bunch  of  logis>c  regression   func>ons,  then  we  get  a  vector  of  outputs   But  we  don’t  have  to  decide   ahead  of  >me  what  variables   these  logis>c  regressions  are   trying  to  predict!   41  
  • 42. A neural network = running several logistic regressions at the same time …  which  we  can  feed  into  another  logis>c  regression  func>on   and  it  is  the  training   criterion  that  will   decide  what  those   intermediate  binary   target  variables  should   be,  so  as  to  make  a   good  job  of  predic>ng   the  targets  for  the  next   layer,  etc.   42  
  • 43. A neural network = running several logistic regressions at the same time •  Before  we  know  it,  we  have  a  mul>layer  neural  network….   How to do unsupervised training? 43  
  • 44. PCA code= latent features h = Linear Manifold = Linear Auto-Encoder … … = Linear Gaussian Factors input reconstruction input  x,  0-­‐mean   Linear  manifold   features=code=h(x)=W  x   reconstruc>on(x)=WT  h(x)  =  WT  W  x   W  =  principal  eigen-­‐basis  of  Cov(X)   reconstruc>on(x)   reconstruc>on  error  vector   x   Probabilis>c  interpreta>ons:   1.  Gaussian  with  full   covariance  WT  W+λI   2.  Latent  marginally  iid   Gaussian  factors  h  with       x  =  WT  h  +  noise   44  
  • 45. Directed Factor Models •  P(h)  factorizes  into  P(h1)  P(h2)…   h1 h2 h3 h4 h5 •  Different  priors:   •  PCA:  P(hi)  is  Gaussian   W3   W1   •  ICA:  P(hi)  is  non-­‐parametric   W5   •  Sparse  coding:  P(hi)  is  concentrated  near  0   •  Likelihood  is  typically  Gaussian  x  |  h     x1 x2          with  mean  given  by  WT  h   •  Inference  procedures  (predic>ng  h,  given  x)  differ   •  Sparse  h:  x  is  explained  by  the  weighted  addi>on  of  selected   filters  hi   x   W1   W3   W5   h1   h3   h5                               =  .9  x                        +  .8  x                      +  .7  x   45  
  • 46. Stacking Single-Layer Learners •  PCA  is  great  but  can’t  be  stacked  into  deeper  more  abstract   representa>ons  (linear  x  linear  =  linear)   •  One  of  the  big  ideas  from  Hinton  et  al.  2006:  layer-­‐wise   unsupervised  feature  learning   Stacking Restricted Boltzmann Machines (RBM) à Deep Belief Network (DBN) 46  
  • 47. Effective deep learning became possible through unsupervised pre-training [Erhan  et  al.,  JMLR  2010]   (with  RBMs  and  Denoising  Auto-­‐Encoders)   Purely  supervised  neural  net   With  unsupervised  pre-­‐training   47  
  • 49. Layer-Wise Unsupervised Pre-training features … input … 49  
  • 50. Layer-Wise Unsupervised Pre-training ? reconstruction … input = … of input features … input … 50  
  • 51. Layer-Wise Unsupervised Pre-training features … input … 51  
  • 52. Layer-Wise Unsupervised Pre-training More abstract … features features … input … 52  
  • 53. Layer-Wise Unsupervised Pre-training Layer-wise Unsupervised Learning ? reconstruction … … = of features More abstract … features features … input … 53  
  • 54. Layer-Wise Unsupervised Pre-training More abstract … features features … input … 54  
  • 55. Layer-wise Unsupervised Learning Even more abstract features … More abstract … features features … input … 55  
  • 56. Supervised Fine-Tuning Output Target f(X) six ? = Y two! Even more abstract features … More abstract … features features … input … •  Addi>onal  hypothesis:  features  good  for  P(x)  good  for  P(y|x)   56  
  • 58. Undirected Models: the Restricted Boltzmann Machine [Hinton  et  al  2006]   •  Probabilis>c  model  of  the  joint  distribu>on  of   h1 h2 h3 the  observed  variables  (inputs  alone  or  inputs   and  targets)  x   •  Latent  (hidden)  variables  h  model  high-­‐order   dependencies   •  Inference  is  easy,  P(h|x)  factorizes   x1 x2 •  See  Bengio  (2009)  detailed  monograph/review:        “Learning  Deep  Architectures  for  AI”.   •  See  Hinton  (2010)            “A  prac,cal  guide  to  training  Restricted  Boltzmann  Machines”  
  • 59. Boltzmann Machines & MRFs •  Boltzmann  machines:        (Hinton  84)     •  Markov  Random  Fields:     Sof  constraint  /  probabilis>c  statement           ¡  More    nteres>ng  with  latent  variables!   i                                                                                                                                                                                                                                                                                                                                                                            
  • 60. Restricted Boltzmann Machine (RBM) •  A  popular  building   block  for  deep   architectures   hidden   •  Bipar)te  undirected   graphical  model   observed
  • 61. Gibbs Sampling in RBMs h1 ~ P(h|x1) h2 ~ P(h|x2) h3 ~ P(h|x3) x1 x2 ~ P(x|h1) x3 ~ P(x|h2) ¡  Easy inference P(h|x)  and  P(x|h)  factorize   ¡  Efficient block Gibbs P(h|x)=  Π  P(hi|x)   sampling xàhàxàh… i  
  • 62. Problems with Gibbs Sampling In  prac>ce,  Gibbs  sampling  does  not  always  mix  well…   RBM trained by CD on MNIST Chains from random state Chains from real digits (Desjardins  et  al  2010)  
  • 63. RBM with (image, label) visible units hidden h U W image y 0 0 1 0 x label y (Larochelle  &  Bengio  2008)  
  • 64. RBMs are Universal Approximators (Le Roux & Bengio 2008) •  Adding  one  hidden  unit  (with  proper  choice  of  parameters)   guarantees  increasing  likelihood     •  With  enough  hidden  units,  can  perfectly  model  any  discrete   distribu>on   •  RBMs  with  variable  #  of  hidden  units  =  non-­‐parametric  
  • 66. RBM Energy Gives Binomial Neurons
  • 67. RBM Free Energy •  Free  Energy  =  equivalent  energy  when  marginalizing       •  Can  be  computed  exactly  and  efficiently  in  RBMs     •  Marginal  likelihood  P(x)  tractable  up  to  par>>on  func>on  Z  
  • 68. Factorization of the Free Energy Let  the  energy  have  the  following  general  form:   Then  
  • 70. Boltzmann Machine Gradient •  Gradient  has  two  components:   positive phase negative phase ¡  In  RBMs,  easy  to  sample  or  sum  over  h|x   ¡  Difficult  part:  sampling  from  P(x),  typically  with  a  Markov  chain  
  • 71. Positive & Negative Samples •  Observed (+) examples push the energy down •  Generated / dream / fantasy (-) samples / particles push the energy up X+ X- Equilibrium:  E[gradient]  =  0  
  • 72. Training RBMs Contras>ve  Divergence:    start  nega>ve  Gibbs  chain  at  observed  x,  run  k   (CD-­‐k)   Gibbs  steps     SML/Persistent  CD:   run  nega>ve  Gibbs  chain  in  background  while   (PCD)    weights  slowly  change   Fast  PCD:   two  sets  of  weights,  one  with  a  large  learning  rate   only  used  for  nega>ve  phase,  quickly  exploring   modes   Herding:   Determinis>c  near-­‐chaos  dynamical  system  defines   both  learning  and  sampling   Tempered  MCMC:   use  higher  temperature  to  escape  modes  
  • 73. Contrastive Divergence Contrastive Divergence (CD-k): start negative phase block Gibbs chain at observed x, run k Gibbs steps (Hinton 2002) h+ ~ P(h|x+) h-~ P(h|x-) Observed x+ k = 2 steps Sampled x- positive phase negative phase push down Free Energy x+ x- push up
  • 74. Persistent CD (PCD) / Stochastic Max. Likelihood (SML) Run  nega>ve  Gibbs  chain  in  background  while  weights  slowly   change  (Younes  1999,  Tieleman  2008):   •    Guarantees  (Younes  1999;  Yuille  2005)     •  If  learning  rate  decreases  in  1/t,          chain  mixes  before  parameters  change  too  much,          chain  stays  converged  when  parameters  change   h+ ~ P(h|x+) previous x- Observed x+ new x- (positive phase)
  • 75. PCD/SML + large learning rate Nega>ve  phase  samples  quickly  push  up  the  energy  of   wherever  they  are  and  quickly  move  to  another  mode   push FreeEnergy down x+ x- push up
  • 76. Some RBM Variants •  Different  energy  func>ons  and  allowed                   values  for  the  hidden  and  visible  units:   •  Hinton  et  al  2006:  binary-­‐binary  RBMs   •  Welling  NIPS’2004:  exponen>al  family  units   •  Ranzato  &  Hinton  CVPR’2010:  Gaussian  RBM  weaknesses  (no   condi>onal  covariance),  propose  mcRBM   •  Ranzato  et  al  NIPS’2010:  mPoT,  similar  energy  func>on   •  Courville  et  al  ICML’2011:  spike-­‐and-­‐slab  RBM     76  
  • 77. Convolutionally Trained Spike & Slab RBMs Samples
  • 78. Training  examples   Generated  samples   ssRBM is not Cheating
  • 80. Auto-Encoders  code=  latent  features   •  MLP  whose  target  output  =  input   •  Reconstruc>on=decoder(encoder(input)),            e  ncoder                      decoder   e.g.    input   …   …      reconstruc>on   •  Probable  inputs  have  small  reconstruc>on  error   because  training  criterion  digs  holes  at  examples   •  With  bobleneck,  code  =  new  coordinate  system   •  Encoder  and  decoder  can  have  1  or  more  layers   •  Training  deep  auto-­‐encoders  notoriously  difficult   80    
  • 81. Stacking Auto-Encoders Auto-­‐encoders  can  be  stacked  successfully  (Bengio  et  al  NIPS’2006)  to  form   highly  non-­‐linear  representa>ons,  which  with  fine-­‐tuning  overperformed   purely  supervised  MLPs     81  
  • 82. Auto-Encoder Variants •  Discrete  inputs:  cross-­‐entropy  or  log-­‐likelihood  reconstruc>on   criterion  (similar  to  used  for  discrete  targets  for  MLPs)   •  Regularized  to  avoid  learning  the  iden>ty  everywhere:   •  Undercomplete  (eg  PCA):    bobleneck  code  smaller  than  input   •  Sparsity:  encourage  hidden  units  to  be  at  or  near  0          [Goodfellow  et  al  2009]   •  Denoising:  predict  true  input  from  corrupted  input          [Vincent  et  al  2008]   •  Contrac>ve:  force  encoder  to  have  small  deriva>ves          [Rifai  et  al  2011]   82  
  • 83. Manifold Learning •  Addi>onal  prior:  examples  concentrate  near  a  lower   dimensional  “manifold”  (region  of  high  density  with  only  few   opera>ons  allowed  which  allow  small  changes  while  staying  on   the  manifold)   83  
  • 84. Denoising Auto-Encoder (Vincent  et  al  2008)   •  Corrupt  the  input   •  Reconstruct  the  uncorrupted  input   Hidden code (representation) KL(reconstruction | raw input) Corrupted input Raw input reconstruction •  Encoder  &  decoder:  any  parametriza>on   •  As  good  or  beber  than  RBMs  for  unsupervised  pre-­‐training  
  • 85. Denoising Auto-Encoder •  Learns  a  vector  field  towards  higher   probability  regions   •  Some  DAEs  correspond  to  a  kind  of   Gaussian  RBM  with  regularized  Score   Corrupted input Matching  (Vincent  2011)   •  But  with  no  par>>on  func>on,  can  measure   training  criterion   Corrupted input
  • 87. Auto-Encoders Learn Salient Variations, like a non-linear PCA •  Minimizing  reconstruc>on  error  forces  to   keep  varia>ons  along  manifold.   •  Regularizer  wants  to  throw  away  all   varia>ons.   •  With  both:  keep  ONLY  sensi>vity  to   varia>ons  ON  the  manifold.   87  
  • 88. Contractive Auto-Encoders (Rifai,  Vincent,  Muller,  Glorot,  Bengio  ICML  2011;  Rifai,  Mesnil,   Vincent,  Bengio,  Dauphin,  Glorot  ECML  2011;  Rifai,  Dauphin,   Vincent,  Bengio,  Muller  NIPS  2011)   Most  hidden  units  saturate:   few  ac>ve  units  represent  the   ac>ve  subspace  (local  chart)   Training  ccontrac>on  in  all   wants   riterion:   cannot  afford  contrac>on  in   direc>ons   manifold  direc>ons    
  • 89. Jacobian’s  spectrum  is  peaked  =   local  low-­‐dimensional   representa>on  /  relevant  factors   89  
  • 91. Input  Point   Tangents   MNIST   91  
  • 92. Input  Point   Tangents   MNIST  Tangents   92  
  • 93. Distributed vs Local (CIFAR-10 unsupervised) Input  Point   Tangents   Local  PCA   Contrac>ve  Auto-­‐Encoder   93  
  • 94. Learned Tangent Prop: the Manifold Tangent Classifier 3  hypotheses:   1.  Semi-­‐supervised  hypothesis  (P(x)  related  to  P(y|x))     2.  Unsupervised  manifold  hypothesis  (data  concentrates  near   low-­‐dim.  manifolds)   3.  Manifold  hypothesis  for  classifica>on  (low  density  between   class  manifolds)   Algorithm:   1.  Es>mate  local  principal  direc>ons  of  varia>on  U(x)  by  CAE   (principal  singular  vectors  of  dh(x)/dx)   2.  Penalize  f(x)=P(y|x)  predictor  by  ||  df/dx  U(x)  ||  
  • 95. Manifold Tangent Classifier Results •  Leading  singular  vectors  on  MNIST,  CIFAR-­‐10,  RCV1:   •  Knowledge-­‐free  MNIST:  0.81%  error     •  Semi-­‐sup.       •  Forest  (500k  examples)    
  • 96. Inference and Explaining Away •  Easy  inference  in  RBMs  and  regularized  Auto-­‐Encoders   •  But  no  explaining  away  (compe>>on  between  causes)   •  (Coates  et  al  2011):  even  when  training  filters  as  RBMs  it  helps   to  perform  addi>onal  explaining  away  (e.g.  plug  them  into  a   Sparse  Coding  inference),  to  obtain  beber-­‐classifying  features   •  RBMs  would  need  lateral  connec>ons  to  achieve  similar  effect   •  Auto-­‐Encoders  would  need  to  have  lateral  recurrent   connec>ons   96  
  • 97. Sparse Coding (Olshausen  et  al  97)   •  Directed  graphical  model:     •  One  of  the  first  unsupervised  feature  learning  algorithms  with   non-­‐linear  feature  extrac>on  (but  linear  decoder)         MAP  inference  recovers  sparse  h  although  P(h|x)  not  concentrated  at  0     •  Linear  decoder,  non-­‐parametric  encoder   •  Sparse  Coding  inference,  convex  opt.  but  expensive   97  
  • 98. Predictive Sparse Decomposition •  Approximate  the  inference  of  sparse  coding  by   an  encoder:   Predic>ve  Sparse  Decomposi>on  (Kavukcuoglu  et  al  2008)   •  Very  successful  applica>ons  in  machine  vision   with  convolu>onal  architectures   98  
  • 99. Predictive Sparse Decomposition •  Stacked  to  form  deep  architectures   •  Alterna>ng  convolu>on,  rec>fica>on,  pooling   •  Tiling:  no  sharing  across  overlapping  filters   •  Group  sparsity  penalty  yields  topographic   maps   99  
  • 101. Stack of RBMs / AEs  Deep MLP •  Encoder  or  P(h|v)  becomes  MLP  layer       h3   ^   y   W3   h2   h3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   101  
  • 102. Stack of RBMs / AEs  Deep Auto-Encoder (Hinton  &  Salakhutdinov  2006)   •  Stack  encoders  /  P(h|x)  into  deep  encoder   •  Stack  decoders  /  P(x|h)  into  deep  decoder   ^   x   ^   WT   1   h1   T   W2   h3   ^   h2   W3   WT   h2   h3   3   W3   h2   h2   W2   W2   h1   h1   W1   h1   x   W1   x   102  
  • 103. Stack of RBMs / AEs  Deep Recurrent Auto-Encoder (Savard  2011)   h3   W3   h2   •  Each  hidden  layer  receives  input  from  below  and   h2   above   W2   h1   •  Halve  the  weights     h1   •  Determinis>c  (mean-­‐field)  recurrent  computa>on   W1   x     h3   W3   T   ½W3   W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   103  
  • 104. Stack of RBMs  Deep Belief Net (Hinton  et  al  2006)   •  Stack  lower  levels  RBMs’  P(x|h)  along  with  top-­‐level  RBM   •  P(x,  h1  ,  h2  ,  h3)  =  P(h2  ,  h3)  P(h1|h2)  P(x  |  h1)   •  Sample:  Gibbs  on  top  RBM,  propagate  down   h3   h2   h1   x   104  
  • 105. Stack of RBMs  Deep Boltzmann Machine (Salakhutdinov  &  Hinton  AISTATS  2009)   •  Halve  the  RBM  weights  because  each  layer  now  has  inputs  from   below  and  from  above   •  Posi>ve  phase:  (mean-­‐field)  varia>onal  inference  =  recurrent  AE   •  Nega>ve  phase:  Gibbs  sampling  (stochas>c  units)   •  train  by  SML/PCD   h3   W3   T   ½W3   ½W3   T   ½W3   h2   T   T   W2   ½W2   ½W2   ½W2   ½W2   h1   T   T   W1   WT   1   ½W1   ½W1   ½W1   ½W1   x   105  
  • 106. Stack of Auto-Encoders  Deep Generative Auto-Encoder (Rifai  et  al  ICML  2012)   •  MCMC  on  top-­‐level  auto-­‐encoder   •  ht+1  =  encode(decode(ht))+σ  noise   where  noise  is  Normal(0,  d/dh  encode(decode(ht)))   •  Then  determinis>cally  propagate  down  with  decoders     h3   h2   h1   x   106  
  • 107. Sampling from a Regularized Auto-Encoder 107  
  • 108. Sampling from a Regularized Auto-Encoder 108  
  • 109. Sampling from a Regularized Auto-Encoder 109  
  • 110. Sampling from a Regularized Auto-Encoder 110  
  • 111. Sampling from a Regularized Auto-Encoder 111  
  • 112. Part  3   Practice, Issues, Questions 112  
  • 113. Deep Learning Tricks of the Trade •  Y.  Bengio  (2012),  “Prac>cal  Recommenda>ons  for  Gradient-­‐ Based  Training  of  Deep  Architectures”     •  Unsupervised  pre-­‐training   •  Stochas>c  gradient  descent  and  se•ng  learning  rates   •  Main  hyper-­‐parameters   •  Learning  rate  schedule   •  Early  stopping   •  Minibatches   •  Parameter  ini>aliza>on   •  Number  of  hidden  units   •  L1  and  L2  weight  decay   •  Sparsity  regulariza>on   •  Debugging   •  How  to  efficiently  search  for  hyper-­‐parameter  configura>ons   113  
  • 114. Stochastic Gradient Descent (SGD) •  Gradient  descent  uses  total  gradient  over  all  examples  per   update,  SGD  updates  afer  only  1  or  few  examples:   •  L  =  loss  func>on,  zt  =  current  example,  θ  =  parameter  vector,  and   εt  =  learning  rate.   •  Ordinary  gradient  descent  is  a  batch  method,  very  slow,  should   never  be  used.  2nd  order  batch  method  are  being  explored  as  an   alterna>ve  but  SGD  with  selected  learning  schedule  remains  the   method  to  beat.   114  
  • 115. Learning Rates •  Simplest  recipe:  keep  it  fixed  and  use  the  same  for  all   parameters.   •  Collobert  scales  them  by  the  inverse  of  square  root  of  the  fan-­‐in   of  each  neuron   •  Beber  results  can  generally  be  obtained  by  allowing  learning   rates  to  decrease,  typically  in  O(1/t)  because  of  theore>cal   convergence  guarantees,  e.g.,            with  hyper-­‐parameters  ε0  and  τ.   115  
  • 116. Long-Term Dependencies and Clipping Trick •  In  very  deep  networks  such  as  recurrent  networks  (or  possibly   recursive  ones),  the  gradient  is  a  product  of  Jacobian  matrices,   each  associated  with  a  step  in  the  forward  computa>on.  This   can  become  very  small  or  very  large  quickly  [Bengio  et  al  1994],   and  the  locality  assump>on  of  gradient  descent  breaks  down.     •  The  solu>on  first  introduced  by  Mikolov    is  to  clip  gradients   to  a  maximum  value.  Makes  a  big  difference  in  Recurrent    Nets     116  
  • 117. Early Stopping •  Beau>ful  FREE  LUNCH  (no  need  to  launch  many  different   training  runs  for  each  value  of  hyper-­‐parameter  for  #itera>ons)   •  Monitor  valida>on  error  during  training  (afer  visi>ng  #   examples  a  mul>ple  of  valida>on  set  size)   •  Keep  track  of  parameters  with  best  valida>on  error  and  report   them  at  the  end   •  If  error  does  not  improve  enough  (with  some  pa>ence),  stop.   117  
  • 118. Parameter Initialization •  Ini>alize  hidden  layer  biases  to  0  and  output  (or  reconstruc>on)   biases  to  op>mal  value  if  weights  were  0  (e.g.  mean  target  or   inverse  sigmoid  of  mean  target).   •  Ini>alize  weights  ~  Uniform(-­‐r,r),  r  inversely  propor>onal  to  fan-­‐ in  (previous  layer  size)  and  fan-­‐out  (next  layer  size):            for  tanh  units  (and  4x  bigger  for  sigmoid  units)    (Glorot  &  Bengio  AISTATS  2010)   118  
  • 119. Handling Large Output Spaces   •  Auto-­‐encoders  and  RBMs  reconstruct  the  input,  which  is  sparse  and  high-­‐ dimensional;  Language  models  have  huge  output  space.   code= latent features expensive cheap … …     sparse input dense output probabilities   •  (Dauphin  et  al,  ICML  2011)  Reconstruct  the  non-­‐zeros  in     the  input,  and  reconstruct  as  many  randomly  chosen   zeros,  +  importance  weights   categories   •  (Collobert  &  Weston,  ICML  2008)  sample  a  ranking  loss   •  Decompose  output  probabili>es  hierarchically  (Morin   &  Bengio  2005;  Blitzer  et  al  2005;  Mnih  &  Hinton   words  within  each  category   2007,2009;  Mikolov  et  al  2011)   119      
  • 120. Automatic Differentiation •  The  gradient  computa>on  can  be   automa>cally  inferred  from  the  symbolic   expression  of  the  fprop.   •  Makes  it  easier  to  quickly  and  safely  try   new  models.   •  Each  node  type  needs  to  know  how  to   compute  its  output  and  how  to  compute   the  gradient  wrt  its  inputs  given  the   gradient  wrt  its  output.   •  Theano  Library  (python)  does  it   symbolically.  Other  neural  network   packages  (Torch,  Lush)  can  compute   gradients  for  any  given  run-­‐>me  value.   (Bergstra  et  al  SciPy’2010)   120  
  • 121. Random Sampling of Hyperparameters (Bergstra  &  Bengio  2012)   •  Common  approach:  manual  +  grid  search   •  Grid  search  over  hyperparameters:  simple  &  wasteful   •  Random  search:  simple  &  efficient   •  Independently  sample  each  HP,  e.g.  l.rate~exp(U[log(.1),log(.0001)])   •  Each  training  trial  is  iid   •  If  a  HP  is  irrelevant  grid  search  is  wasteful   •  More  convenient:  ok  to  early-­‐stop,  con>nue  further,  etc.   121  
  • 123. Why is Unsupervised Pre-Training Working So Well? •  Regulariza>on  hypothesis:     •  Unsupervised  component  forces  model  close  to  P(x)   •  Representa>ons  good  for  P(x)  are  good  for  P(y|x)   •  Op>miza>on  hypothesis:   •  Unsupervised  ini>aliza>on  near  beber  local  minimum  of  P(y|x)   •  Can  reach  lower  local  minimum  otherwise  not  achievable  by  random  ini>aliza>on   •  Easier  to  train  each  layer  using  a  layer-­‐local  criterion   (Erhan  et  al  JMLR  2010)  
  • 124. Learning Trajectories in Function Space •  Each  point  a  model  in   func>on  space   •  Color  =  epoch   •  Top:  trajectories  w/o   pre-­‐training   •  Each  trajectory   converges  in  different   local  min.   •  No  overlap  of  regions   with  and  w/o  pre-­‐ training  
  • 125. Dealing with a Partition Function •  Z  =  Σx,h  e-­‐energy(x,h)   •  Intractable  for  most  interes>ng  models   •  MCMC  es>mators  of  its  gradient   •  Noisy  gradient,  can’t  reliably  cover  (spurious)  modes   •  Alterna>ves:   •  Score  matching  (Hyvarinen  2005)   •  Noise-­‐contras>ve  es>ma>on  (Gutmann  &  Hyvarinen  2010)   •  Pseudo-­‐likelihood   •  Ranking  criteria  (wsabie)  to  sample  nega>ve  examples  (Weston  et  al.   2010)   •  Auto-­‐encoders?   125  
  • 126. Dealing with Inference •  P(h|x)  in  general  intractable  (e.g.  non-­‐RBM  Boltzmann  machine)   •  But  explaining  away  is  nice   •  Approxima>ons   •  Varia>onal  approxima>ons,  e.g.  see  Goodfellow  et  al  ICML  2012   (assume  a  unimodal  posterior)   •  MCMC,  but  certainly  not  to  convergence   •  We  would  like  a  model  where  approximate  inference  is  going  to  be  a  good   approxima>on   •  Predic>ve  Sparse  Decomposi>on  does  that   •  Learning  approx.  sparse  decoding    (Gregor  &  LeCun  ICML’2010)   •  Es>ma>ng  E[h|x]  in  a  Boltzmann  with  a  separate  network  (Salakhutdinov  &   Larochelle  AISTATS  2010)   126  
  • 127. For gradient & inference: More difficult to mix with better trained models •  Early  during  training,  density  smeared  out,  mode  bumps  overlap   •  Later  on,  hard  to  cross  empty  voids  between  modes   127  
  • 128. Poor Mixing: Depth to the Rescue •  Deeper  representa>ons  can  yield  some  disentangling   •  Hypotheses:     •  more  abstract/disentangled  representa>on  unfold  manifolds   and  fill  more  the  space   •  can  be  exploited  for  beber  mixing  between  modes   •  E.g.  reverse  video  bit,  class  bits  in  learned  object   representa>ons:  easy  to  Gibbs  sample  between  modes  at   Layer   abstract  level   0   1   2   Points  on  the  interpola>ng  line  between  two  classes,  at  different  levels  of  representa>on   128  
  • 129. Poor Mixing: Depth to the Rescue •  Sampling  from  DBNs  and  stacked  Contras>ve  Auto-­‐Encoders:   1.  MCMC  sample  from  top-­‐level  singler-­‐layer  model   2.  Propagate  top-­‐level  representa>ons  to  input-­‐level  repr.   •  Visits  modes  (classes)  faster   Toronto  Face  Database   h3   h2   h1   x   129   #  classes  visited    
  • 130. What are regularized auto-encoders learning exactly? •  Any  training  criterion  E(X,  θ)  interpretable  as  a  form  of  MAP:   •  JEPADA:  Joint  Energy  in  PArameters  and  Data    (Bengio,  Courville,  Vincent  2012)   This  Z  does  not  depend  on  θ.  If  E(X,  θ)  tractable,  so  is  the  gradient   No  magic;  consider  tradi>onal  directed  model:       Applica>on:  Predic>ve  Sparse  Decomposi>on,  regularized  auto-­‐encoders,  …     130  
  • 131. What are regularized auto-encoders learning exactly? •  Denoising  auto-­‐encoder  is  also  contrac>ve   •  Contrac>ve/denoising  auto-­‐encoders  learn  local  moments   •  r(x)-­‐x      es>mates  the  direc>on  of  E[X|X  in  ball  around  x]   •  Jacobian                  es>mates  Cov(X|X  in  ball  around  x)   •  These  two  also  respec>vely  es>mate  the  score  and  (roughly)  the   Hessian    of  the  density   131  
  • 132. More Open Questions •  What  is  a  good  representa>on?  Disentangling  factors?  Can  we   design  beber  training  criteria  /  setups?   •  Can  we  safely  assume  P(h|x)  to  be  unimodal  or  few-­‐modal?If   not,  is  there  any  alterna>ve  to  explicit  latent  variables?     •  Should  we  have  explicit  explaining  away  or  just  learn  to  produce   good  representa>ons?   •  Should  learned  representa>ons  be  low-­‐dimensional  or  sparse/ saturated  and  high-­‐dimensional?   •  Why  is  it  more  difficult  to  op>mize  deeper  (or  recurrent/ recursive)  architectures?  Does  it  necessarily  get  more  difficult  as   training  progresses?  Can  we  do  beber?   132