Ad
More Related Content
What's hot (20)
Viewers also liked (20)
![Big Data [sorry] & Data Science: What Does a Data Scientist Do?](https://cdn.slidesharecdn.com/ss_thumbnails/dslatcloudmsevent20130125-130126065651-phpapp01-thumbnail.jpg?width=560&fit=bounds)
Ad
Similar to Deep neural networks (20)
Ad
Recently uploaded (20)
Deep neural networks
- 1. Deep Learning and Application in Neural Networks Hugo Larochelle Geoffrey Hinton Yoshua Bengio Andrew Ng. Jerome Louradour Andrew L. Nelson Pascal Lamblin R. Salskhutdinov
- 2. Biological Neurons Terminal Branches Dendrites of Axon Axon
- 3. Artificial Neural Networks (ANN) Terminal Branches Dendrites of Axon x1 w1 x2 w2 x3 w3 Axon wn xn Slide credit : Andrew L. Nelson
- 4. Layered Networks Input nodes Output nodes Hidden nodes Connections Output : yi f ( wi1x1 wi2x 2 wi3x 3 wim x m ) f( wij x j ) j
- 5. Neural network application • ALVINN drives 70 mph on highways Slide credit : T.Michell
- 6. Neural network application Slide credit : Andraw Ng.
- 7. The simplest model- the Perceptron •The Perceptron was introduced in 1957 by Frank Rosenblatt. Perceptron: - D0 d D1 Activation D2 functions: Learning: Input Output Layer Destinations Layer Update Slide credit : Geoffrey Hinton
- 8. Second generation neural networks (~1985) Compare outputs with Back-propagate correct answer to get error signal to get error signal derivatives for learning outputs hidden layers input vector Slide credit : Geoffrey Hinton
- 9. BP-algorithm 1 .5 0 -5 0 5 Activations errors The error: Update Update Weights: Slide credit : Geoffrey Hinton
- 10. Back Propagation Advantages • Multi layer Perceptron network can be trained by the back propagation algorithm to perform any mapping between the input and the output. What is wrong with back-propagation? •It requires labeled training data. Almost all data is unlabeled. •The learning time does not scale well It is very slow in networks with multiple hidden layers. •It can get stuck in poor local optima. Slide credit : Geoffrey Hinton
- 11. Why Deep multi-layered neural network object models object parts (combination of edges) edges pixels
- 13. Deep Neural Networks • Standard learning strategy – Randomly initializing the weights of the network – Applying gradient descent using backpropagation • But, backpropagation does not work well (if randomly initialized) – Deep networks trained with back-propagation (without unsupervised pre-train) perform worse than shallow networks – ANN have limited to one or two layers
- 17. Unsupervised greedy layer-wise training procedure.
- 27. Layer-Local Unsupervised Learning • Restricted Boltzmann Machine (SRBM) (Hinton et al, NC’2006) • Auto-encoders (Bengio et al, NIPS’2006) • Sparse auto-encoders (Ranzato et al, NIPS’2006) • Kernel PCA (Erhan 2008) • Denoising auto-encoders (Vincent et al, ICML’2008) • Unsupervised embedding (Weston et al, ICML’2008) • Slow features (Mohabi et al, ICML’2009, Bergstra & Bengio NIPS’2009)
- 28. Experiments • MNIST data set – A benchmark for handwritten digit recognition – The number of classes is 10 (corresponding to the digits from 0 to 9) – The inputs were scaled between 0 and 1 Exploring Strategies for Training Deep Neural Networks. Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, Pascal Lamblin; 10(Jan):1--40, 2009.
- 29. Classification error on MNIST training
- 30. Experiments • Histograms presenting the test errors obtained on MNIST using models trained with or without pre-training. The difficulty of training deep architectures and the effect of unsupervised pre-training. Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent; pages 153-160, 2009.
- 31. Experiments • NORB data set – 5 object categories, 5 difference objects within each category. • Classification error rate – DBM : 10.8 % – SVM : 11.6 % – Logistic Regression : 22.5 % – KNN : 18.4 % Efficient Learning of Deep Boltzmann Machines. Ruslan Salakhutdinov, Hugo Larochelle ; JMLR W&CP 9:693-700, 2010.
- 32. Conclusion • Deep Learning : powerful arguments & generalization priciples • Unsupervised Feature Learning is crucial many new algorithms and applications in recent years • Deep Learning suited for multi-task learning, domain adaptation and semi-learning with few labels
- 33. Application • Classification (Bengio et al., 2007; Ranzato et al., 2007b; Larochelle et al., 2007; Ranzato et al.,2008) • Regression (Salakhutdinov and Hinton, 2008) • Dimensionality Reduction (Hinton Salakhutdinov, 2006; Salakhutdinov and Hinton, 2007) • Modeling Textures (Osindero and Hinton, 2008) • Information Retrieval (Salakhutdinov and Hinton, 2007) • Robotics (Hadsell et al., 2008) • Natural Language Processing (Collobert and Weston, 2008; Weston et al., 2008) • Collaborative Filtering (Salakhutdinov et al., 2007)
- 34. Recent Deep Learning Highlights
- 35. Recent Deep Learning Highlights • Google Goggles uses Stacked Sparse Auto Encoders (Hartmut Neven @ ICML 2011) • The monograph or review paper Learning Deep Architectures for AI (Foundations & Trends in Machine Learning, 2009). • Exploring Strategies for Training Deep Neural Networks, Hugo Larochelle, Yoshua Bengio, Jerome Louradour and Pascal Lamblin in: The Journal of Machine Learning Research, pages 1-40, 2009. • The LISA publications database contains a deep architectures category. http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/ ReadingOnDeepNetworks • Deep Machine Learning – A New Frontier in Artificial Intelligence Research – a survey paper by Itamar Arel, Derek C. Rose, and Thomas P. Karnowski.
- 36. Thank You