![[DL輪読会]Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs (CV...](https://cdn.slidesharecdn.com/ss_thumbnails/zero-shot-gcn-180803004516-thumbnail.jpg?width=560&fit=bounds)
![[GomGuard] 뉴런부터 YOLO 까지 - 딥러닝 전반에 대한 이야기](https://cdn.slidesharecdn.com/ss_thumbnails/part1mlp2cnn-slidesharev-180308015531-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to autoencoder-190813144108.pptx (20)
Ad
More from kiran814572 (6)
Ad
Recently uploaded (20)
autoencoder-190813144108.pptx
- 1. Auto Encoders In the name of God Mehrnaz Faraz Faculty of Electrical Engineering K. N. Toosi University of Technology Milad Abbasi Faculty of Electrical Engineering Sharif University of Technology 1
- 2. Auto Encoders • An unsupervised deep learning algorithm • Are artificial neural networks • Useful for dimensionality reduction and clustering Unlabeled data 𝑧 = 𝑠 𝑤𝑥 + 𝑏 𝑥 ^= 𝑠 𝑤′z + 𝑏′ 𝑥 ^is 𝑥’s reconstruction 𝑧 is some latent representation or code and 𝑠 is a non-linearity such as the sigmoid 𝑧 𝑥 ^ 𝑥 2 Encoder Decoder
- 3. Auto Encoders • Simple structure: 𝒙𝟏 𝒙𝟑 𝒙𝟐 𝒙˜𝟏 𝒙˜𝟑 𝒙˜𝟐 Input Reconstructed Output Hidden Encoder 3 Decoder
- 4. Undercomplete AE • Hidden layer is Undercomplete if smaller than the input layer – Compresses the input – Hidden nodes will be Good features for the training 𝑥 ^ 𝑤′ 𝑧 𝑤 𝑥 4
- 5. Overcomplete AE • Hidden layer is Overcomplete if greater than the input layer – No compression in hidden layer. – Each hidden unit could copy a different input component. 𝑥 ^ 𝑤′ 𝑧 𝑤 𝑥 5
- 6. Deep Auto Encoders • Deep Auto Encoders (DAE) • Stacked Auto Encoders (SAE) 6
- 7. Training Deep Auto Encoder • First layer: 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒙^𝟏 𝒙^𝟒 𝒙^𝟑 𝒙^𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 Encoder Decoder 7
- 8. Training Deep Auto Encoder • Features of first layer: 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝑎1 𝑎2 𝑎3 8
- 9. Training Deep Auto Encoder • Second layer: 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒂^𝟏 𝒂^𝟑 𝒂^𝟐 𝒃𝟐 𝒃𝟏 9
- 10. Training Deep Auto Encoder • Features of second layer: 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒃𝟐 𝒃𝟏 𝑏1 𝑏2 10
- 11. Using Deep Auto Encoder 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 • Feature extraction • Dimensionality reduction • Classification 𝒙𝟏 𝒂𝟏 𝒃𝟐 𝒃𝟏 Inputs Features 𝒙𝟒 Encoder 11
- 12. Using Deep Auto Encoder • Reconstruction 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒃𝟐 𝒃𝟏 𝒂^𝟏 𝒂^𝟑 𝒂^𝟐 𝒙^𝟒 𝒙^𝟑 𝒙^𝟐 𝒙^𝟏 Encoder Decoder 12
- 13. Using AE • Denoising • Data compression • Unsupervised learning • Manifold learning • Generative model 13
- 14. Types of Auto Encoder • Stacked auto encoder (SAE) • Denoising auto encoder (DAE) • Sparse Auto Encoder (SAE) • Contractive Auto Encoder (CAE) • Convolutional Auto Encoder (CAE) • Variational Auto Encoder (VAE) 14
- 15. Generative Models • Given training data, generate new samples from same distribution – Variational Auto Encoder (VAE) – Generative Adversarial Network (GAN) 15
- 16. Variational Auto Encoder Encoder Decoder Output 𝐱˜ ∅ 𝒒 𝒛|𝒙 𝒑𝜽 𝒙|𝒛 𝒙𝟏 16 𝒙𝟒 Input x 𝒙𝟑 𝒙𝟐 𝒙˜𝟏 𝒙˜𝟒 𝒙˜𝟑 𝒙˜𝟐 𝒛𝟏 𝒛𝟐
- 17. Variational Auto Encoder • Use probabilistic encoding and decoding – Encoder: – Decoder: • x: Unknown probability distribution • z: Gaussian probability distribution 𝑞∅ 𝑧|𝑥 𝑝𝜃 𝑥|𝑧 17
- 18. Training Variational Auto Encoder • Latent space: 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒉𝟏 𝒉𝟐 𝒉𝟑 𝝈 𝝁 𝒛 𝑞∅ 𝑧|𝑥 18 Mean Variance 1 dimensional Gaussian probability distribution If we have n neurons for 𝝈 and 𝝁 then we have n dimensional distribution
- 19. Training Variational Auto Encoder • Generating new data: – Example: MNIST Database 𝐱 Encoder Latent space Decoder 19
- 20. Generative Adversarial Network • VAE: • GAN: – Can generate samples – Trained by competing each other – Use neural network – Z is some random noise (Gaussian/Uniform). – Z can be thought as the latent representation of the image. x Decoder 𝐱 ˜ z Encoder z Generator 𝐱 ˜ x Discriminator Fake or real? 20 Loss
- 21. GAN’s Architecture Discriminator Generated fake samples Fine tune training Latent space Noise Is D correct? Generator • Overview: Real samples
- 22. Using GAN • Image generation: 22
- 23. Using GAN • Data manipulation: 23
- 24. Denoising Auto Encoder • Add noise to its input, and train it to recover this original. 24
- 25. Denoising Auto Encoder Input Output Hidden 3 Hidden 2 Hidden 1 + Noise Input Output Hidden 3 Hidden 2 Hidden 1 Dropout 25 Randomly switched input Gaussian noise
- 26. Sparse Auto Encoder • Reduce the number of active neurons in the coding layer. – Add sparsity loss into the cost function. • Sparsity loss: – Kullback-Leibler(KL) divergence is commonly used. 26
- 27. Sparse Auto Encoder j j j ˆ KL ̂ log 1 log 1 1 ̂ Jsparse w,b J w,b KL ̂j j1 27