![ Traditional Convolutional feed-forward networks :
9
𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1)
𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1) + 𝑥𝑙−1
ResNets :
DenseNets :
𝑥𝑙 = 𝐻𝑙 ([𝑥0, 𝑥1, … , 𝑥𝑙−1])
Where [𝑥0, 𝑥1, … , 𝑥𝑙−1] refers to the concatenation of the feature-maps produced
in layers 0….. 𝑙-1.](https://image.slidesharecdn.com/denselyconnectedconvolutionalnetworks-180613103256/85/Densely-Connected-Convolutional-Networks-9-320.jpg)
![19
Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]
Remember feature visualization](https://image.slidesharecdn.com/denselyconnectedconvolutionalnetworks-180613103256/85/Densely-Connected-Convolutional-Networks-19-320.jpg)
Densely Connected Convolutional Networks
- 1. DENSELY Prof. Mohammad-R.Akbarzadeh-T Ferdowsi University of Mashhad A Presentation by: • Hosein Mohebbi • M.-Sajad Abavisani
- 2. CVPR 2017 BEST PAPER AWARD 2 Gao Huang Cornell University h-index: 12 Zhuang Liu Tsinghua University h-index: 5 Laurens van der Maaten Facebook AI Research h-index: 29 Kilian Weinberger Associate Professor Cornell University h-index: 41
- 3. 3
- 4. Normalized initialization and intermediate normalization layers The main culprit :Vanishing/exploding gradients Not caused by overfitting 4
- 5. 5 RESNET
- 6. STOCHASTIC DEPTH Deep network during testing,but shallower network during training. 6 𝐻𝑙 = 𝑅𝑒𝐿𝑈 𝑏𝑙 𝑓𝑙(𝐻𝑙−1 + 𝑖𝑑 𝐻𝑙−1 ) 𝑏𝑙 ∈ 0,1 They all share a key characteristic: They create short paths from early layers to later layers
- 8. DENSE 8 The growth rate regulates how much new information each layer contributes to the global state.
- 9. Traditional Convolutional feed-forward networks : 9 𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1) 𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1) + 𝑥𝑙−1 ResNets : DenseNets : 𝑥𝑙 = 𝐻𝑙 ([𝑥0, 𝑥1, … , 𝑥𝑙−1]) Where [𝑥0, 𝑥1, … , 𝑥𝑙−1] refers to the concatenation of the feature-maps produced in layers 0….. 𝑙-1.
- 10. 10
- 11. DENSENET 11
- 13. DENSENET 13 Compression in transition layer Pooling reduces Feature map sizes Feature map sizes match Within each block
- 14. DENSE
- 15. 15 Direct access: Deep Supervision with single classifier Reduces overfitting on tasks with smaller training set sizes
- 16. 16
- 17. 17 ResNet connectivity: DenseNet connectivity: #parameters: k: Growth rate
- 19. 19 Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013] Remember feature visualization
- 21. Feature reuse Information flow from the first to the last layers of the block Compression in transition layer Concentrate on high level feature for final classification 21
- 23. CIFAR-10 23
- 25. IMAGENET 25
- 26. 26 Kaiming He,et al."Deep residual learning for image recognition" CVPR 2016 Chen-Yu Lee,et al."Deeply-supervised nets" AISTATS 2015 Gao Huang,et al."Deep networks with stochastic depth" ECCV 2016 CS231n:Convolutional Neural Networks forVisual Recognition Gao Huang,Zhuang Liu,Kilian QWeinberger,and Laurens van der Maaten. Densely connected convolutional networks.Conference on ComputerVision and Pattern Recognition,2017 Geoff Pleiss,et al."Memory-Efficient Implementation of DenseNets”, arXiv preprint arXiv:1707.06990 (2017)