Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring 2021
1 like14,174 views
This document summarizes key applications of deep learning in computer vision discussed in a lecture, including: 1. Classification, localization, detection, and segmentation using convolutional neural networks (CNNs) like AlexNet, VGG, GoogLeNet, ResNet and their variants. 2. CNN architectures for detection tasks like YOLO, SSD, Faster R-CNN, and Mask R-CNN using region proposals, bounding box regression and segmentation. 3. Other applications discussed include 3D shape inference, face landmark recognition, pose estimation, style transfer and generative adversarial networks (GANs). Labeled datasets are important for many vision tasks.
![GoogLeNet: Inception Module
• Instead of picking 1x1, 3x3, 5x5, or pooling, send each input through all of them, and
concatenate the results depthwise.
• Reduce with 1x1 conv first.
• This reduces the amount of computation and follows the “Inception hypothesis”: cross-
channel correlations and spatial correlations are decoupled and can be mapped separately.
hypothesis than the Inception hypothesis, and assume that
cross-channel correlations and spatial correlations can be
mapped completely separately?
Figure 1. A canonical Inception module (Inception V3).
Figure 2. A simplified Inception module.
Figure 4. An “extreme” vers
spatial convolution per outpu
network design as early a
popular since its inclusion
in 2016.
A depthwise separabl
Xception: Deep Learning with Depthwise Separable Convolutions
Xception: Deep Learning with Depthwise Separable Convolutions
François Chollet
Google, Inc.
arXiv:1610.02357v3 [cs.CV] 4 Apr 2017](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-23-320.jpg)
![Localization
• Simply predict the bounding box coordinates (x1, y1, x2, y2) as well
as the class, using the same network.
[‘x1’: 0.05, ‘y1’: 0.1, ‘x2’: 0.8, ‘y2’: 0.65]
[‘dog’: 0.01, ‘cat’: 0.01, ‘duck’: 0.9, ‘bg’: 0.08]](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-43-320.jpg)
![Overfeat
http://www.image-net.org/challenges/LSVRC/2013/slides/overfeat_ilsvrc2013.pdf
[Sermanet, 2013]](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-48-320.jpg)
![Overfeat
http://www.image-net.org/challenges/LSVRC/2013/slides/overfeat_ilsvrc2013.pdf
[Sermanet, 2013]
• For each location considered, predict class and bounding box
• Then count votes and unify via non-maximum suppression](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-49-320.jpg)
![Region proposal methods
• There is an alternative to looking in every place in an image: look only at
regions that seem interesting
• Different methods, most based on some kind of image segmentation
https://ivi.fnwi.uva.nl/isis/publications/bibtexbrowser.php?key=UijlingsIJCV2013
[Ujnings, 2013]](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-56-320.jpg)
![R-CNN
http://cs231n.stanford.edu/
Region proposals
(slow external method)
[Girschick, 2014]
AlexNet on regions
warped to square
Predicting both class
and bounding box
• An interim solution, bridging old-school
detection methods and convents](https://image.slidesharecdn.com/fsdl-berkeley-lecture2-vision-applications-210129205844/85/Lecture-2-B-Computer-Vision-Applications-Full-Stack-Deep-Learning-Spring-2021-57-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring 2021 (20)
Ad
More from Sergey Karayev (10)
Ad
Recently uploaded (20)
Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring 2021
- 1. Full Stack Deep Learning - UC Berkeley Spring 2021 - Sergey Karayev, Josh Tobin, Pieter Abbeel Lecture 2.B Computer Vision Applications
- 2. Goal for this lecture 1. Review notable applications of DL in computer vision 2. Build a toolbox of model architectures and tricks
- 3. Agenda 1. Tour of Convnet Architectures 2. Localization, Detection, Segmentation 3. Misc
- 4. Agenda 1. Tour of Convnet Architectures 2. Localization, Detection, Segmentation 3. Misc
- 6. Progress in Convnet Architectures • AlexNet’s outstanding performance on ImageNet in 2012 launched the convnet revolution • Progress since then is well tracked by ImageNet winners
- 7. ImageNet • ImageNet Large Scale Visual Recognition Challenge (ILSVRC): running since 2010 • 1.2M images from 1000 object categories (test on 150K) • Classification, Localization, and Detection tracks. Classification has gotten the most attention.
- 10. • First convnet winner • Innovated with ReLU and Dropout • Heavy data augmentation (flip, scale, etc) • Fun fact: usually drawn as two parts, because GPU only had 3GB memory http://cs231n.stanford.edu/ AlexNet
- 13. • Essentially AlexNet, but with a smaller CONV1 and higher-channel CONV3,4,5 • Reduced error 5%! ZFNet
- 14. • Essentially AlexNet, but with a smaller CONV1 and higher-channel CONV3,4,5 • Reduced error 5%! • Better known for cool “deconvolutional” visualizations ZFNet
- 17. • Simple but deep architecture of only 3x3 stride-1 convs, 2x2 stride-2 pools. • Channel dimension increases with each layer. • From 11.7% (ZFNet) down to 7.3% top-5 error VGG 4 stacked 3x3 convs get the same receptive field as a 9x9 conv, but use fewer parameters https://tryolabs.com/blog/2018/01/18/faster-r-cnn-down-the-rabbit-hole-of-modern-object-detection/
- 22. GoogLeNet • Just as deep as VGG, but has only 3% of the parameters! • No fully-connected layers • Is just a stack of Inception modules, which is a network module more creative than the standard (conv->relu->conv->relu->pool)
- 23. GoogLeNet: Inception Module • Instead of picking 1x1, 3x3, 5x5, or pooling, send each input through all of them, and concatenate the results depthwise. • Reduce with 1x1 conv first. • This reduces the amount of computation and follows the “Inception hypothesis”: cross- channel correlations and spatial correlations are decoupled and can be mapped separately. hypothesis than the Inception hypothesis, and assume that cross-channel correlations and spatial correlations can be mapped completely separately? Figure 1. A canonical Inception module (Inception V3). Figure 2. A simplified Inception module. Figure 4. An “extreme” vers spatial convolution per outpu network design as early a popular since its inclusion in 2016. A depthwise separabl Xception: Deep Learning with Depthwise Separable Convolutions Xception: Deep Learning with Depthwise Separable Convolutions François Chollet Google, Inc. arXiv:1610.02357v3 [cs.CV] 4 Apr 2017
- 24. GoogLeNet • Another cool trick: additional inject classifier gradient earlier in the network • And, remove FC layers for speed
- 27. ResNet • Down to 3.57% top-5 error (lower than human performance of 5%) • Problem: deeper models should be able to perform at least as well as shallower networks, but don’t due to vanishing gradient • Solution: add an option to simply skip layers, such that if the gradient vanishes, at least no harm is done!
- 28. ResNet • Why “residual”? Because fitting F(x) = H(x) - x http://cs231n.stanford.edu/
- 29. ResNet • Increase number of filters with depth; downsample spatially using strides. http://cs231n.stanford.edu/
- 30. Residual Net Variants • Add more skip connections! http://cs231n.stanford.edu/ DenseNet ResNeXt • Cardinality of a module: the amount of paths through split/ transform/aggregate
- 33. SENet • Add a module of global pooling + FC to adaptively reweight feature output maps • Kind of like attention: the network gets to choose what to use
- 34. SqueezeNet • AlexNet-level accuracy with 50x fewer params and <0.5MB model, through constant 1x1 bottlenecking • Interesting for hardware/mobile deployment
- 35. Overall Comparison • Some ResNets, or Inception-v4 (Inception + ResNet) are probably in the sweet spot of accuracy to operations / memory.
- 38. Fast Training https://dawn.cs.stanford.edu/benchmark/#imagenet-train-time
- 39. Agenda 1. Tour of Convnet Architectures 2. Localization, Detection, Segmentation 3. Misc
- 40. Localization, Detection, Segmentation • Classification: output the single object’s class • Localization: output the single object’s class and its location • Detection: output every object’s class and location • Segmentation: label every pixel in the image as belonging to an object or to background • Instance Segmentation additionally differentiates between different objects of the same class http://cs231n.stanford.edu/
- 41. Localization, Detection, Segmentation • Classification: output the single object’s class • Localization: output the single object’s class and its location • Detection: output every object’s class and location • Segmentation: label every pixel in the image as belonging to an object or to background • Instance Segmentation additionally differentiates between different objects of the same class http://cs231n.stanford.edu/
- 42. Localization, Detection, Segmentation • Classification: output the single object’s class • Localization: output the single object’s class and its location • Detection: output every object’s class and location • Segmentation: label every pixel in the image as belonging to an object or to background • Instance Segmentation additionally differentiates between different objects of the same class http://cs231n.stanford.edu/
- 43. Localization • Simply predict the bounding box coordinates (x1, y1, x2, y2) as well as the class, using the same network. [‘x1’: 0.05, ‘y1’: 0.1, ‘x2’: 0.8, ‘y2’: 0.65] [‘dog’: 0.01, ‘cat’: 0.01, ‘duck’: 0.9, ‘bg’: 0.08]
- 44. Detection • Localization does not scale when there are multiple objects.
- 45. Detection • One solution: slide a classifier over the image (at multiple scales) Very computationally expensive But there is a way to ameliorate that
- 46. Turning FC layers into conv layers Andrew Ng Turning FC layer into convolutional layers 10 10 16 5 5 16 14 14 3 5 5 2 2 MAX POOL FC FC y softmax (4) 14 14 3 10 10 16 5 5 16 5 5 2 2 MAX POOL 1 1 400 1 1 400 1 1 4 FC 5 5 FC 1 1 400 ⋮ 400 ⋮ deeplearning.ai
- 47. Turning FC layers into conv layers Andrew Ng Turning FC layer into convolutional layers 10 10 16 5 5 16 14 14 3 5 5 2 2 MAX POOL FC FC y softmax (4) 14 14 3 10 10 16 5 5 16 5 5 2 2 MAX POOL 1 1 400 1 1 400 1 1 4 FC 5 5 FC 1 1 400 ⋮ 400 ⋮ deeplearning.ai
- 49. Overfeat http://www.image-net.org/challenges/LSVRC/2013/slides/overfeat_ilsvrc2013.pdf [Sermanet, 2013] • For each location considered, predict class and bounding box • Then count votes and unify via non-maximum suppression
- 50. • Non-maximum Supression (NMS) means that when bounding boxes overlap significantly, only the one with the highest score should remain. • Intersection over Union (IOU) is the most common metric for localization quality. NMS and IOU https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/single-shot-detectors/yolo.html https://www.slideshare.net/TessFerrandez/notes-from-coursera-deep-learning-courses-by-andrew-ng
- 51. YOLO/SSD • YOLO (You Only Look Once) and SSD (Single Shot Detector) scaled this approach up • Each cell in a fixed grid predicts the presence of an object centerpoint, and has several “anchor boxes” of different aspect ratios • Predict class and bounding box for each anchor box and cell, NMS predictions afterward http://cs231n.stanford.edu/
- 52. YOLO https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/single-shot-detectors/yolo.html • YOLO is nice and fast, and is in active development • Good off-the-shelf solution for many detection tasks
- 53. Dataset
- 56. Region proposal methods • There is an alternative to looking in every place in an image: look only at regions that seem interesting • Different methods, most based on some kind of image segmentation https://ivi.fnwi.uva.nl/isis/publications/bibtexbrowser.php?key=UijlingsIJCV2013 [Ujnings, 2013]
- 57. R-CNN http://cs231n.stanford.edu/ Region proposals (slow external method) [Girschick, 2014] AlexNet on regions warped to square Predicting both class and bounding box • An interim solution, bridging old-school detection methods and convents
- 58. Faster R-CNN • Insert Region Proposal Network after the last convnet layer • The proposals are then ROI-pooled, and classified • Fast, because everything is in-network!
- 59. RPN • Region Proposal Network is a fully convolutional method for scoring a bunch of candidate windows for “objectness” • At each location, it considers multiple canonical anchor boxes, and also predicts bounding box offset from each anchor (like YOLO)
- 60. Faster R-CNN Training • Four losses total: classifier and bbox regression for both RPN and the object classifier • …what if we added another loss, for segmentation? Classification Loss Bbox-regression Loss Classification Loss Bbox-regression Loss
- 61. Mask R-CNN https://github.com/wannabeOG/Mask-RCNN • ROIs are not only classified and bbox-regressed, but also go through an instance segmentation module • RoiAlign is needed to preserve exact locations (10-50% improvement)
- 62. Mask R-CNN
- 63. Mask R-CNN • Each region is masked with just a couple of non-downsampling convolutions • What if we wanted to apply that to the whole image, not just RoI’s?
- 64. Fully Convolutional Nets • Convolutions at the original scale get expensive • Encode/Decode: shrink down, then upsample • But how to upsample?
- 65. Upsampling • Unpooling: Remember which pixels max-pooling selected, and use the same when upsampling Unpooling
- 66. Upsampling • Unpooling: Remember which pixels max-pooling selected, and use the same when upsampling • Transpose convolutions: learnable upsampling Unpooling Transpose Convolutions
- 67. Upsampling • Unpooling: Remember which pixels max-pooling selected, and use the same when upsampling • Transpose convolutions: learnable upsampling • + Dilated convolutions: lose less information as you downsample Unpooling Transpose Convolutions Dilated Convolutions
- 68. Agenda 1. Tour of Convnet Architectures 2. Localization, Detection, Segmentation 3. Misc
- 69. 3D Shape Inference • Given a 2D image, detect objects and predict their 3D meshes. • Need labeled data!
- 71. Face Landmark Recognition • Given an image, predict landmarks, such as face keypoints. • Need labeled data!
- 72. Annotated Faces in the Wild https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/
- 73. Pose Estimation • Given an image, predict landmarks, such as joint locations. • Need labeled data!
- 75. Basically, any type of vision task where there is a lot of labeled training data is a good match.
- 76. Adversarial Attacks • Convnets are powerful, but brittle in surprising ways. • White-box attacks have access to model parameters; black-box don’t. • Intuition is that neural nets learned to predict the training data manifold, and are not effective when pushed off of it.
- 77. Adversarial Attacks • Can even work printed out, under a variety of viewing angles
- 78. Adversarial Attacks • Attacking • Some form of modifying the input in the direction of the loss gradient • Defending • Train with adversarial examples in the set (doesn’t work well) • Smooth the class decision boundaries • “Defensive distillation”
- 80. GANs • Will be covered in Research Directions lecture!