Mask-RCNN for Instance Segmentation
7 likes4,653 views
Mask R-CNN is an algorithm for instance segmentation that builds upon Faster R-CNN by adding a branch for predicting masks in parallel with bounding boxes. It uses a Feature Pyramid Network to extract features at multiple scales, and RoIAlign instead of RoIPool for better alignment between masks and their corresponding regions. The architecture consists of a Region Proposal Network for generating candidate object boxes, followed by two branches - one for classification and box regression, and another for predicting masks with a fully convolutional network using per-pixel sigmoid activations and binary cross-entropy loss. Mask R-CNN achieves state-of-the-art performance on standard instance segmentation benchmarks.
![RoIAlign
1024
1024
540
540
Input image
Object
64
1024/16 = 64
540/16 = 33.75
33.75
Feature map
RoI
16X less
33.75 / 7 = 4.82 each bin
7x7
Small feature map
(for each RoI)
RoI
Use bilinear interpolation to
calculate exact value at each bin
No quantization
(From [1])
FCN](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-23-320.jpg)
![Identify Feature Pyramid level for RoIs
Resize
P2
P3
P4
P5
w, h: width & height of a RoI
224: canonical ImageNet pre-training size
k0: target level of the RoI whose w*h = 2242
(here, k0 = 5)
Target level k of a RoI is identified by:
Crop the RoIs on
their feature map
Intuitions:
Features of large RoIs from smaller feature map (high semantic level)
Features of small RoIs from larger feature map (low semantic level)
RoIs
(From [6])](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-24-320.jpg)
![Mask-RCNN head network
• A classifier to identify the class for each RoI: K classes + background
• A regressor to predict the 4 values dy, dx, dh, dw for each RoI
• Fully Convolutional Network (FCN) [5] to predict mask per class
• Represent a mask as m x m matrix
• For each RoI, try to predict mask for each class
• Use sigmoid to predict how probability for each pixel
• Use binary loss to train the network](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-26-320.jpg)
![Mask-RCNN on COCO data
(From [1])](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-32-320.jpg)
![Evolution of R-CNN
= Faster R-CNN [2] + Fully Convolutional Network [5]
RoIPool RoIAlign Per-pixel softmax Per-pixel sigmoid
Mask R-CNN [1]
Faster R-CNN = Fast R-CNN [3] +
Fast R-CNN = R-CNN [4] + ConvNet on whole input image first, then apply RoIPooling layer
R-CNN [1]
Region proposal on input image + +
+=
+
+](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-33-320.jpg)
![References
[1] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. IEEE
International Conference on Computer Vision (ICCV), 2017.
[2] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection
with region proposal networks. In NIPS, 2015.
[3] R. Girshick. Fast R-CNN. In ICCV, 2015.
[4] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object
detection and semantic segmentation. In CVPR, 2014
[5] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic
segmentation. In CVPR, 2015.
[6] T.Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid
networks for object detection. In CVPR, 2017
by Nguyen Phuoc Tat Dat](https://image.slidesharecdn.com/maskrcnn-181001023652/85/Mask-RCNN-for-Instance-Segmentation-35-320.jpg)
Mask-RCNN for Instance Segmentation
- 1. ( ( 8C 8B8 ). D C / /). B8 A 09D 8 1 D C , C ). 8 9 A D 28 . -
- 2. Visual perception tasks 1 2 2. 2 3 . . 2 2. 1 2 . 1 2 4
- 3. Agenda • Visual perception tasks • Mask-RCNN • Mask-RCNN architecture • Feature Pyramid Network • Region Proposal Network • RoIAlign • Mark-RCNN head network • Result • Summary
- 4. Introduction to MaskRCNN • Mask-RCNN stands for Mask-Region Convolutional Neural Network • State-of-the-art algorithm for Instance Segmentation • Evolved through 4 main versions: • RCNN → Fast-RCNN → Faster-RCNN → Mask-RCNN • The first 3 versions are for Object Detection • Improvements over Faster RCNN: use RoIAlign instead of RoIPool • Employ Fully Convolutional Network (FCN) for mask prediction. Predict mask for each class independently • 2 main stages: • 1st stage: use Region Proposal Network (RPN) to propose candidate object bounding boxes • 2nd stage: classify the candidate boxes, refine the boxes and predict masks
- 5. Terms • Bounding box: rectangle identifying location of an object • Mask: set of pixels which belong to an object • Anchor: a bounding box is generated independently from image content • RoI: Region of Interest, a bounding box which may contain an object • Non-Maximum Suppression (NMS): a method to eliminate duplicated bounding box using scores and IoU threshold • IoU: Intersection over Union, a metrics to evaluate how 2 areas likely to be similar to each other. • RoIAlign: a method to extract features for RoIs from feature maps • Feature Pyramid Network (FPN): a neural network to extract feature maps with different scale • Region Proposal Network (RPN): a neural network to propose RoI for an image • Fully Convolutional Network (FCN): a convolution-based neural network to extract masks
- 6. MaskRCNN architecture 1 1 + + : 7 : : : :B 7 : : 1 1 +C 1 A: 1 : + : : 7 1 A: 1 : : ( 1 1 7 1 Background + number of classes) : :B 1 1 : : :B 1
- 8. Approaches for multiple scaled objects 2 . . . 2 2 . 2 .- . 2 2. -. . . 2 . 2 2 . . - 2A. . 2 .- 2A. 2 2 . - . . 1 2 :2 2 2 . 2.- : : . 29 . 2 2.- - . 2A. . 2 2 . - - . 2 . ( )
- 9. Feature Pyramid Network (FPN) ) ) ) ) 4 +4 + 3 4 6 5 4 2 53 0(1
- 10. To detect boundaries of objects: • 1 1 1 • 11 1 1 1 • 1 1 1 Bounding box regression
- 11. Bounding box regression )* ) ) * * • / DI : = D F F G : ) ) * * • AGD F F G : F D : : = & • F A IA : FD ) ) * * G- . * ) . ) (&, . * ) . ) (&, • ) ) * * F AGD F FG A A IA : FD G- ) . (&, * . ) ) . (&, * . ) / DF DF F : : DI : = D 4 2FDI : FI D 2 • 4 4 4 4 - F F : : = D F : : D 4 • 2 2 2 2 - F F : : = D =FDI : FI D 2 • 0 • : . 2 4 4 : . 2 4 4 • : . AD= 2 4 : . AD= 2 4 • : : G G A F F GA D D F D 4 • : : G AD= G F GA D G D : = D 4 • F G F F : : DI : = D 4 G DG D AD = G F =F GG D : IG : DI : = D G : : G- 4 4 4 4 2 2 2 2 1FD 5+
- 12. ) (
- 13. Region Proposal Network Anchor generator Proposal layer 4 4 4 4 4 1 4 4 4 Filter out negative anchors with Rpn_probs and Non-Max Suppression
- 14. RPN head network , F G = BC A4 :)C4 , F G 4 BC3A B3= 4 G = BC 4G /A 3AB 5C , F G 4 BC3A B3= 4 G = BC /A 355 G 1 BC 2 1 BC 2 F4= C A B 4 B B 5 : 5 G B :B CC ( B 1 2
- 15. Anchor generator ) -=38 A 4 -=38 3 = A31 1 1 1 1 ( . 1 1 /8 A 12A 3 2 61A 3== 4 = 8 122 8 6 4 = 2: 3 4 3 =0 1=38 1=38 8 1
- 16. IoU – Intersection over Union 3 I I= : 5 = = : =G ?9 G ?= :=G E = / • 4 7 1 2 2 7 • 2 7 2 • & 7 7 11 2 2 7 1 0 1 4 7 0 1 . ) ( ( 2 9BEA= 0 1 4 7 0 1 . % . % 0 1 4 7 0 1 . , , . %% = A9E = G=
- 17. Proposal layer • Sort all anchors by rpn_probs (how likely an anchor contains an object) • Choose top N anchors, throw the remainings (e.g., N ~ 6000) • Apply Non-Maximum Suppresion (NMS) to eliminate duplicated boxes. Keep up to M anchors (e.g., M ~ 2000).
- 18. Non-Maximum Suppresion (NMS) • . C: A 5 C9 C • 09 A = = C: C ?E ? ?? ? A ? ( ,> -: C A A 9 , 1 C9A 9 5 0 0) .( . 9 -: C 9 ) ?A: = 12 1?A C9 A 12 - C: C9 A : A = : : ( 9 C9 :C9 : ?A :C :=:> A = : : :C9 ,? ) 2 ?> - := = 1 A :?> ?A: =
- 19. Non-Maximum Suppresion (NMS) =A , A = 2= 5 . = 5 = 5 2= =0 A 5 =: 5 ( - - == 5 2= 5 -A =A , A = = 5 2= 5 ( > : ) > A 5 2= 5 2 = 5 ) ,== A : A 5 5 = 5 2= == 5 A 5 2= C A > :: A ) ) : 2 5 2= 5 2 C A 1 ( , 2 :A: A==> 2 > :
- 20. Train the RPN • Positive boxes: IoU >= 0.7 with any GT box • Negative boxes: IoU < 0.3 with all GT boxes • Ratio of positive boxes: 1/3 • Fixed num of anchors per image for train: 256
- 21. Loss function • i is the index of an anchor in a mini-batch • pi is the predicted probability of anchor i being an object • Ground truth label is 1 if the anchor is positive, and is 0 if the anchor is negative • ti is a vector representing the 4 parameterized coordinates (dy, dx, dh, dw) of the predicted bounding box • is that of the ground-truth box associated with a positive anchor • Classification loss Lcls is log loss over two classes (object vs. not object) • For regression loss Lreg, use , where R is smoothL1 defined as: • While both positive and negative anchors contribute to classification loss, only positive anchors contribute to regression loss. • Ncls is normalized by the mini-batch size ( ), Nreg is normalized by the number of anchor locations ( ), set
- 23. RoIAlign 1024 1024 540 540 Input image Object 64 1024/16 = 64 540/16 = 33.75 33.75 Feature map RoI 16X less 33.75 / 7 = 4.82 each bin 7x7 Small feature map (for each RoI) RoI Use bilinear interpolation to calculate exact value at each bin No quantization (From [1]) FCN
- 24. Identify Feature Pyramid level for RoIs Resize P2 P3 P4 P5 w, h: width & height of a RoI 224: canonical ImageNet pre-training size k0: target level of the RoI whose w*h = 2242 (here, k0 = 5) Target level k of a RoI is identified by: Crop the RoIs on their feature map Intuitions: Features of large RoIs from smaller feature map (high semantic level) Features of small RoIs from larger feature map (low semantic level) RoIs (From [6])
- 25. -
- 26. Mask-RCNN head network • A classifier to identify the class for each RoI: K classes + background • A regressor to predict the 4 values dy, dx, dh, dw for each RoI • Fully Convolutional Network (FCN) [5] to predict mask per class • Represent a mask as m x m matrix • For each RoI, try to predict mask for each class • Use sigmoid to predict how probability for each pixel • Use binary loss to train the network
- 27. Mask-RCNN head network architecture 7x7x256 Small feature map (for each RoI) 1024 Fully connected layer implemented by CNN Shared weights over multiple RoIs Softmax (K+1) x 4 (K+1) 14x14x256 3x3 (256 filters) Conv1 14x14x256 Conv4 14x14x256 3x3 (256 filters) Conv Transpose (Up sampling) 2x2 (256 filters) (stride 2) 28x28x256 ... x 4 conv layers Conv 28x28x(K+1) 1x1 (K+1 filters) Sigmoid activation 28x28x(K+1) 7x7 (1024 filters) Conv1 Conv2 (BG + num classes) K+1 Dense Dense (K+1) x 4 1024 K+1 Predict mask per class BG vs K classes 4 box regression values: dy, dx, dh, dw 1x1 (1024 filters)
- 28. Loss functions • For each sampled RoI, a multi-task loss is applied: where • Lcls is classification loss • Lloc is bounding-box regression loss • Lmask is mask loss • The final loss is calculated as mean of loss over samples
- 29. Classification loss Lcls • For a RoI, denotes: • : true class of the RoI • : predicted probability distribution over K+1 classes • The classification loss Lcls for a RoI is a log-loss calculated as:
- 30. Bounding-box regression loss Lloc • For a RoI, denotes: • : true class of the RoI • : true bounding-box regression targets of the RoI • : predicted bounding-box regression for the class u. • The bounding-box regression loss Lloc for the RoI is calculated as: where
- 31. Mask loss Lmask • For a RoI, denotes: • : true class of the RoI • : the true mask and the predicted mask for the class of the RoI respectively ( ) • The mask loss Lmask for the RoI is the average binary cross-entropy loss, calculated as:
- 32. Mask-RCNN on COCO data (From [1])
- 33. Evolution of R-CNN = Faster R-CNN [2] + Fully Convolutional Network [5] RoIPool RoIAlign Per-pixel softmax Per-pixel sigmoid Mask R-CNN [1] Faster R-CNN = Fast R-CNN [3] + Fast R-CNN = R-CNN [4] + ConvNet on whole input image first, then apply RoIPooling layer R-CNN [1] Region proposal on input image + + += + +
- 34. Summary • Introduced MaskRCNN, an algorithm for Instance Segmentation • Detect both bounding boxes and masks of objects in an end-to-end neural network • Improve RoIPool from Faster-RCNN with RoIAlign • Employ Fully Convolutional Network for mask detection
- 35. References [1] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. IEEE International Conference on Computer Vision (ICCV), 2017. [2] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015. [3] R. Girshick. Fast R-CNN. In ICCV, 2015. [4] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014 [5] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. [6] T.Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017 by Nguyen Phuoc Tat Dat
- 36. Appendix: Some popular DL-based algorithms for visual perception tasks by Nguyen Phuoc Tat Dat Visual perception tasks Algorithms Image Classification AlexNet Inception GooLeNet/Inception v1 ResNet VGGNet Object Detection Fast/Faster R-CNN SSD YOLO Semantic Segmentation Fully Convolutional Network (FCN) U-Net Instance Segmentation Mask R-CNN
- 37. Thank you for listening! !