![PolyLoss: A POLYNOMIAL EXPANSION PERSPECTIVE
OF CLASSIFICATION LOSS FUNCTIONS[1]
▪ 著者：
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang
Cheng, Dragomir Anguelov (Waymo LLC, Google LLC)
▪ ICLR 2022
▪ 一言
▪ PolyLossという新しいフレームワークで分類損失関数を理解し設計する
4](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-4-320.jpg)
![17
Dataset: IMAGENET[2]-1K, IMAGENET-21K
Model: EfficientNetV2[3]
★ 𝜖 が増えるほど（最初の項の係数が小さいほど）Accuracyを向上
★ 𝜖 = 1時は予測自信度を向上、ImageNet-21Kの自信不足を改善
2D CLASSIFICATION](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-17-320.jpg)
![18
Dataset: COCO Dataset[4]
Model: Mask R-CNN[5] (𝐿𝑀𝑎𝑠𝑘𝑅𝐶𝑁𝑁 = 𝐿𝑐𝑙𝑠 + 𝐿𝑏𝑜𝑥 + 𝐿𝑚𝑎𝑠𝑘の𝐿𝑐𝑙𝑠だけ置換え）
★ 𝜖 が減らすほど（最初の項の係数が小さいほど）Mask R-CNNのAPとARを向上
★ 𝜖 = −1時過度自信の予測を低下させ、不均衡データセットでの性能を改善
2D INSTANCE SEGMENTATION & OBJECT DETECTION](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-18-320.jpg)
![19
Dataset: WAYMO Open Dataset[6]
Model: PointPillars[7], Range Sparse Net(RSN)[8]
3D OBJECT DETECTION](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-19-320.jpg)
![20
[1] Zhaoqi Leng, Mingxing Tan ~Mingxing_Tan3 , Chenxi Liu, Ekin Dogus Cubuk, Jay
Shi, Shuyang Cheng, Dragomir Anguelov. PolyLoss: A Polynomial Expansion
Perspective of Classification Loss Functions. In ICLR 2022.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A
large-scale hierarchical image database. In 2009 IEEE conference on computer vision
and pattern recognition, pp. 248–255. Ieee, 2009.
[3] Mingxing Tan and Quoc V Le. Efficientnetv2: Smaller models and faster training.
In International Conference on Machine Learning, 2021.
[4] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in
context. In ´ European conference on computer vision, pp. 740–755. Springer, 2014.
[5] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ´
Proceedings of the IEEE international conference on computer vision, pp. 2961–2969,
2017.
Reference](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-20-320.jpg)
![21
[6] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai
Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.
Scalability in perception for autonomous driving: Waymo open dataset. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 2446–2454, 2020.
[7] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar
Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 12697–12705, 2019.
[8] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang,
Christian Sminchisescu, and Dragomir Anguelov. Rsn: Range sparse net for efficient,
accurate lidar 3d object detection. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2021.
Reference](https://image.slidesharecdn.com/20220513cvplotpolyloss-220606043410-5ac15ab6/85/PolyLoss-A-POLYNOMIAL-EXPANSION-PERSPECTIVE-OF-CLASSIFICATION-LOSS-FUNCTIONS-21-320.jpg)
![[DL輪読会]Non-Autoregressive Machine Translation with Latent Alignments](https://cdn.slidesharecdn.com/ss_thumbnails/20200522matsudatefinal-200529045007-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Focal Loss for Dense Object Detection](https://cdn.slidesharecdn.com/ss_thumbnails/focalloss-180208092846-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...](https://cdn.slidesharecdn.com/ss_thumbnails/20210806journalclub-210806023711-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://cdn.slidesharecdn.com/ss_thumbnails/swintransformer-210514020542-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]A Higher-Dimensional Representation for Topologically Varying Neural R...](https://cdn.slidesharecdn.com/ss_thumbnails/ahigher-dimensionalrepresentationfortopologicallyvaryingneuralradiancefields1-210924021911-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to PolyLoss: A POLYNOMIAL EXPANSION PERSPECTIVE OF CLASSIFICATION LOSS FUNCTIONS 論文紹介 (20)
![SSII2022 [OS3-02] Federated Learningの基礎と応用](https://cdn.slidesharecdn.com/ss_thumbnails/ssii2022-os3-02-220607020834-2b5f93ff-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Plot Hong (8)
Ad
PolyLoss: A POLYNOMIAL EXPANSION PERSPECTIVE OF CLASSIFICATION LOSS FUNCTIONS 論文紹介
- 1. 2022.05.13 洪 嘉源 株式会社 Mobility Technologies PolyLoss: A POLYNOMIAL EXPANSION PERSPECTIVE OF CLASSIFICATION LOSS FUNCTIONS 論文紹介
- 2. 2 Agenda 01｜概要 02｜PolyLoss & CE Loss & Focal Loss 03｜多項式係数調整 04｜実験分析
- 3. 3 01 概要
- 4. PolyLoss: A POLYNOMIAL EXPANSION PERSPECTIVE OF CLASSIFICATION LOSS FUNCTIONS[1] ▪ 著者： Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, Dragomir Anguelov (Waymo LLC, Google LLC) ▪ ICLR 2022 ▪ 一言 ▪ PolyLossという新しいフレームワークで分類損失関数を理解し設計する 4
- 5. ▪ Polyloss のフレームワークで損失関数を研究する。Cross-entropy Loss とFocal Loss もPolyloss の特例としてみなす ▪ Polyloss の多項式係数調整について分析し、ハイパーパラメータが一個のみのシンプル なPoly-1 Loss を提案 ▪ Cross-entropy Loss とFocal Lossの弱点を分析、不均衡なデータセットで有効な Polylossの設計について考案 ▪ 各種のタスク・モデル・データセットでPolyLossを実験し、性能改善している 5 貢献
- 6. 6 02 PolyLoss & CE Loss & Focal Loss
- 7. Cross-entropy Loss 𝐿𝐶𝐸 = − log 𝑃𝑡 Focal Loss 𝐿𝐹𝐿 = − 1 − 𝑃𝑡 𝛾 log 𝑃𝑡 ※ 𝑃𝑡は目標クラスの予測確率 上記を 1 − 𝑃𝑡 ベースでTaylor展開 ↓ Cross-entropy Loss 𝐿𝐶𝐸 = − log 𝑃𝑡 = 𝑗=1 ∞ 1 𝑗 (1 − 𝑃𝑡)𝑗 = 1 − 𝑃𝑡 + 1 2 (1 − 𝑃𝑡)2 … Focal Loss 𝐿𝐹𝐿 = − 1 − 𝑃𝑡 𝛾 log 𝑃𝑡 = 𝑗=1 ∞ 1 𝑗 1 − 𝑃𝑡 𝑗+𝛾 = (1 − 𝑃𝑡)1+𝛾 + 1 2 (1 − 𝑃𝑡)2+𝛾 … 7 Cross-entropy Loss & Focal Loss
- 8. 勾配降下法で損失を最適化する時は𝑃𝑡 に対して勾配を求める Cross-entropy Lossは定数1の項があって、 Focal Lossの方はそれをなくしている。 𝑃𝑡 が1に近くなる場合は 1 − 𝑃𝑡 𝛾 はγによって 抑制されて、Majority Classでのoverfitを 避ける 8 Cross-entropy Loss & Focal Loss 最初の1項をドロップ 最初の2項をドロップ
- 9. 𝐿𝑃𝐿 = 𝑗=1 ∞ 𝛼𝑗 (1 − 𝑃𝑡)𝑗= 𝛼1 1 − 𝑃𝑡 + 𝛼2(1 − 𝑃𝑡)2… , where 𝛼𝑗 ∈ 𝑅+ メリット： 1. この形は各種タスクによって𝛼𝑗を調整できる 2. フレキシブルに係数を調整できる 9 PolyLoss
- 10. 𝐿𝑃𝐿 = 𝑗=1 ∞ 𝛼𝑗 (1 − 𝑃𝑡)𝑗= 𝛼1 1 − 𝑃𝑡 + 𝛼2(1 − 𝑃𝑡)2… , where 𝛼𝑗 ∈ 𝑅+ 分類タスクの中、多項式の中の 1 − 𝑃𝑡 の1はGTの確率y=1とみなせ、 (1 − 𝑃𝑡)𝑗は(y − 𝑃𝑡)𝑗と表示できる ↓ Cross-entropy Loss & Focal Lossは予測とGTの距離のj次の加重アンサ ンブルと解釈できる 10 PolyLossと回帰の関係
- 11. 11 03 多項式係数調整
- 12. PolyLossのハイパーパラメータの探索空間を減らすため、 論文の中ではCross-entropy Lossの多項式の係数調整の方法について三 つ考察する ①𝐿𝐷𝑟𝑜𝑝: 高次の項をドロップする ②𝐿𝑃𝑂𝐿𝑌−𝑁: 前のN項の係数を調整する ③𝐿𝑃𝑂𝐿𝑌−1: 最初の項の係数を調整する 12 多項式係数の調整
- 13. 13 𝐿𝐷𝑟𝑜𝑝 = 𝑗=1 𝑁 𝛼𝑗 (1 − 𝑃𝑡)𝑗 特に学習の初期で、 𝑃𝑡が0に近い時、高次の項が学習に大きく影響する 例えば𝑃𝑡~ 0.001時、第500項の勾配は0.999499 ~ 0.6 ※なぜ高次の項が重要なのか論文の中では数学的な証明がある ①高次の項をドロップ 少なくとも600項を残す必要がある
- 16. 16 04 実験分析
- 17. 17 Dataset: IMAGENET[2]-1K, IMAGENET-21K Model: EfficientNetV2[3] ★ 𝜖 が増えるほど（最初の項の係数が小さいほど）Accuracyを向上 ★ 𝜖 = 1時は予測自信度を向上、ImageNet-21Kの自信不足を改善 2D CLASSIFICATION
- 18. 18 Dataset: COCO Dataset[4] Model: Mask R-CNN[5] (𝐿𝑀𝑎𝑠𝑘𝑅𝐶𝑁𝑁 = 𝐿𝑐𝑙𝑠 + 𝐿𝑏𝑜𝑥 + 𝐿𝑚𝑎𝑠𝑘の𝐿𝑐𝑙𝑠だけ置換え） ★ 𝜖 が減らすほど（最初の項の係数が小さいほど）Mask R-CNNのAPとARを向上 ★ 𝜖 = −1時過度自信の予測を低下させ、不均衡データセットでの性能を改善 2D INSTANCE SEGMENTATION & OBJECT DETECTION
- 19. 19 Dataset: WAYMO Open Dataset[6] Model: PointPillars[7], Range Sparse Net(RSN)[8] 3D OBJECT DETECTION
- 20. 20 [1] Zhaoqi Leng, Mingxing Tan ~Mingxing_Tan3 , Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, Dragomir Anguelov. PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions. In ICLR 2022. [2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009. [3] Mingxing Tan and Quoc V Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, 2021. [4] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ´ European conference on computer vision, pp. 740–755. Springer, 2014. [5] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ´ Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017. Reference
- 21. 21 [6] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454, 2020. [7] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12697–12705, 2019. [8] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Christian Sminchisescu, and Dragomir Anguelov. Rsn: Range sparse net for efficient, accurate lidar 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. Reference