FPGA2018: A Lightweight YOLOv2: A binarized CNN with a parallel support vector regression for an FPGA
10 likes5,832 views
This document presents a mixed-precision convolutional neural network (CNN) called a Lightweight YOLOv2 for real-time object detection on an FPGA. The network uses binary precision for the feature extraction layers and half precision for the localization and classification layers. An FPGA implementation of the network achieves 40.81 FPS for object detection, outperforming an embedded GPU and CPU. Future work will apply this approach to other CNN-based applications such as semantic segmentation and pose estimation.
![Comparison
30
Platform Embedded CPU Embedded GPU FPGA
Device Quad‐core
ARM Cortex‐A57
256‐core
Pascal GPU
Zynq UltraScale+
MPSoC
Clock Freq. [GHz] 1.9 1.3 0.3
Memory 32 GB eMMC Flash 8GB LPDDR4 32.1 Mb BRAM
Time [msec]
(FPS) [sec‐1]
4210.0
(0.23)
715.9
(1.48*)
24.5
(40.81)
Dynamic Power [W] 4.0 7.0 4.5
Efficiency [FPS/W] 0.057 0.211 9.060
NVidia Jetson TX2 Xilinx ZCU102
* Chainer (version 1.24.0), source code: https://github.com/leetenki/YOLOv2](https://image.slidesharecdn.com/fpga2018nakahara-180228050631/85/FPGA2018-A-Lightweight-YOLOv2-A-binarized-CNN-with-a-parallel-support-vector-regression-for-an-FPGA-30-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to FPGA2018: A Lightweight YOLOv2: A binarized CNN with a parallel support vector regression for an FPGA (20)
Ad
More from Hiroki Nakahara (20)
Ad
Recently uploaded (20)
FPGA2018: A Lightweight YOLOv2: A binarized CNN with a parallel support vector regression for an FPGA
- 1. A Lightweight YOLOv2: A Binarized CNN with a Parallel Support Vector Regression for an FPGA Hiroki Nakahara, Haruyoshi Yonekawa, Tomoya Fujii, Shimpei Sato Tokyo Institute of Technology, Japan FPGA2018 @Monterey
- 2. Outline • Background • Convolutional Neural Network (CNN) • Mixed‐precision CNN for a Lightweight YOLOv2 • Binary precision CNN • Half precision support vector regression (SVR) • FPGA Implementation • Experimental Results • Conclusion 2
- 4. Convolutional Neural Network (CNN) • Convolutional + Fully connected + Pooling • State‐of‐the‐art performance in an image recognition task • Widely applicable 4Source: https://www.mathworks.com/discovery/convolutional‐neural‐network.html
- 5. Image Recognition Tasks • Classification • Answer “category” of the object in an image • Object Detection • Classification + localization 5 Easy Hard Baby (44%) Son (23%) Daughter (33%) Son Baby Daughter
- 8. Requirements in Embedded System 8 Cloud Embedded Many classes (1000s) Few classes (<10) Large workloads Frame rates (15‐30 FPS) High efficiency (Performance/W) Low cost & low power (1W‐5W) Server form factor Custom form factor J. Freeman (Intel), “FPGA Acceleration in the era of high level design”, HEART2017
- 9. Deep Learning Inference Device 9 Flexibility Power Performance Efficiency CPU (Raspberry Pi3) GPU (Jetson TX2) FPGA (UltraZed) ASIC (Movidius) • Flexibility: R&D costs for keeping on evolving algorithms • Power performance efficiency • FPGA has flexibility&better performance
- 10. Outline • Background • Convolutional Neural Network (CNN) • Mixed‐precision CNN for a Lightweight YOLOv2 • Binary precision CNN • Half precision support vector regression (SVR) • FPGA Implementation • Experimental Results • Conclusion 10
- 11. Object Detection Problem • Detecting and classifying multiple objects at the same time • Evaluation criteria (from Pascal VOC): 11 Ground truth annotation Detection results: >50% overlap of bounding box(BBox) with ground truth One BBox for each object Confidence value for each object Person (50%) # . # . # . # 1 11 , ∈ ,. ,…, Average Precision (AP):
- 12. YOLOv2 (You Only Look Once version 2) 12 Input Image (Frame) Feature maps CONV+Pooling CNN CONV+Pooling Class score Bounding Box Detection J. Redmon and A. Farhadi, “YOLO9000: Better, Faster, Stronger," arXiv preprint arXiv:1612.08242, 2016. • Single CNN (One‐shot) object detector • Both a classification and a BBox estimation for each grid
- 13. 2D Convolutional Operation 13 Input feature map Output feature map Kernel (Binary) X0,0 x W0,0 X0,1 x W0,1 X0,2 x W0,2 X1,0 x W1,0 X1,1 x W1,1 X1,2 x W1,2 X2,0 x W2,0 X2,1 x W2,1 +) X2,2 x W2,2 y • Computational intensive part of the YOLOv2
- 14. Binarized CNN 14 x1 w0 (Bias) fsgn(Y) Y z w1 x2 w2 xn wn ... x1 x2 Y ‐1 ‐1 1 ‐1 +1 ‐1 +1 ‐1 ‐1 +1 +1 1 x1 x2 Y 0 0 1 0 1 0 1 0 0 1 1 1 M. Courbariaux, I. Hubara, D. Soudry, R.E.Yaniv, Y. Bengio, “Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or ‐1," Computer Research Repository (CoRR), Mar., 2016, http://arxiv.org/pdf/1602.02830v3.pdf
- 15. Improvements by Binarization 15 x1 w0 (Bias) fsgn(Y) Y z w1 x2 w2 xn wn ... x1 x2 Y ‐1 ‐1 1 ‐1 +1 ‐1 +1 ‐1 ‐1 +1 +1 1 x1 x2 Y 0 0 1 0 1 0 1 0 0 1 1 1 EXNORs → Many MACs Binary Precision → On‐chip Memory
- 16. Near Memory Realization by Binarization E. Joel et al., “Tutorial on Hardware Architectures for Deep Neural Networks,” MICRO‐49, 2016. 16 On-chip Memory J. Dean, “Numbers everyone should know” Source: https://gist.github.com/2841832 • High bandwidth (Left) • Less power consumption (Right)
- 17. Typical CNN for Classification 17 Feature maps CONV+Pooling CONV+Pooling “5” Input image ... Feature extraction layers Classification layers 3 2 0 1 4 5 6 7 8 9
- 18. Hypothesis • Does binarized feature map has a location? → Yes 18 Feature maps CONV+Pooling CONV+Pooling “5” Input image ... Feature extraction layers Classification 3 2 0 1 4 5 6 7 8 9 Regression
- 20. Proposed YOLOv2 20 • Feature extraction layer: Binary precision • Localization and classification layer: Half precision CNN (Feature extraction) F. maps Flatten SVR Parallel SVR x y h w conf class Feature Extraction (+Location) Localization + Classification
- 21. Support Vector Regression (SVR) • Regression version of the Support Vector Machine (SVM)*1 • Passive aggressive (On‐line) training is supported*2 • Model decompression (sparse like) can be applied*3 : weight, : ‐th input, and : bias. 21 *1 H. Drucker, C. J. C. Burges, L. Kaufman, A. J. Smola and V. N. Vapnik, “Support Vector Regression Machines," Neural Information Processing Systems, No. 9, NIPS 1996, pp. 155‐161, 1997. *3 T. Downs, K. E. Gates and A. Masters, “Exact simplication of support vector solutions," Journal of Machine Learning Research, Vol. 2, 2001, pp. 293‐297. *2 M. Martin, “On‐Line Support Vector Machine Regression," ECML, pp.282‐294, 2002.
- 22. Outline • Background • Convolutional Neural Network (CNN) • Mixed‐precision CNN for a Lightweight YOLOv2 • Binary precision CNN • Half precision support vector regression (SVR) • FPGA Implementation • Experimental Results • Conclusion 22
- 23. Pipelined Conv2D Circuit x00 x01 x02 x03 x04 x10 x11 x12 x13 x14 x20 x21 x22 x23 x24 x30 x31 x32 x33 x34 x40 x41 x42 x43 x44 x22 x21 x20 x14 x13 x12 x11 x10 x04 x03 x02 x01 x00 + Binarized Weight Mem. Integer Bias Mem. Write Ctrl. Logic Counter Binarized Feature Maps (L=5, K=3) 9 Binarized MACs (EXNORs + Adder Tree) Sign bit Shift Register (2L+K bits) Read M F.Maps at a time B. Bosi, G. Bois and Y. Savaria, “Reconfigurable pipelined 2‐D convolvers for fast digital signal processing," IEEE Trans. on Very Large Scale Integration (VLSI) Systems, Vol. 7, No. 3, pp. 299‐308, 1999.
- 24. Parallel Binarized CNN Circuit 24 v v v + B. Weight Mem. Int. Bias Mem. Write Ctrl. Logic Counter Binarized MACs Sign 2L+K bits Shift Register v v Ni F. maps Read Ni bitsUpdate Ni+1 bits On Chip Memory
- 25. Parallel SVR Circuit 25 Binarized Feature map memory SVR for class_20 SVR for class_1 SVR for conf SVR for w SVR for h SVR for y SVR for x Parallel SVR Localization and Classification Feature Extracted CNN From Binarized F. Map memory -1×w w 0 1 + Reg + bias Weight cache Index Clear Circuit for a SVR
- 27. Outline • Background • Convolutional Neural Network (CNN) • Mixed‐precision CNN for a Lightweight YOLOv2 • Binary precision CNN • Half precision support vector regression (SVR) • FPGA Implementation • Experimental Results • Conclusion 27
- 28. Training Result • Environment • CNN: Proposed YOLOv2 • Binarized Darknet19+SVR • Dataset: Pascal VOC2007 • 21 class, 224x224 image size • Framework • Binary precision CNN: GUINNES*1 • Half precision SVR: Pegasos*2 • Accuracy (mAP) • 67.6% 28 *1 H. Nakahara et. al, “A demonstration of the GUINNESS: A GUI based neural NEtwork SyntheSizer for an FPGA,” FPL, 2017, page 1. https://github.com/HirokiNakahara/GUINNESS *2 S. S. Shwartz et. al, “Pegasos: primal estimated sub‐gradient solver for SVM,” Mathematical Programming, Vol . 127, No. 1, 2011, pp. 3‐30. https://github.com/ejlb/pegasos
- 29. Implementation Setup • Board: Xilinx Inc. Zynq UltraScale+ MPSoC zcu102 evaluation board • Zynq UltraScale+ MPSoC FPGA • Design tool: SDSoC 2017.4 • Timing constraint: 299.97MHz 29 Module #LUTs #FFs #18Kb BRAMs #DSP48Es Binary CNN (2D bin. Conv.) 108,138 (103,924) 358,868 (313,839) 1680 (0) 135 (0) Parallel SVR 27,243 11,431 26 242 Total (%) 135,381 (49.3) 370,299 (67.5) 1,706 (93.5) 377 (14.9)
- 30. Comparison 30 Platform Embedded CPU Embedded GPU FPGA Device Quad‐core ARM Cortex‐A57 256‐core Pascal GPU Zynq UltraScale+ MPSoC Clock Freq. [GHz] 1.9 1.3 0.3 Memory 32 GB eMMC Flash 8GB LPDDR4 32.1 Mb BRAM Time [msec] (FPS) [sec‐1] 4210.0 (0.23) 715.9 (1.48*) 24.5 (40.81) Dynamic Power [W] 4.0 7.0 4.5 Efficiency [FPS/W] 0.057 0.211 9.060 NVidia Jetson TX2 Xilinx ZCU102 * Chainer (version 1.24.0), source code: https://github.com/leetenki/YOLOv2
- 31. Conclusion • Lightweight YOLOv2 for a real‐time object detector • Mixed‐precision CNN • Binary precision CNN: Feature extraction • Half precision SVR: Classification and localization • FPGA Implementation • Outperforms an embedded GPU and a CPU • Future Work: Applied to CNN‐based applications • Single‐shot object detector (SSD, PVANet) • Semantic segmentation (FCN, U‐Net) • Pose estimation (OpenPose) • CNN SLAM 31