SlideShare a Scribd company logo

A CGRA-based Approach for Accelerating Convolutional Neural Networks

•
Shinya Takamaeda-Y, profile picture
Shinya Takamaeda-Y

The document presents an approach for accelerating convolutional neural networks (CNNs) using a coarse-grained reconfigurable array (CGRA) called EMAX. EMAX features processing elements with local memory to improve data locality and memory bandwidth utilization. CNN computations like convolutions are mapped to EMAX by assigning weight matrices to constant registers and performing numerous small matrix multiplications in parallel. Evaluation shows EMAX achieves better performance per memory bandwidth and area than GPUs for CNN workloads due to its optimization for small matrix operations.

Technology
Related topics:
Deep Learning
1 of 37
Downloaded 76 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
A CGRA-based Approach for Accelerating Convolutional Neural Networks Masakazu Tanomoto, Shinya Takamaeda-Yamazaki, Jun Yao, and Yasuhiko Nakashima Nara Institute of Science and Technology (NAIST), Japan E-mail: shinya_at_is_naist_jp IEEE MCSoC'15 @Torino September 23, 2015
Outline n  Motivation: Deep learning on embedded computers l  Target: Convolutional Neural Network (CNN) n  Our approach: CGRA-based CNN acceleration l  EMAX (Energy-aware Multi-mode Accelerator eXtension) l  Mapping CNN on EMAX n  Evaluation l  Performance per memory bandwidth l  Performance per area n  Conclusion MCSoC15 Shinya T-Y, NAIST 2
Deep learning n  Recognition (Convolutional Neural Network (CNN)) l  Extracting high-level features automatically from raw data l  Ex) Image, speech, and text recognition, image search n  Reinforcement learning (Deep Q-Network (DQN)) l  Learning appropriate strategy for controlling something l  Ex) Gaming AI, Robot control MCSoC15 Shinya T-Y, NAIST 3 Playing Atari 2006 automatically (Human-level control through deep reinforcement learning [Nature'15]) Extracted features of human and cat (Building High-level Features Using Large Scale Unsupervised Learning [ICML'12])
Convolutional Neural Network (CNN) n  Nesting multiple processing layers l  Convolution: Multiple small matrix-matrix multiplications •  Each weight matrix corresponds to a learned feature map •  Feature can be automatically learned by error propagation l  Pooling and Max-out: selection from multiple values l  Full connection: Large matrix-matrix multiplication n  Performance Bottleneck: Convolution l  Numerous small matrix-matrix multiplication with stencil MCSoC15 Shinya T-Y, NAIST 4 Input Layer Hidden Layers Output Layer Convolution Pooling Max Out Convolution Full Connection
Motivation: DNN on embedded computers n  Machine learning on IoT: Learning and decision on edge computers will become more important l  Sending all data to data centers?: Network traffic problemL l  Decision on data centers?: Very long latencyL n  Challenge: Energy efficient embedded accelerators l  Why not GPU?: GPU is very energy hungry and requires absolute energy •  Not only energy efficiency, but also absolute peak energy amount is important l  Why not ASIC?: Limited capability of algorithm customization •  Algorithms of machine learning are rapidly evolving l  Why not FPGA?: Energy overhead to building computing logics l  CGRA? MCSoC15 Shinya T-Y, NAIST 5
Computation pattern: Full connection n  Output vector is determined by a simple vector-matrix multiplication l  Input and output size is certainly large: more than 1024 l  Weight matrix size is also large n  GPU is OK: suitable for matrix multiplication l  GPU has matrix libraries: CUBLAS, ... MCSoC15 Shinya T-Y, NAIST 6 dot =Weight Output Vector Input Vector
Computation pattern: Convolution n  A value of the result matrix is calculated by numerous matrix-matrix multiplication with a small weight matrix l  Weight matrix size is usually small: from 3 to 8 n  I know GPU is very fast for matrix-matrix multiplication l  Really? MCSoC15 Shinya T-Y, NAIST 7 3 dot Weight WeightWeight = Weight Weight Nextdot Weight WeightWeight dot Weight WeightWeight
SGEMM performance on GPU n  GPU is fast, if the matrix size is large enough l  GPU is throughput-oriented processor l  In case of small matrix, parallelisms and memory bandwidth are not exploited efficiently MCSoC15 Shinya T-Y, NAIST 8 0 5 10 15 20 25 0 50 100 150 200 250 64 128 256 512 1024 2048 4096 #activewarpsperactivecycle Performance[GFLOPS] Matrix size warp/cycle (small kernel) GFLOPS warp/cycle (large kernel) NVIDIA Jetson TK1 (GK20A)
Preprocessing for Convolution on GPU n  In order to use fast matrix multiplication library of GPU, data duplication is usually utilized l  Converting sub-regions into a single large matrix n  Faster than the naive convolution, but still just a performance overhead MCSoC15 Shinya T-Y, NAIST 9 3 k=3 k=3 n n Input vector [n-3, n-3] [n-1, n-1] Duplication Duplication [0,0] [0,1] [0,2] [1,0] [1,1] [1,2] [2,0] [2,1] [2,2] [0,1] [0,2] [0,3] [1,1] [1,2] [1,3] [2,1] [2,2] [2,3] Duplication 9 (=k2) (n-2)2 Temporal array for matrix multiplication
Our approach: EMAX Energy-aware Multi-mode Accelerator eXtension n  A CGRA of local memory based PEs with several buses l  Each PE has a local memory for data locality MCSoC15 Shinya T-Y, NAIST 10 Interconnection DRAM CPU Core PE PE PE PE MemoryInterface EMAX PE PE PE PE PE PE PE PE
Real Chip of EMAX n  12.5mm x 12.5mm in 180nm technology MCSoC15 Shinya T-Y, NAIST 11
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 12 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 13 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const LMM: Local Memory
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 14 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const FIFO
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 15 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Execution units
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 16 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Constant registers
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 17 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Internal Shuffle Bus External Shuffle Bus
Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 18 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Memory Bus
EMAX instruction MCSoC15 Shinya T-Y, NAIST 19 Type1: row#, col#, dist [count] ALU_OP & MEM_OP RGI LMM_CONTROL Type2: row#, col#, dist [count] ALU_OP Type3: row#, col#, dist [count] & MEM_OP RGI LMM_CONTROL 32-bit operation add/add3/sub/sub3 16-bitx2 operation mauh/mauh3/msuh3 Misc operation mulh/mmrg3/msad/minl/minl3/mh2bw/ mcas/mmid3/mmax/mmax3/mmin/mmin3 Load from EX_FIFO ldb/ldub/ldh/lhuh/ld Floating Point Operation fmul/fma3/fadd 32-bit operation and/or/xor 16-bitx2 operation mauh/mauh3/msuh3 Load from LMM or LMM_FIFO ldb/ldub/ldh/lhuh/ld Store to LMM stb/sth/st/cst (a) Instruction format (a) EX1 operation (b) EX2 operation (c) LMM operation
Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 20 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 21 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 22 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 23 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 24 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil 3x3 weight matrix in constant registers
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 25 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil 3 Input data sets in LMMs Same read data is forwarded via FIFOs
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 26 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Reading data from the constant register, LMM, and execution unit in the previous stage Operation result is passed to the next stage
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 27 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Final result is stored into LMM in the next stage
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 28 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Write back the previous data to the main memory
CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 29 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Load the next input data from the main memory
Evaluation setup n  Benchmark: deep learning datasets and networks l  Imagenet (Alexnet-2), CIFAR10, MNIST (Lenet) n  Hardware: l  CPU (Corei7, ARM), GPU (Desktop, Mobile), EMAX l  Metric: Performance per bandwidth, Performance per area •  Estimation from actual LSI of EMAX and software simulations MCSoC15 Shinya T-Y, NAIST 30
Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 31 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM
Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 32 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM Alexnet: since matrix size is large, desktop GPU is 3.17x better
Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 33 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM CIFAR-10: 1.41x better than mobile GPU Lenet: 1.75x better than mobile GPU
Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 34 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7
Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 35 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7 Alexnet: since matrix size is large, desktop GPU is 2.2x better
Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 36 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7 CIFAR-10: 1.76x better than mobile GPU Lenet: 1.95x better than mobile GPU
Conclusion n  A CGRA-based acceleration approach of convolutional neural network (CNN) for embedded accelerators l  EMAX (Energy-aware Multi-mode Accelerator eXtension) n  EMAX outperforms GPU in embedded class data sets l  1.75x better performance per memory bandwidth l  1.95x better performance per area ( energy) MCSoC15 Shinya T-Y, NAIST 37 Interconnection DRAM CPU Core PE PE PE PE MemoryInterface EMAX PE PE PE PE PE PE PE PE

More Related Content

PDF
A short Commentary on Quantum-dot Cellular Automata (QCA)
Soudip Sinha Roy
 
PPTX
Power density Spectrum
Saifur Rahman
 
PDF
Verilog 모듈 연결하기
Jihyun Lee
 
PPTX
Digital VLSI Design and FPGA Implementation
Amber Bhaumik
 
PPT
Pass Transistor Logic
Diwaker Pant
 
PPTX
Avalanche photodiode &amp; there bandwidth
SHARMAGOLU
 
PPTX
MOSFET fabrication 12
HIMANSHU DIWAKAR
 
PDF
FPGA and CPLD comparision
Engineering Funda
 
A short Commentary on Quantum-dot Cellular Automata (QCA)
Soudip Sinha Roy
 
Power density Spectrum
Saifur Rahman
 
Verilog 모듈 연결하기
Jihyun Lee
 
Digital VLSI Design and FPGA Implementation
Amber Bhaumik
 
Pass Transistor Logic
Diwaker Pant
 
Avalanche photodiode &amp; there bandwidth
SHARMAGOLU
 
MOSFET fabrication 12
HIMANSHU DIWAKAR
 
FPGA and CPLD comparision
Engineering Funda
 

What's hot (20)

PPTX
Unit no. 5 cmos logic design
swagatkarve
 
PDF
Energy harvesting sensor nodes
ahmad abdelhafeez
 
PPTX
Finfet
Aditya Singh
 
PPT
Chap6 photodetectors
Arturo Gonzales Ccama
 
PDF
Solved problems in waveguides
subhashinivec
 
PDF
Lect12 photodiode detectors
wtyru1989
 
PPTX
Physical Design Flow Challenges at 28nm on Multi-million Gate Blocks
eInfochips (An Arrow Company)
 
PPTX
Simulation power analysis low power vlsi
GargiKhanna1
 
PPTX
Digital communication viva questions
ishan0019
 
PDF
M-ary Modulation, noise modelling, bandwidth, Bandpass Modulation
DrAimalKhan
 
PPTX
Reduced ordered binary decision diagram
Team-VLSI-ITMU
 
PPTX
FPGA implementation of synchronous and asynchronous counter and simulation of...
ASHIMA GUPTA
 
PPTX
PIC-MICROCONTROLLER TUTORIALS FOR BEGINNERS
VISHNU KP
 
PPT
Low power vlsi design
Vinchipsytm Vlsitraining
 
PPT
Pass transistor logic
Tripurna Chary
 
PDF
Polyteda: Power DRC/LVS, October 2016
Oleksandra Nazola
 
PDF
DFT and IDFT Matlab Code
Bharti Airtel Ltd.
 
PPTX
Ic technology- diffusion and ion implantation
kriticka sharma
 
PDF
Embedded Systems Syllabus
GR Techno Solutions
 
PPTX
VIVADO PPT.pptx
SouravRoyElectronics
 
Unit no. 5 cmos logic design
swagatkarve
 
Energy harvesting sensor nodes
ahmad abdelhafeez
 
Finfet
Aditya Singh
 
Chap6 photodetectors
Arturo Gonzales Ccama
 
Solved problems in waveguides
subhashinivec
 
Lect12 photodiode detectors
wtyru1989
 
Physical Design Flow Challenges at 28nm on Multi-million Gate Blocks
eInfochips (An Arrow Company)
 
Simulation power analysis low power vlsi
GargiKhanna1
 
Digital communication viva questions
ishan0019
 
M-ary Modulation, noise modelling, bandwidth, Bandpass Modulation
DrAimalKhan
 
Reduced ordered binary decision diagram
Team-VLSI-ITMU
 
FPGA implementation of synchronous and asynchronous counter and simulation of...
ASHIMA GUPTA
 
PIC-MICROCONTROLLER TUTORIALS FOR BEGINNERS
VISHNU KP
 
Low power vlsi design
Vinchipsytm Vlsitraining
 
Pass transistor logic
Tripurna Chary
 
Polyteda: Power DRC/LVS, October 2016
Oleksandra Nazola
 
DFT and IDFT Matlab Code
Bharti Airtel Ltd.
 
Ic technology- diffusion and ion implantation
kriticka sharma
 
Embedded Systems Syllabus
GR Techno Solutions
 
VIVADO PPT.pptx
SouravRoyElectronics
 
Ad

Viewers also liked (17)

PDF
PyCoRAM (高位合成友の会@ドワンゴ, 2015年1月16日)
Shinya Takamaeda-Y
 
PDF
An FPGA-based Scalable Simulation Accelerator for Tile Architectures @HEART2011
Shinya Takamaeda-Y
 
PDF
PyCoRAMによるPythonを用いたポータブルなFPGAアクセラレータ開発 (チュートリアル@ESS2014)
Shinya Takamaeda-Y
 
PDF
助教が吼える！ 各界の若手研究者大集合「ハードウェアはやわらかい」
Shinya Takamaeda-Y
 
PDF
Pythonによる高位設計フレームワークPyCoRAMでFPGAシステムを開発してみよう
Shinya Takamaeda-Y
 
PDF
Mapping Applications with Collectives over Sub-communicators on Torus Network...
Shinya Takamaeda-Y
 
PDF
Veriloggen: Pythonによるハードウェアメタプログラミング（第3回 高位合成友の会 ＠ドワンゴ）
Shinya Takamaeda-Y
 
PDF
マルチパラダイム型高水準ハードウェア設計環境の検討
Shinya Takamaeda-Y
 
PDF
PythonとPyCoRAMでお手軽にFPGAシステムを開発してみよう
Shinya Takamaeda-Y
 
PDF
PyCoRAMを用いたグラフ処理FPGAアクセラレータ
Shinya Takamaeda-Y
 
PDF
Pythonを用いた高水準ハードウェア設計環境の検討
Shinya Takamaeda-Y
 
PDF
PythonとVeriloggenを用いたRTL設計メタプログラミング
Shinya Takamaeda-Y
 
PDF
コンピュータアーキテクチャ研究の最新動向〜ISCA2015参加報告〜 ＠FPGAエクストリーム・コンピューティング 第7回 (#fpgax #7)
Shinya Takamaeda-Y
 
PDF
Zynq+PyCoRAM(+Debian)入門
Shinya Takamaeda-Y
 
PPTX
Zynq + Vivado HLS入門
narusugimoto
 
PDF
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Shinya Takamaeda-Y
 
PDF
FPGA・リコンフィギャラブルシステム研究の最新動向
Shinya Takamaeda-Y
 
PyCoRAM (高位合成友の会@ドワンゴ, 2015年1月16日)
Shinya Takamaeda-Y
 
An FPGA-based Scalable Simulation Accelerator for Tile Architectures @HEART2011
Shinya Takamaeda-Y
 
PyCoRAMによるPythonを用いたポータブルなFPGAアクセラレータ開発 (チュートリアル@ESS2014)
Shinya Takamaeda-Y
 
助教が吼える！ 各界の若手研究者大集合「ハードウェアはやわらかい」
Shinya Takamaeda-Y
 
Pythonによる高位設計フレームワークPyCoRAMでFPGAシステムを開発してみよう
Shinya Takamaeda-Y
 
Mapping Applications with Collectives over Sub-communicators on Torus Network...
Shinya Takamaeda-Y
 
Veriloggen: Pythonによるハードウェアメタプログラミング（第3回 高位合成友の会 ＠ドワンゴ）
Shinya Takamaeda-Y
 
マルチパラダイム型高水準ハードウェア設計環境の検討
Shinya Takamaeda-Y
 
PythonとPyCoRAMでお手軽にFPGAシステムを開発してみよう
Shinya Takamaeda-Y
 
PyCoRAMを用いたグラフ処理FPGAアクセラレータ
Shinya Takamaeda-Y
 
Pythonを用いた高水準ハードウェア設計環境の検討
Shinya Takamaeda-Y
 
PythonとVeriloggenを用いたRTL設計メタプログラミング
Shinya Takamaeda-Y
 
コンピュータアーキテクチャ研究の最新動向〜ISCA2015参加報告〜 ＠FPGAエクストリーム・コンピューティング 第7回 (#fpgax #7)
Shinya Takamaeda-Y
 
Zynq+PyCoRAM(+Debian)入門
Shinya Takamaeda-Y
 
Zynq + Vivado HLS入門
narusugimoto
 
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Shinya Takamaeda-Y
 
FPGA・リコンフィギャラブルシステム研究の最新動向
Shinya Takamaeda-Y
 
Ad

Similar to A CGRA-based Approach for Accelerating Convolutional Neural Networks (20)

PDF
“New Methods for Implementation of 2-D Convolution for Convolutional Neural N...
Edge AI and Vision Alliance
 
PDF
Intro to Machine Learning for GPUs
Sri Ambati
 
PDF
Hardware for Deep Learning AI ML CNN.pdf
AhmedSaeed115917
 
PDF
Trip down the GPU lane with Machine Learning
Renaldas Zioma
 
PPTX
B.tech_project_ppt.pptx
supratikmondal6
 
PPTX
DeepLearningAlgorithmAccelerationOnHardwarePlatforms_V2.0
Sahil Kaw
 
PDF
[05][cuda 및 fermi 최적화 기술] hryu optimization
laparuma
 
PDF
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
Edge AI and Vision Alliance
 
PDF
"Using SGEMM and FFTs to Accelerate Deep Learning," a Presentation from ARM
Edge AI and Vision Alliance
 
PDF
(Im2col)accelerating deep neural networks on low power heterogeneous architec...
Bomm Kim
 
PDF
Survey_Report_Deep Learning Algorithm
Sahil Kaw
 
PDF
Intel's Presentation in SIGGRAPH OpenCL BOF
Ofer Rosenberg
 
PPTX
Jeff Johnson, Research Engineer, Facebook at MLconf NYC
MLconf
 
PDF
customization of a deep learning accelerator, based on NVDLA
Shien-Chun Luo
 
PPTX
APSys Presentation Final copy2
Junli Gu
 
PDF
Adaptive Linear Solvers and Eigensolvers
inside-BigData.com
 
PDF
Keynote (Mike Muller) - Is There Anything New in Heterogeneous Computing - by...
AMD Developer Central
 
PDF
Deep Convolutional Neural Network acceleration on the Intel Xeon Phi
Gaurav Raina
 
PDF
Deep Convolutional Network evaluation on the Intel Xeon Phi
Gaurav Raina
 
PDF
Thesis Report - Gaurav Raina MSc ES - v2
Gaurav Raina
 
“New Methods for Implementation of 2-D Convolution for Convolutional Neural N...
Edge AI and Vision Alliance
 
Intro to Machine Learning for GPUs
Sri Ambati
 
Hardware for Deep Learning AI ML CNN.pdf
AhmedSaeed115917
 
Trip down the GPU lane with Machine Learning
Renaldas Zioma
 
B.tech_project_ppt.pptx
supratikmondal6
 
DeepLearningAlgorithmAccelerationOnHardwarePlatforms_V2.0
Sahil Kaw
 
[05][cuda 및 fermi 최적화 기술] hryu optimization
laparuma
 
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
Edge AI and Vision Alliance
 
"Using SGEMM and FFTs to Accelerate Deep Learning," a Presentation from ARM
Edge AI and Vision Alliance
 
(Im2col)accelerating deep neural networks on low power heterogeneous architec...
Bomm Kim
 
Survey_Report_Deep Learning Algorithm
Sahil Kaw
 
Intel's Presentation in SIGGRAPH OpenCL BOF
Ofer Rosenberg
 
Jeff Johnson, Research Engineer, Facebook at MLconf NYC
MLconf
 
customization of a deep learning accelerator, based on NVDLA
Shien-Chun Luo
 
APSys Presentation Final copy2
Junli Gu
 
Adaptive Linear Solvers and Eigensolvers
inside-BigData.com
 
Keynote (Mike Muller) - Is There Anything New in Heterogeneous Computing - by...
AMD Developer Central
 
Deep Convolutional Neural Network acceleration on the Intel Xeon Phi
Gaurav Raina
 
Deep Convolutional Network evaluation on the Intel Xeon Phi
Gaurav Raina
 
Thesis Report - Gaurav Raina MSc ES - v2
Gaurav Raina
 

More from Shinya Takamaeda-Y (15)

PDF
オープンソースコンパイラNNgenでつくるエッジ・ディープラーニングシステム
Shinya Takamaeda-Y
 
PDF
DNNのモデル特化ハードウェアを生成するオープンソースコンパイラNNgenのデモ
Shinya Takamaeda-Y
 
PDF
ディープニューラルネットワーク向け拡張可能な高位合成コンパイラの開発
Shinya Takamaeda-Y
 
PDF
Veriloggen.Stream: データフローからハードウェアを作る（2018年3月3日 高位合成友の会 第5回 @東京工業大学）
Shinya Takamaeda-Y
 
PDF
Veriloggen.Thread & Stream: 最高性能FPGAコンピューティングを 目指したミックスドパラダイム型高位合成 (FPGAX 201...
Shinya Takamaeda-Y
 
PDF
Pythonによるカスタム可能な高位設計技術 (Design Solution Forum 2016@新横浜)
Shinya Takamaeda-Y
 
PDF
ゆるふわコンピュータ (IPSJ-ONE2017)
Shinya Takamaeda-Y
 
PDF
A Framework for Efficient Rapid Prototyping by Virtually Enlarging FPGA Resou...
Shinya Takamaeda-Y
 
PDF
A High Performance Heterogeneous FPGA-based Accelerator with PyCoRAM (Runner ...
Shinya Takamaeda-Y
 
PDF
PyCoRAM: Python-Verilog高位合成とメモリ抽象化によるFPGAアクセラレータ向けIPコア開発フレームワーク (FPGAX #05)
Shinya Takamaeda-Y
 
PDF
メモリ抽象化フレームワークPyCoRAMを用いたソフトプロセッサ混載FPGAアクセラレータの開発
Shinya Takamaeda-Y
 
PDF
PyCoRAM: Yet Another Implementation of CoRAM Memory Architecture for Modern F...
Shinya Takamaeda-Y
 
PDF
むかし名言集bot作りました！
Shinya Takamaeda-Y
 
PDF
APGAS言語X10を用いたオンチップネットワークシミュレーションの並列化
Shinya Takamaeda-Y
 
PDF
Network Performance of Multifunction On-chip Router Architectures (IEICE-CPSY...
Shinya Takamaeda-Y
 
オープンソースコンパイラNNgenでつくるエッジ・ディープラーニングシステム
Shinya Takamaeda-Y
 
DNNのモデル特化ハードウェアを生成するオープンソースコンパイラNNgenのデモ
Shinya Takamaeda-Y
 
ディープニューラルネットワーク向け拡張可能な高位合成コンパイラの開発
Shinya Takamaeda-Y
 
Veriloggen.Stream: データフローからハードウェアを作る（2018年3月3日 高位合成友の会 第5回 @東京工業大学）
Shinya Takamaeda-Y
 
Veriloggen.Thread & Stream: 最高性能FPGAコンピューティングを 目指したミックスドパラダイム型高位合成 (FPGAX 201...
Shinya Takamaeda-Y
 
Pythonによるカスタム可能な高位設計技術 (Design Solution Forum 2016@新横浜)
Shinya Takamaeda-Y
 
ゆるふわコンピュータ (IPSJ-ONE2017)
Shinya Takamaeda-Y
 
A Framework for Efficient Rapid Prototyping by Virtually Enlarging FPGA Resou...
Shinya Takamaeda-Y
 
A High Performance Heterogeneous FPGA-based Accelerator with PyCoRAM (Runner ...
Shinya Takamaeda-Y
 
PyCoRAM: Python-Verilog高位合成とメモリ抽象化によるFPGAアクセラレータ向けIPコア開発フレームワーク (FPGAX #05)
Shinya Takamaeda-Y
 
メモリ抽象化フレームワークPyCoRAMを用いたソフトプロセッサ混載FPGAアクセラレータの開発
Shinya Takamaeda-Y
 
PyCoRAM: Yet Another Implementation of CoRAM Memory Architecture for Modern F...
Shinya Takamaeda-Y
 
むかし名言集bot作りました！
Shinya Takamaeda-Y
 
APGAS言語X10を用いたオンチップネットワークシミュレーションの並列化
Shinya Takamaeda-Y
 
Network Performance of Multifunction On-chip Router Architectures (IEICE-CPSY...
Shinya Takamaeda-Y
 

Recently uploaded (20)

PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PPTX
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
PPTX
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
PDF
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
PDF
Best ERP System for Manufacturing in India | Elite Mindz
Elite Mindz
 
PPTX
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
PDF
Software Development Company | KodekX
KodekX
 
PPTX
ChatGPT's Deck on The Enduring Legacy of Fax Machines
Greg Swan
 
PPTX
How to Build a Scalable Micro-Investing Platform in 2025 - A Founder’s Guide ...
Third Rock Techkno
 
PDF
Doc9.....................................
SofiaCollazos
 
PPTX
Stamford - Community User Group Leaders_ Agentblazer Status, AI Sustainabilit...
Amol Dixit
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PDF
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
PDF
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
PDF
DevOps & Developer Experience Summer BBQ
AUGNYC
 
PDF
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
PDF
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
Best ERP System for Manufacturing in India | Elite Mindz
Elite Mindz
 
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
Software Development Company | KodekX
KodekX
 
ChatGPT's Deck on The Enduring Legacy of Fax Machines
Greg Swan
 
How to Build a Scalable Micro-Investing Platform in 2025 - A Founder’s Guide ...
Third Rock Techkno
 
Doc9.....................................
SofiaCollazos
 
Stamford - Community User Group Leaders_ Agentblazer Status, AI Sustainabilit...
Amol Dixit
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
DevOps & Developer Experience Summer BBQ
AUGNYC
 
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 

A CGRA-based Approach for Accelerating Convolutional Neural Networks

  • 1. A CGRA-based Approach for Accelerating Convolutional Neural Networks Masakazu Tanomoto, Shinya Takamaeda-Yamazaki, Jun Yao, and Yasuhiko Nakashima Nara Institute of Science and Technology (NAIST), Japan E-mail: shinya_at_is_naist_jp IEEE MCSoC'15 @Torino September 23, 2015
  • 2. Outline n  Motivation: Deep learning on embedded computers l  Target: Convolutional Neural Network (CNN) n  Our approach: CGRA-based CNN acceleration l  EMAX (Energy-aware Multi-mode Accelerator eXtension) l  Mapping CNN on EMAX n  Evaluation l  Performance per memory bandwidth l  Performance per area n  Conclusion MCSoC15 Shinya T-Y, NAIST 2
  • 3. Deep learning n  Recognition (Convolutional Neural Network (CNN)) l  Extracting high-level features automatically from raw data l  Ex) Image, speech, and text recognition, image search n  Reinforcement learning (Deep Q-Network (DQN)) l  Learning appropriate strategy for controlling something l  Ex) Gaming AI, Robot control MCSoC15 Shinya T-Y, NAIST 3 Playing Atari 2006 automatically (Human-level control through deep reinforcement learning [Nature'15]) Extracted features of human and cat (Building High-level Features Using Large Scale Unsupervised Learning [ICML'12])
  • 4. Convolutional Neural Network (CNN) n  Nesting multiple processing layers l  Convolution: Multiple small matrix-matrix multiplications •  Each weight matrix corresponds to a learned feature map •  Feature can be automatically learned by error propagation l  Pooling and Max-out: selection from multiple values l  Full connection: Large matrix-matrix multiplication n  Performance Bottleneck: Convolution l  Numerous small matrix-matrix multiplication with stencil MCSoC15 Shinya T-Y, NAIST 4 Input Layer Hidden Layers Output Layer Convolution Pooling Max Out Convolution Full Connection
  • 5. Motivation: DNN on embedded computers n  Machine learning on IoT: Learning and decision on edge computers will become more important l  Sending all data to data centers?: Network traffic problemL l  Decision on data centers?: Very long latencyL n  Challenge: Energy efficient embedded accelerators l  Why not GPU?: GPU is very energy hungry and requires absolute energy •  Not only energy efficiency, but also absolute peak energy amount is important l  Why not ASIC?: Limited capability of algorithm customization •  Algorithms of machine learning are rapidly evolving l  Why not FPGA?: Energy overhead to building computing logics l  CGRA? MCSoC15 Shinya T-Y, NAIST 5
  • 6. Computation pattern: Full connection n  Output vector is determined by a simple vector-matrix multiplication l  Input and output size is certainly large: more than 1024 l  Weight matrix size is also large n  GPU is OK: suitable for matrix multiplication l  GPU has matrix libraries: CUBLAS, ... MCSoC15 Shinya T-Y, NAIST 6 dot =Weight Output Vector Input Vector
  • 7. Computation pattern: Convolution n  A value of the result matrix is calculated by numerous matrix-matrix multiplication with a small weight matrix l  Weight matrix size is usually small: from 3 to 8 n  I know GPU is very fast for matrix-matrix multiplication l  Really? MCSoC15 Shinya T-Y, NAIST 7 3 dot Weight WeightWeight = Weight Weight Nextdot Weight WeightWeight dot Weight WeightWeight
  • 8. SGEMM performance on GPU n  GPU is fast, if the matrix size is large enough l  GPU is throughput-oriented processor l  In case of small matrix, parallelisms and memory bandwidth are not exploited efficiently MCSoC15 Shinya T-Y, NAIST 8 0 5 10 15 20 25 0 50 100 150 200 250 64 128 256 512 1024 2048 4096 #activewarpsperactivecycle Performance[GFLOPS] Matrix size warp/cycle (small kernel) GFLOPS warp/cycle (large kernel) NVIDIA Jetson TK1 (GK20A)
  • 9. Preprocessing for Convolution on GPU n  In order to use fast matrix multiplication library of GPU, data duplication is usually utilized l  Converting sub-regions into a single large matrix n  Faster than the naive convolution, but still just a performance overhead MCSoC15 Shinya T-Y, NAIST 9 3 k=3 k=3 n n Input vector [n-3, n-3] [n-1, n-1] Duplication Duplication [0,0] [0,1] [0,2] [1,0] [1,1] [1,2] [2,0] [2,1] [2,2] [0,1] [0,2] [0,3] [1,1] [1,2] [1,3] [2,1] [2,2] [2,3] Duplication 9 (=k2) (n-2)2 Temporal array for matrix multiplication
  • 10. Our approach: EMAX Energy-aware Multi-mode Accelerator eXtension n  A CGRA of local memory based PEs with several buses l  Each PE has a local memory for data locality MCSoC15 Shinya T-Y, NAIST 10 Interconnection DRAM CPU Core PE PE PE PE MemoryInterface EMAX PE PE PE PE PE PE PE PE
  • 11. Real Chip of EMAX n  12.5mm x 12.5mm in 180nm technology MCSoC15 Shinya T-Y, NAIST 11
  • 12. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 12 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const
  • 13. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 13 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const LMM: Local Memory
  • 14. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 14 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const FIFO
  • 15. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 15 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Execution units
  • 16. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 16 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Constant registers
  • 17. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 17 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Internal Shuffle Bus External Shuffle Bus
  • 18. Processing Element (PE) n  Local memory on each PE for efficient data locality and memory bandwidth utilization MCSoC15 Shinya T-Y, NAIST 18 EX1 LMM EX FIFO LMM FIFO DIN ADDR DOUT EX2 EAG Memory Bus PE Internal Shuffle Bus External Shuffle Bus Const Const Const Const Const Const Memory Bus
  • 19. EMAX instruction MCSoC15 Shinya T-Y, NAIST 19 Type1: row#, col#, dist [count] ALU_OP & MEM_OP RGI LMM_CONTROL Type2: row#, col#, dist [count] ALU_OP Type3: row#, col#, dist [count] & MEM_OP RGI LMM_CONTROL 32-bit operation add/add3/sub/sub3 16-bitx2 operation mauh/mauh3/msuh3 Misc operation mulh/mmrg3/msad/minl/minl3/mh2bw/ mcas/mmid3/mmax/mmax3/mmin/mmin3 Load from EX_FIFO ldb/ldub/ldh/lhuh/ld Floating Point Operation fmul/fma3/fadd 32-bit operation and/or/xor 16-bitx2 operation mauh/mauh3/msuh3 Load from LMM or LMM_FIFO ldb/ldub/ldh/lhuh/ld Store to LMM stb/sth/st/cst (a) Instruction format (a) EX1 operation (b) EX2 operation (c) LMM operation
  • 20. Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 20 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
  • 21. Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 21 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
  • 22. Forward propagation n  Weight matrix is constant in the inter-most loop l  Assigned into constant registers n  Index of In increases linearly l  Burst bulk transfer from the external memory MCSoC15 Shinya T-Y, NAIST 22 Operations per activation of EMAX Operations per clock cycle on EMAX for(i1=0; i1<InDim; i++){ for(j1=0; j1<(Nimg-Nk+1); j1++){ for(i2=0; i2<OutDim; i2++){ for(j2=0; j2<(Nimg-Nk+1)*(Nbatch); j2++){ for(ky=0; ky<Nk; iy++){ for(kx=0; kx<Nk; kx++){ Out[i2][j1][j2] += Weight[i1][i2][ky][kx]*In[i1][j1+ky][j2+kx]; } } } } } }� InDim: Dimension of input data, OutDim: Dimension of output data Nimg: Side length of input data, Nbatch: Bath size (= # number of pixels) Nk: Convolution window size
  • 23. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 23 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil
  • 24. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 24 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil 3x3 weight matrix in constant registers
  • 25. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 25 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil 3 Input data sets in LMMs Same read data is forwarded via FIFOs
  • 26. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 26 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Reading data from the constant register, LMM, and execution unit in the previous stage Operation result is passed to the next stage
  • 27. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 27 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Final result is stored into LMM in the next stage
  • 28. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 28 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Write back the previous data to the main memory
  • 29. CNN on EMAX (3x3 convolution) MCSoC15 Shinya T-Y, NAIST 29 LMM OPEX1/EX2 FIFO OPConst Const w[0][2] in[i-1][] LMM LD FIFO Const FIFO Const FMUL Const w[1][2] in[ i ][] LMM LD FMUL FIFO Const FMUL FIFO Const w[0][1] w[0][0] w[1][1] w[1][0] FMA Const w[2][2] in[i+1][] LMM LD FMA FIFO Const FMA FIFO Const w[2][1] w[2][0] Preload FMA FMA FMA LMM LD out[i][j] FADD Preload FADD Drain FADD LMM ST Preload a next input (in[i+2][]) from memory Preload a next input (out[i+1][]) from memory Drain the previous result to memory Loop Control MemoryInterface Store the current result to LMM ColumnRow Weight values are assigned on constant registers. Input data are stored on LMM 0 1 2 3 4 5 6 7 0 1 2 3 Next input data set as stencil Load the next input data from the main memory
  • 30. Evaluation setup n  Benchmark: deep learning datasets and networks l  Imagenet (Alexnet-2), CIFAR10, MNIST (Lenet) n  Hardware: l  CPU (Corei7, ARM), GPU (Desktop, Mobile), EMAX l  Metric: Performance per bandwidth, Performance per area •  Estimation from actual LSI of EMAX and software simulations MCSoC15 Shinya T-Y, NAIST 30
  • 31. Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 31 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM
  • 32. Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 32 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM Alexnet: since matrix size is large, desktop GPU is 3.17x better
  • 33. Performance per memory bandwidth n  EMAX achieves better performance in embedded class datasets MCSoC15 Shinya T-Y, NAIST 33 0 2 4 6 8 10 12 14 16 18 Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) Operations/Byte EMAX GTX980 GK20A Core i7 ARM CIFAR-10: 1.41x better than mobile GPU Lenet: 1.75x better than mobile GPU
  • 34. Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 34 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7
  • 35. Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 35 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7 Alexnet: since matrix size is large, desktop GPU is 2.2x better
  • 36. Performance per area n  EMAX achieves much better performance in embedded class datasets: CGRA is better for embedded systems? MCSoC15 Shinya T-Y, NAIST 36 0 100 200 300 400 500 600 700 800Alexnet-2C IFAR 10-1C IFAR 10-2C IFAR 10-3 C IFAR 10 (Avg) Lenet-1 Lenet-2Lenet(Avg) AreaPerf[FLOPS/Tr] EMAX GTX980 Corei7 CIFAR-10: 1.76x better than mobile GPU Lenet: 1.95x better than mobile GPU
  • 37. Conclusion n  A CGRA-based acceleration approach of convolutional neural network (CNN) for embedded accelerators l  EMAX (Energy-aware Multi-mode Accelerator eXtension) n  EMAX outperforms GPU in embedded class data sets l  1.75x better performance per memory bandwidth l  1.95x better performance per area ( energy) MCSoC15 Shinya T-Y, NAIST 37 Interconnection DRAM CPU Core PE PE PE PE MemoryInterface EMAX PE PE PE PE PE PE PE PE