2018 Deep Learning
2018 Deep Learning
0LFURVFRSLF&HOO3LFWXUHV8WLOL]LQJ6XSSRUW9HFWRU0DFKLQH
.1HDUHVW1HLJKERUDQG'HHS/HDUQLQJ
6DFKLQ3DVZDQ<RJHVK5DWKRUH
6WXGHQW07HFK&6(5,7(($VVLVWDQW3URIHVVRU5,7((
5DLSXU,QGLD5DLSXU,QGLD
(PDLO6NVDFKLQ#JPDLOFRP(PDLO\RJHVKUDWKRUH#JPDLOFRP
$%675$&7/HXNHPLDLVDGHYHORSPHQWRIWKH WKHUHIRUHPDGHWRGHYLVHDSSURDFKIRUURERWL]HG
EORRG DQG ERQH PDUURZ WKH IOH[LEOH WLVVXH UHJLRQ DQG VXEUHTXHVW RI 6HULRXV
DUUDQJHG WKH ERQHV ZKHUH SODWHOHWV DUH PDGH /\PSKREODVWLF /HXNHPLD $// XWLOL]LQJ
([WUDRUGLQDU\ P\HORLG OHXNHPLD $0/ LV D SLFWXUH SUHSDULQJ DQG PDFKLQH OHDUQLQJ
FKDPSLRQ DPRQJ WKH PRVW XQGHUVWRRG VRUWV RI IUDPHZRUNV $V QHFHVVLWLHV EH WR SDUW WKH
OHXNHPLD DPRQJ DGXOWV 7KH VLJQV DQG VLJQV RI FRQYHQWLRQDO PDNH O\PSKRF\WH DQG UXLQRXV
OHXNHPLD DUH QRQVSHFLILF LQ QDWXUH DQG EHVLGHV O\PSKREODVW SLFWXUHV LQWR FRQVWLWXHQW
WKH\ DUH IRU DOO LQWHQWV DQG SXUSRVHV PRUSKRORJLFDO DUHDV QRYHO SODQV KDYH EHHQ
LQGLVWLQJXLVKDEOH WR WKH V\PSWRPV RI RWKHU SURSRVHG 7KHVH SURSRVHG VWUDWHJLHV GHSHQG
VKDUHG LVVXH 0DQXDO PLFURVFRSLF H[DPLQDWLRQ XSRQ QHXUDO IUDPHZRUN MRLQ VSDFH FOXVWHULQJ
RI UHFRORUHG EORRG VSUHDG RU ERQH PDUURZ DQG 690 ILHOG GHOLQHDWLQJ ZKHUH WKH GLYLVLRQ
VXFWLRQ LV WKH PRVW LGHDO DSSURDFK WR IUXLWIXO LVVXH LV PDVWHUPLQGHG DV SL[HO SRUWUD\DO SL[HO
LQYHVWLJDWLRQ RI OHXNHPLD 5HJDUGOHVV WKLV JDWKHULQJ DQG SL[HO FKHFNLQJ LVVXH
SURFHGXUHLVPRQRWRQRXVDQGOHVVFRUUHFW,QWKLV LQGHSHQGHQWO\ 7R RUGLQDULO\ REVHUYH
SDSHUDWHFKQLTXHIRUFXVWRPL]HGGLVFORVXUHDQG O\PSKREODVWVDQGUHFRJQL]H$//LQIULQJHEORRG
SRUWUD\DORI$0/LQEORRGVSUHDGLVVKRZQ. WHVWV D FRPSHWHQW V\VWHP LV SURSRVHG
LPSOLHVFDOFXODWLRQLVXWLOL]HGIRUGLYLVLRQ.11 0RUSKRORJLFDO WH[WXUDO DQG VKDGLQJ KLJKOLJKWV
&11DQG690DUHXWLOL]HGIRUJURXSLQJ*/&0 DUH GLVHQJDJHG IURP WKH DSSRLQWHG IRFXV DQG
LV XWLOL]HG IRU VWUHDPOLQLQJ WKH JKRVWO\ F\WRSODVP]RQHVRIWKHO\PSKRF\WHSLFWXUHV$Q
KLJKOLJKWV 1HLJKERUKRRG GRXEOH H[DPSOH LV RXWILWRIFODVVLILHUVWHQGHGWRDV&11FRQWDLQLQJ
XWLOL]HGIRUVXUIDFHGHSLFWLRQ%ORRGPDJQLI\LQJ WKUHH FODVVLILHUV VKRZV PRVW DVWRQLVKLQJ FRXUVH
LQVWUXPHQWSLFWXUHVZHUHWULHGDQGWKHH[HFXWLRQ RI DFWLRQ SUHFLVLRQ RI RQ WKH RWKHU KDQG
RI WKH FODVVLILHU ZDV GLVVHFWHG $W ORQJ ODVW %\ ZLWKVLQJOHLQGLYLGXDOV
XWLOL]LQJ &11 H[DFWQHVV RI KDV EHHQ
DFFRPSOLVKHG (DFK PRGHO LV FRQWHPSODWHG XQLQKLELWHGO\ DQG
WHVWVDUHFRRUGLQDWHGWRVWXG\WKHLULQWURGXFWLRQV
.H\ZRUGV $XWRPDWHG OHXNHPLD GLVFRYHU\ ([HFXWLRQ PHDVXUHV LH DFFXUDF\
LQWHQVH O\PSKREODVWLF OHXNHPLD /\PSKRF\WH DSSURSULDWHQHVV RI WKH SURSRVHG PRGHUQL]HG
SLFWXUH GLYLVLRQ &RQYROXWLRQ QHXUDO V\VWHP VWUXFWXUHV ZLWK WKDW RI VWDQGDUG LQYHVWLJDWLYH
0DFKLQHOHDUQLQJDQGVRIRUWK IUDPHZRUNV
,,1752'8&7,21 $ /HXNHPLD /DE WHVWV HQDEOH WKH DFH WR
GLVFRYHUWKHNLQGRIOHXNHPLDWKDW\RXKDYH)RU
0LQXWH H[DPLQDWLRQ RI IULQJH EORRG VSUHDG LV D HYHU\ VRUW RI OHXNHPLD WKH WUHDWPHQW VHWXS LV
EDVLFDGYDQFHLQUHJLRQRIOHXNHPLD5HJDUGOHVV UHPDUNDEOH>@
WKLV VRUW RI OLJKW PRGHVW DVVHVVPHQW LV GXOO
RUGLQDULO\ VXEMHFWLYH DQG LV DGGUHVVHG E\ 0RVW(VVHQWLDO6RUWVRI/HXNHPLD
KHPDWRSDWKRORJLVWV FOLQLFDO LQVLJKWIXOQHVV DQG
H[SHULHQFH ,Q WKLV SDSHU HQGHDYRUV DUH
978-1-5386-2459-3/18/$31.00 2018
c IEEE 525
([FHSWLRQDOP\HORLGOHXNHPLD(0/LPSDFWV VWUXFWXUHUHFRJQL]HVDQGRUFKHVWUDWHVZLOGHUQHVV
P\HORLG FHOOV DQG GHYHORSV UDSLGO\ /HXNHPLF IHYHUSDUDVLWHVLQ*LHPVDUHFRORUHGEORRGVOLGHV
LQIOXHQFH FHOOV FROOHFW LQ WKH ERQH PDUURZ DQG SLFWXUHV%\WKHQDIWHUSDUDVLWDHPLDHYDOXDWLRQLV
EORRG GRQH 0RUSKRORJLFDO DSSURDFK WR PDQDJH FHOO
SLFWXUH GLYLVLRQ LV PRUH FRUUHFW WKDQ WKH
6HULRXV O\PSKREODVWLF OHXNHPLD 6// FXVWRPDU\ ZDWHUVKHGEDVHG FRPSXWDWLRQ LV
LPSDFWV O\PSKRLG FHOOV DQG GHYHORSV UDSLGO\ VKRZHGXSLQWKLVSDSHU
/HXNHPLF LQIOXHQFH FHOOV UHJXODUO\ DJJUHJDWH LQ
WKHERQHPDUURZDQGEORRG>@ ,Q >@ D V\VWHP PDVWHUPLQGV DQG SHUFHLYH
LQWHVWLQDO LQIHFWLRQ SDUDVLWH E\ XVLQJ PLQXWH
8QUHPLWWLQJ P\HORLG OHXNHPLD 80/ SLFWXUHV RI SODWHOHWV 0RUSKRORJLFDO DSSURDFK
LPSDFWV P\HORLG FHOOV DQG UHJXODUO\ GHYHORSV DQG WKH FULWLFDO QHFHVVLWLHV LQ ZRUNLQJ XS WKLV
FRQVWDQWO\ DW ILUVW %ORRG WHVWV GHPRQVWUDWH DQ VWUXFWXUH DUH WKH EHVW IUDPHZRUNV IRU SODWHOHW
DGYDQFHPHQW LQ WKH PHDVXUH RI ZKLWH SODWHOHWV SLFWXUHVGLYLVLRQ$QXQHTXLYRFDOORJLFDOPRGHO
7KHUH PLJKW EH IHZ OHXNHPLF VKRRW FHOOV LQ WKH IRU WUDLWV RI FHOO FHQWHUV OLNH VL]H DQG VKDSH
ERQHPDUURZ PHDVXUHV LV FRQVROLGDWHG )RU HDFK UHFRJQL]HG
FHQWHUDVXUHQHVVVFRUHLVILJXUHGE\HYDOXDWLQJ
6WHDG\ O\PSKRF\WLF OHXNHPLD 6// LPSDFWV ILWWLQJQHVV RI FHQWHU LQ WKH PRGHO >@ 7KH
O\PSKRLG FHOOV DQG DV D JHQHUDO UXOH GHYHORSV VWUHQJWKRIDQDOWHUHGPRUSKRORJLFDOIUDPHZRUN
ZHOO RUGHUHG %ORRG WHVWV GLVSOD\ DQ WR VHH WKH 6HULRXV /\PSKRF\WLF /HXNHPLD
LPSURYHPHQW LQ WKH PHDVXUH RI ZKLWH SODWHOHWV $//ZLWKWKHDVVLVWDQFHRISLFWXUHVRIRXWVNLUWV
7KH DVWRQLVKLQJ FHOOV ZRUN DOPRVW DQG LQ OLNH EORRG RSHQLQJ XS SRLQW RI FRQYHUJHQFH LV GRQH
PDQQHUWKHEDVLF:%&>@ LQ>@7KHVKRZHGIUDPHZRUNLQGLYLGXDWHVWKH
OHXFRF\WHVIURPWKHRWKHUVSODWHOHWVDIWHUWKDWLW
%([FHSWLRQDODQG8QHQGLQJ/HXNHPLDV SLFNV WKH O\PSKRF\WH FHOOV WKH FHOOV FDXVHV
/HXNHPLDV DUH QDPHG IRU KRZ UDSLGO\ WKH VHULRXV OHXNHPLD PRUSKRORJLFDO UHFRUGV IURP
WDLQWLQJ PDNHV DQG EUHDNV GRZQ([WUHPH WKRVH FHOOV DUH RYHUYLHZHG WKHQ DIWHU LQ
OHXNHPLD JHQHUDOO\ JURZV UDSLGO\ $ ERQH FRQFOXVLRQSRUWUD\DOLVSHUIRUPHG'LIIUHQWGHHS
PDUURZ WHVW PD\ GLVSOD\ DQ RGG VWDWH RI OHDUQLQJ DSSURDFKHV RQ PHGLFDO LPDJLQJ LV
OHXNHPLD FHOOV DQG ORZ OHYHOV RI QRUPDO DQDO\]H LQ >@ ZKLFK JHQHUDWHV UHPDUNDEOH
SODWHOHWV UHVXOWVLQFODVVLILFDWLRQ
8QHQGLQJ 3HUSHWXDO OHXNHPLDDVDUXOHFUHDWHV ,,, 6<67(002'(/
ZHOO RUGHUHG 7KH OHXNHPLD FHOOV ZRUN DOPRVW
DQG IXUWKHUPRUH EDVLF ZKLWH SODWHOHWV $ PDJQLI\LQJ OHQV KDV EHHQ H[HFXWHG E\ XV IRU
,QGLYLGXDOV PD\ QRW XQGHUVWDQG ZLSHG DW DFTXLULQJWKHSODWHOHWSLFWXUHVDQGKDVFRQQHFWHG
UHJDUGOHVV DQG WKH IXQGDPHQWDO LQGLFDWLRQ RI 0$7/$% SURJUDPPLQJ 5E IRU SLFWXUH
VLFNQHVVPLJKWLQWULJXHFRPHVDERXWRQDUXQRI GHDOLQJ ZLWK PLQXWH SODWHOHW SLFWXUHV RI
WKH PLOO EORRG WHVW )RU LQVWDQFH WKH EORRG WHVW YDULRXVVL]HVKDYHEHHQREWDLQHGIRUWHVWLQJ2XU
PD\GHPRQVWUDWHDQRGGVWDWHRIOHXNHPLDFHOOV VXEPLWWHG DSSURDFK JXDUDQWHHV YLDEOH JURXSLQJ
RI EORRG WLQ\ SLFWXUH ZLWK H[DFWQHVV 7KH
GLYLVLRQLVDFNQRZOHGJHGRXWLQWKHVXEVHTXHQW
,, /,7(5$785(5(9,(: VWDJHVZKLFKDUHVDLGEHQHDWK
,Q>@DQLWHUDWLYHWKUHVKROGLQJFRXQWLVXVHGIRU $6HFXULQJRI3LFWXHUV
GLYLVLRQ UHDVRQ SDUWLFXODUO\ IURP WXPXOWXRXV
%\ WKH JXLGH RI PDJQLI\LQJ OHQV WKH EORRG
SLFWXUHV 7KLV FRXQW FUXVKHV WKH LVVXH RI FHOO
SLFWXUHV DUH JDLQHG LQ WKLV SLFWXUH DUH VKRZQ LQ
H[WUDFWLRQ DQG GLYLVLRQ IURP FRQVLGHUDEOH
' JULG ZKHUH WKH SL[HOV RI WKH SLFWXUH DUH
ERLVWHURXV SLFWXUHV 7KLV FRPSXWDWLRQ ZRUNV
HQYLVLRQHGDVWKHFRPSRQHQWVXFKQHWZRUNVDUH
RYHU WKH DGMXVWHG HGJH RI SLFWXUHV LWHUDWLYHO\
FRPSOHWHO\ UHOLDQW RQ ILHOG RI YLHZ DQG ODWWLFH
RIIHULQJKHDOWKLQHVVWRSLFWXUH,Q>@GLVFXVVHV
HVWLPDWH 2XU SURSRXQGHG DSSURDFK H[HFXWHV
WKH PDODULDO SLFWXUH GHDOLQJ ZLWK VWUXFWXUH 7KLV
IUDFWDO HVWLPDWLRQV ,Q UHDVRQDEOH DSSOLFDWLRQV 'HHS OHDUQLQJ LV D ]RQH RI PDFKLQH OHDUQLQJ
WKH FRQWDLQHU FKHFNLQJ HVWLPDWLRQ LV JHQHUDOO\ UHVHDUFK ZKLFK KDV EHHQ SUHVHQWHG ZLWK WKH
XWLOL]HGEHFDXVHRIKRZLWLVXQTXHVWLRQDEO\QRW GHVWLQDWLRQV RI PDFKLQH DGDSWLQJ PRUH OLNH RQH
KDUG WR H[HFXWH ,Q ER[ FKHFNLQJ ILJXULQJV WKH RI LWV XQLTXH REMHFWLYHV ,W LV JLYHQ HQHUJL]LQJ
PHDVXUHRIER[HVFRYHULQJDSRLQWVHWLVDSRZHU DQVZHUV IRU UHVWRUDWLYH SLFWXUH H[DPLQDWLRQV
ODZ XWPRVW RI WKH FDVH PHDVXUH $OO IUDFWDO LVVXH DQG LW LV YLHZHG DV D NH\ WHFKQLTXH IRU
HVWLPDWLRQV DUH DVVHVVHG DV WKH LQVWDQFH RI VXFK IXWXUHDSSOLFDWLRQV
SRZHUODZVDQGDUHUHDOQXPEHUVWKDWSRUWUD\WKH
IUDFWDOQHVVVXUIDFHRUEUXWDOLW\RIWKHWKLQJV &RQYROXWLRQQHXUDOV\VWHP&11
&11LVLQYROYHGDWOHDVWRQHFRQYROXWLRQOD\HUV
,9([SHULPHQWDO6WUDWHJ\ DQG DIWHUZDUG WRRN DIWHU E\ DW OHDVW RQH
$.1HDUHVW1HLJKERU FRPSOHWHO\ DVVRFLDWHG OD\HUV DV LWV VWDQGDUG
PXOWLOD\HUHGQHXUDOV\VWHPV
7KH .11 WZRIROG DV WZR FODVV LV JLYHV PRUH
7KHUHFLSHIRUFRPSXWLQJWKH\LHOGHVWLPDWH
FRUUHFWGDWDSRUWUD\DOZKLFKEHQHILFLDOWRSLFNN
DV DQ RGG QXPEHU ZKLFK DYRLGV WKH FDSULFLRXV
2 :.3)»S ZKHUH 2 LV \LHOG : LV
GDWD 7KH .11 IUDPHZRUN LV WKH SURFHGXUH LQ
LQSXW WDOOQHVVOHQJWK . LV WKH FKDQQHO HVWLPDWH
0/ VWUDWHJLHV ,W LV DQ LQTXLU\ ZKLFK JDWKHUHG
3LVWKHFXVKLRQLQJDQG6LVWKHZDON
WKURXJK D VWDQGDUG VHOHFWLRQ RI LWV QHLJKERUV
ZLWK WKH FRQILUPDWLRQ GROHG RXW RFFDVLRQ IRU
PRVWVKDUHGFODVVDPRQJLWVNQHDUHVWQHLJKERUV
N LV D SRVLWLYH HQWLUH QXPEHU JHQHUDOO\ OLWWOH
&XVWRPDULO\ (XFOLGHDQ SDUWLWLRQ LV XVHG DV WKH
GLYLVLRQPHWULFKRZHYHUWKLVLVVLPSO\ILWWLQJWR
ERXQGOHVVYDULDEOHV,QVXFKVLWXDWLRQOLNHJDPH
SODQRIHOHFWLYHPHWULFFURVVLQJSRLQWPHWULFRU
+DPPLQJGLYLVLRQFDQEHXVHG îîîîîîîî
îîîîîî
%6XSSRUW9HFWRU0DFKLQH
)LJXUH9**$UFKLWHFWXUHXVHGIRU
690 LV QRUPDOO\ D WZRIROG UHTXHVW IUDPHZRUN
&ODVVLILFDWLRQ
WKHUHDUHGLVWLQFWLYHV\VWHPVDFFHVVLEOHWRH[WHQG
LW WR D PXOWLFODVV FODVVLILHU 690 LV QRUPDOO\ D 9**DUFKLWHFWXUHRIGHHSOHDUQLQJLVXVHGIRU
WZRIROG FODVVLILHU DORQJ WKHVH OLQHV LW FDQ FODVVLILFDWLRQ RI OHXNHPLD KHUH ZH XVH WKH
RUJDQL]HIXVHLQWRERWKRIWZRFODVVHVIRUZKLFK ,PDJH1HW SUH WUDLQHG PRGHO IRU
LW KDV EHHQ SUHSDUHG >@ 7R XWLOL]H 690 IRU FODVVLILFDWLRQ>@
PXOWLFODVV LVVXH GLVWLQFWLYH IUDPHZRUNV KDYH
EHHQXWLOL]HGDPLGWKHWLPH>@WKDWXWLOL]DWLRQ 95HVXOWV$QDO\VLV
D PL[ RI D IHZ WZRIROG 690 FODVVLILHUV ,Q WKH
HVVHQWLDO DSSURDFK RQH DJDLQVW DOO WKH WHVW 7KH GDWDVHW XVHG DV D SLHFH RI WKLV SDSHU
LQIRUPDWLRQ TXHVWLRQ LV FRRUGLQDWHG LQ OLJKW RI LQYROYHG ERQD ILGH PLFURVFRS\ SLFWXUHV RI
)LJXUH$FFXUDF\&RPSDULVRQJUDSK
)LJXUH3UHSURFHVVLQJSKDVHRI/HXNHPLD
9,$&.12:/(*0(17
7KLVSDSHUSURSRVHGDWHFKQLTXHWRFRPSXWHUL]H
WKH VHJPHQWDWLRQ LQFOXGH H[WUDFWLRQ DQG
FODVVLILFDWLRQRIUHGDQGZKLWHSODWHOHWVXWLOL]LQJ
.11&11DQG690FODVVLILFDWLRQFDOFXODWLRQ
)LJXUH+DXVGRUIIIHDWXUHSKDVHRI/HXNHPLD $ IHZ FKDQJHV ZHUH PDGH WR WKH 690
FDOFXODWLRQLQFOXGLQJDQLQWURGXFWLRQDGYDQFHWR
GLVFRYHU QHLJKERU DVVRFLDWHG VHJPHQW
$GGLWLRQDOO\ WKH SURSRVHG GHPRQVWUDWH
KLJKOLJKWVDQXSJUDGHGSUHFLVLRQRIFKRRVLQJWKH
ULJKW KRYHU IURP WKUHH FDQGLGDWH FLUFOHV WKH
FDSDFLW\ WR UHFRJQL]H XQSUHGLFWDEOH FHOOV WKH
XWLOL]DWLRQ RI G\QDPLF QXPEHU RI F\FOHV DQG
HQKDQFHG GLVFRYHU\ RI FRYHULQJ FHOOV 7KH
)LJXUH/%3IHDWXUHSKDVHRI/HXNHPLD SURSRVHG VWUDWHJ\ SOD\HG RXW WKH VHJPHQWDWLRQ
DQG FODVVLILFDWLRQ RI :%&V DQG 5%&V ZHOO
,Q /%3 VWUDWHJ\ HYHU\ SL[HO LV UHHVWDEOLVKHG ZKHQ FRPHV DERXW ZHUH FRQWUDVWHG DQG WKH
WKURXJK D WZRIROG H[DPSOH WKDW LV JRWWHQ IURP JURXQG WUXWK ZKLFK ZDV FRQWUROOHG E\
WKHSL[HOVQHLJKERUKRRG VSHFLDOLVWV %\ XVLQJ '((3 /($51,1*
DFFXUDF\ LV 7KH DFFRPSDQ\LQJ
VHJPHQWDWLRQ DQG WDOO\LQJ H[DFWQHVV¶V ZHUH
DFFRPSOLVKHGXWLOL]LQJWKHSURSRVHGVWUDWHJ\
5()(5(1&(6